Information processing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

To reduce an operational burden by improving the degree of freedom of virtual viewpoint operation in generation of a virtual viewpoint image.SOLUTION: An information processing apparatus according to the present invention includes an operation unit configured to operate a plurality of operation axes of a virtual camera arranged in a virtual three dimensional space, a storage unit configured to store a virtual camera parameter, an editing unit configured to edit the virtual camera parameter stored in the storage unit based on an operation of the operation axis, a setting unit configured to set, for each of the plurality of operation axes, whether editing is permitted, and a generation unit configured to generate an image virtually captured by the virtual camera based on the virtual camera parameter. The virtual camera parameters are edited by maintaining the values of the virtual camera parameters that are not to be changed.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to a technique for performing an operation to specify an arbitrary virtual viewpoint in generating a virtual viewpoint image. [Background technology]

[0002] In recent years, a technology has been attracting attention in which multiple cameras are placed at different positions to synchronously capture images from multiple viewpoints, and the multiple viewpoint images obtained by the capture are used to generate virtual viewpoint images that are not only images at the positions where the cameras are placed but also images from any viewpoint. The generation and viewing of virtual viewpoint images based on multiple viewpoint images is realized by collecting images captured by the multiple cameras in an image processing unit such as a server, performing processing such as rendering based on the virtual viewpoint in the image processing unit, and then displaying the virtual viewpoint image on a user terminal.

[0003] In this virtual viewpoint image, the viewpoint is changed by manipulating the position and attitude of a virtual camera placed in a virtual three-dimensional space with a controller. Specifically, the position of the virtual camera is controlled along the three axes of X, Y, and Z, and the attitude of the virtual camera is controlled along the three axes of pan, tilt, and roll. Furthermore, the angle of view of the virtual camera is controlled along the zoom axis, and the playback speed is controlled along the time axis.

[0004] Until now, when changing the viewpoint in a virtual viewpoint image, multiple operators have been responsible for operating each of the above-mentioned operation axes. Alternatively, one operator has operated two controllers simultaneously using both hands, or a controller capable of simultaneously operating each of the above-mentioned operation axes has been used. Furthermore, Patent Document 1 discloses a system that stores a history of operations performed by other users in the past and presents (distributes) the operation history as recommendations in order to obtain an image from a desired viewpoint with reduced operational burden. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2016 / 178340 Summary of the Invention [Problem to be solved by the invention]

[0006] However, when multiple operators share the work, there is a risk that the other operators may perform operations different from the operator's intention. This can result in the desired camera parameters not being obtained. Furthermore, when creating multiple virtual viewpoint images simultaneously, the labor costs increase with the number of operators, which can lead to higher production costs.

[0007] On the other hand, when one operator simultaneously operates each of the above-mentioned operation axes, it is difficult to accurately adjust the values ​​of all operation axes. Also, even when an operator wants to edit (adjust) the camera parameters that have already been created by specifying the values ​​of only some of the operation axes, it is necessary to start over with all operation axes, making it difficult to obtain a desired virtual viewpoint image. This is also the case with Patent Document 1, which means that there are restrictions on the degree of freedom in virtual viewpoint operation.

[0008] Therefore, the present invention has been made to solve at least one of the above-mentioned problems of the conventional art, and has an object to improve the operability of a virtual viewpoint. [Means for solving the problem]

[0009] In order to solve the above problem, the information processing device according to the present disclosure includes: tentativeAn information processing device for controlling a virtual camera corresponding to a virtual viewpoint image, the information processing device comprising: a first setting means for performing a first setting in accordance with a user operation to set either a mode that allows a position of the virtual camera to be changed on a first axis corresponding to a first movement direction, or a mode that prohibits a position change on the first axis; a second setting means for performing a second setting in accordance with a user operation to set either a mode that allows a position of the virtual camera to be changed on a second axis corresponding to a second movement direction different from the first movement direction, or a mode that prohibits a position change on the second axis; and a display control means for controlling the display of first information indicating whether the mode that allows a position to be changed on the first axis or the mode that prohibits a position change on the first axis is set, and second information indicating whether the mode that allows a position to be changed on the second axis or the mode that prohibits a position change on the second axis is set, and the first setting and the second setting can be performed independently of each other. [Effects of the Invention]

[0010] According to the present invention, the operability of the virtual viewpoint is improved. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an image processing system. [Figure 2] FIG. 2 is a schematic diagram of an input / output unit of an information processing device. [Figure 3] FIG. 10 is a diagram for explaining a camera view window. [Figure 4] FIG. 10 is a diagram illustrating the pilot tab of the pilot window. [Figure 5] FIG. 10 is a diagram illustrating the camera control tab of the pilot window. [Figure 6] This is an excerpt of a portion of the pilot window when the camera control tab is selected. [Figure 7] FIG. 10 is a diagram illustrating a replay window. [Figure 8]FIG. 2 is a diagram for explaining a three-axis controller. [Figure 9] FIG. 2 is a diagram illustrating a zoom controller. [Figure 10] FIG. 10 is a diagram illustrating a replay controller. [Figure 11] FIG. 10 is a diagram illustrating an operation axis of the virtual camera. [Figure 12] FIG. 2 is a diagram illustrating functional blocks of the information processing device. [Figure 13] 10 is a flowchart showing a processing procedure in an information processing device. [Figure 14] FIG. 10 is a diagram showing the result of editing camera parameters. [Figure 15] FIG. 10 is a diagram illustrating an operation axis of the virtual camera. [Figure 16] FIG. 10 is a diagram illustrating an operation axis of the virtual camera. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. In addition, various other embodiments within the scope of the gist of the present invention are also included in the present invention, and parts of the following embodiments can also be combined as appropriate.

[0013] Fig. 1 is a diagram for explaining an image processing system 10. More specifically, Fig. 1(a) is a diagram showing the overall configuration of the image processing system 10, and Fig. 1(b) is a diagram showing the hardware configuration of an information processing device 103 of the image processing system 10. Each of Fig. 1(a) and Fig. 1(b) will be described below.

[0014] 1(a), the image processing system 10 includes an imaging system 101, a virtual viewpoint image generation server 102, and an information processing device 103. The image processing system 10 can generate a virtual viewpoint image.

[0015] The imaging system 101 has multiple cameras arranged at different positions, and synchronously captures multiple images from multiple viewpoints. The imaging system 101 transmits the multiple images synchronously captured from multiple viewpoints to the virtual viewpoint image generation server 102.

[0016] The virtual viewpoint image generation server 102 acquires multiple images captured synchronously from multiple viewpoints, and generates a virtual viewpoint image viewed from a virtual camera based on the multiple images. Note that the virtual camera here refers to a virtual camera that can move freely within the shooting space. In other words, this virtual camera can capture images from a viewpoint different from that of any camera installed (placed) within the shooting space. However, certain restrictions may be imposed on the position and orientation of the virtual camera. The viewpoint of the virtual camera is controlled by camera parameters determined by the information processing device 103, which will be described later.

[0017] The virtual viewpoint image generation server 102 sequentially generates virtual viewpoint images from the received multiple images. Therefore, for example, a live virtual viewpoint image can be generated. Note that the live virtual viewpoint image generated in the image processing system 10 is a virtual viewpoint image based on an image captured by the image capturing system 101 at a time that takes into consideration processing delays in the image capturing system 101 and the virtual viewpoint image generation server 102 relative to the current time.

[0018] Furthermore, the virtual viewpoint image generation server 102 has a predetermined database and a function for recording the received multiple images. Therefore, from the recorded multiple images, it is possible to generate a past virtual viewpoint image, that is, a replay virtual viewpoint image (i.e., a virtual viewpoint image based on an image captured by the imaging system 101 at any time).

[0019] Additionally, in the following description, unless otherwise specified, the term "image" includes the concepts of both moving images and still images. In other words, the image processing system 10 can process both still images and moving images.

[0020] The information processing device 103 controls the virtual camera and determines camera parameters that indicate the viewpoint of the virtual camera. The camera parameters of the virtual camera include parameters for specifying at least one of the position, orientation, zoom, and time of the virtual camera.

[0021] The position of the virtual camera specified by the camera parameters is indicated, for example, by three-dimensional coordinates. Specifically, it can be indicated by coordinates in a Cartesian coordinate system of three axes: X-axis, Y-axis, and Z-axis. In this case, the position of the virtual camera specified by the camera parameters indicates coordinates and is composed of parameters for the three axes: X-axis, Y-axis, and Z-axis. In addition, the origin can be set to any position within the shooting space.

[0022] The attitude of the virtual camera specified by the camera parameters is indicated by, for example, the angle formed with three axes of pan, tilt, and roll. In this case, the attitude of the virtual camera specified by the camera parameters is made up of parameters for the three axes of pan, tilt, and roll. The zoom of the virtual camera specified by the camera parameters is indicated by a single-axis parameter of focal length. Similarly, the time is also indicated by a single-axis parameter.

[0023] In this way, the camera parameters of the virtual camera include parameters for eight axes, and the information processing device 103 can control these eight axes. Note that the camera parameters may include parameters that define other elements, or may not include all of the parameters for the eight axes described above.

[0024] The information processing device 103 transmits the determined camera parameters of the virtual camera to the virtual viewpoint image generation server 102. Upon receiving the camera parameters of the virtual camera, the virtual viewpoint image generation server 102 generates a virtual viewpoint image based on the received camera parameters and transmits the generated virtual viewpoint image to the information processing device 103. The information processing device 103 then displays the received virtual viewpoint image on the camera view 301. Note that, although a live virtual viewpoint image and a replay virtual viewpoint image may be generated by a single information processing device 103 as in this embodiment, two information processing devices may be used, and the live virtual viewpoint image and the replay virtual viewpoint image may each be generated by a different information processing device. Furthermore, although the virtual viewpoint image generation server 102 and the information processing device 103 are configured as separate entities in the above-described FIG. 1(a), the information processing device 103 may also be configured to include the virtual viewpoint image generation server 102.

[0025] 1B, the hardware configuration of the information processing device 103 will be described. The information processing device 103 includes a CPU 111, a RAM 112, a ROM 113, a communication unit 114, and an input / output unit 115 as its hardware configuration.

[0026] The CPU (Central Processing Unit) 111 controls the operation of the information processing device 103 using data stored in the RAM 112 and computer programs stored in the ROM 113. The RAM (Random Access Memory) 112 temporarily stores computer programs read from the ROM 113, intermediate data that is the intermediate results of calculations, data supplied from the outside via the communication unit 114, etc. The ROM (Read Only Memory) 113 holds computer programs and data that do not require modification. Note that the ROM 113 also includes non-volatile memory for storing data that needs to be retained even when the power is cut off.

[0027] The communication unit 114 includes a communication means such as Ethernet or USB (Universal Serial Bus) and communicates with the virtual viewpoint image generation server 102. The input / output unit 115 includes a plurality of controllers for controlling the virtual cameras and a plurality of display units for displaying the status of the virtual cameras, etc.

[0028] FIG. 2 is a schematic diagram of the input / output unit 115 of the information processing device 103. As shown in FIG. 2, the input / output unit 115 includes three display units (201a, 201b, 201c). The display unit 201a displays a camera view window. The camera view window is a window for displaying a virtual viewpoint image received from the virtual viewpoint image generation server 102. The display unit 201b displays a pilot window. The pilot window is a window for controlling the virtual camera. The display unit 201c displays a replay window. The replay window is a window for generating and editing a replay virtual viewpoint image.

[0029] In the following description, the display units 201a, 201b, and 201c may be collectively referred to as the display unit 201. The information processing device 103 may also include a touch panel, a mouse, a keyboard, and the like (not shown) for operating the above-mentioned windows.

[0030] 2, the input / output unit 115 includes four controllers 202a, 202b, 203, and 204. The input / output unit 115 receives instructions for controlling the virtual camera in response to a user's operation of the controller. That is, the input / output unit 115 receives instructions for changing (controlling) the position, orientation, etc. of the virtual camera.

[0031] The three-axis controller 202a and the three-axis controller 202b are controllers that control (operate) three axes. Any control can be assigned to each axis of the three-axis controller by setting. For example, control of the X-axis, Y-axis, and Z-axis for specifying the position of the virtual camera can be assigned to each axis of the three-axis controller 202a. Furthermore, control of pan, tilt, and roll for specifying the attitude of the virtual camera can be assigned to each axis of the three-axis controller 202b.

[0032] The zoom controller 203 controls the zoom of the virtual camera. The replay controller 204 controls the function for generating a replay virtual viewpoint image. The replay controller 204 is also assigned the control of the time of the virtual camera.

[0033] 2 shows an example of a configuration including three display units for input / output unit 115, but it may be configured to include one, two, or four or more display units. Similarly, while FIG. 2 shows an example of a configuration including four controllers for input / output unit 115, it may be configured to include three or less, or five or more controllers.

[0034] 3 is a diagram illustrating a camera view window. As described above, the camera view window is displayed on the display unit 201a. The camera view 301 is a display area that displays a virtual viewpoint image received from the virtual viewpoint image generation server 102. This virtual viewpoint image is a virtual viewpoint image seen from a virtual camera controlled by the user. The information processing device 103 can also display past virtual viewpoint images by controlling the time of the virtual camera.

[0035] The scene time 302 is the time at which a captured image is captured and used to generate the virtual viewpoint image displayed by the camera view 301. In other words, it is the time at which the virtual viewpoint image is displayed in the camera view 301.

[0036] The scene playback speed 303 is the playback speed of the virtual viewpoint image displayed in the camera view 301. When the scene playback speed 303 is 100%, the virtual viewpoint image is played at the normal playback speed. When the scene playback speed 303 is less than 100%, the virtual viewpoint image is played back slowly. For example, when one second of video in real time is played back in two seconds, the scene playback speed 303 is 50%. When the scene playback speed 303 is greater than 100%, the virtual viewpoint image is played back at fast forward.

[0037] When the scene playback speed 303 is 0%, the virtual viewpoint image at a certain shooting time is displayed in a time-stopped state in the camera view 301, and the scene time 302 does not change. However, even when the scene playback speed 303 is 0%, it is possible to control the virtual camera. Specifically, for example, in soccer, it is possible to generate a virtual viewpoint image that circles around a specific player the moment the player kicks the ball.

[0038] The camera view state 304 is the state of the virtual viewpoint image displayed in the camera view 301. Specifically, for example, the camera view state 304 has the following five states ("Review replay clip", "Live", "Recording", "Edit replay", and "None").

[0039] "Review replay clip" is the state in which the virtual viewpoint image of the replay clip is being played back. "Live" is the state in which the live virtual viewpoint image is being played back. "Recording" is the state in which the replay clip is being recorded. "Edit replay" is the state in which the recorded replay clip is being edited. "None" is the other state.

[0040] FIG. 4 is a diagram illustrating the pilot tab of the pilot window. As described above, the pilot window is displayed on display unit 201b. Switch button 401 is a button for switching the content displayed on the pilot window, and is composed of a pilot tab and a camera control tab. When the pilot tab is selected, the status of the virtual camera is mainly displayed on display unit 201b. When the camera control tab is selected, information related to the control of the virtual camera is mainly displayed on display unit 201b.

[0041] The context view 408 is a display area that displays, from a bird's-eye view, the position, orientation, etc. of the virtual camera controlled by the user. In the example shown in Fig. 4, the pilot window includes four context views 408. As shown in Fig. 4, the upper left context view 408 displays an image viewed from directly above the field, and the lower left context view 408 displays an image viewed from the left side of the upper left context view 408 along the long side of the field. Furthermore, the lower right context view 408 displays an image viewed from below the upper left context view 408 along the short side of the field, and the upper right context view 408 displays an image looking down on the field from diagonally above.

[0042] These images are generated using, for example, a CG (Computer Graphics) image of a stadium model. Alternatively, a virtual camera for the context view 408 may be prepared separately to generate virtual viewpoint images. By checking the context view 408, the user can easily understand the position and orientation of the virtual camera in the shooting space (for example, the stadium).

[0043] The camera parameter display 410 displays the camera parameters of the virtual camera. Here, the camera parameters of the virtual camera displayed include coordinate values ​​indicating the position of the virtual camera, and pan, tilt, roll, and zoom indicating the attitude of the virtual camera. Furthermore, the information processing device 103 can change the values ​​of the camera parameters when it receives an operation on a bar corresponding to each parameter in a GUI (Graphical User Interface). Among the camera parameters, the time of the virtual camera is displayed in the scene time 302. The virtual camera angle of view display 412 is CG that indicates the position and attitude of the virtual camera in the shooting space.

[0044] 5 is a diagram illustrating the camera control tab of the pilot window. X-axis control settings 501 is a setting section for making various settings related to movement of the virtual camera in the X-axis direction (left and right direction). Y-axis control settings 502 is a setting section for making various settings related to movement of the virtual camera in the Y-axis direction (front and back direction). Z-axis control settings 503 is a setting section for making various settings related to movement of the virtual camera in the Z-axis direction (up and down direction).

[0045] Pan control setting 504 is a setting section for making various settings related to the rotation of the virtual camera in the pan direction. Tilt control setting 505 is a setting section for making various settings related to the rotation of the virtual camera in the tilt direction. Roll control setting 506 is a setting section for making various settings related to the rotation of the virtual camera in the roll direction. Zoom control setting 507 is a setting section for making various settings related to the zoom of the virtual camera.

[0046] Figure 6 is an excerpt of a portion of the pilot window when the camera control tab in Figure 5 is selected. In Figure 6, operation axis name 511 indicates the item name of each operation axis. Camera parameter value 512 indicates the current value of each camera parameter determined in response to the operation of each operation axis. Slide bar 513 is a slide bar for making various settings such as the rate of change of the camera parameter in response to the amount of operation of the controller.

[0047] The On switch 514 is a switch for enabling editing operations for each operation axis, and the Off switch 515 is a switch for disabling editing operations for each operation axis. Specifically, when the On switch 514 is clicked, the operation axis becomes capable of editing operations (On state), the On switch 514 for that operation axis is highlighted, and the Off switch 515 for that operation axis is not highlighted. When the Off switch 515 is clicked, the operation axis becomes disabled of editing operations (Off state), the Off switch 515 for that operation axis is highlighted, and the On switch 514 for that operation axis is not highlighted. In Figure 6, Figure 6(a) shows a case where all operation axes are in the On state, and Figure 6(b) shows a case where the X-axis, Y-axis, and Z-axis for specifying the position of the virtual camera are in the Off state, and the other operation axes are in the On state.

[0048] Note that the editing operation of each operation axis is not necessarily limited to the above-described form (i.e., On switch 514, Off switch 515) as long as a function for switching between enabling and disabling is implemented. Therefore, for example, a single switch or button may be used to switch between enabling and disabling the editing operation of each operation axis. Furthermore, On switch 514 and Off switch 515 may be displayed in another window (for example, camera parameter display 410 in the pilot window when the pilot tab in FIG. 4 is selected).

[0049] Fig. 7 is a diagram illustrating the replay window. The replay window mainly displays information related to the generation of replay clips. The replay clip list 604 is a list for managing replay clips 605 generated by the user. Note that the replay clip list 604 in Fig. 7 displays replay clips 605 in order from the most recent.

[0050] The replay clip 605 is a recording of the camera parameters of a virtual camera. That is, the information processing device 103 records the replay clip 605 not as an image but as camera parameters. Then, virtual viewpoint images of the recorded camera parameters are generated in order to form replay images.

[0051] When the Edit Replay Clip button 606 is pressed in the replay window, editing of the replay clip begins. In replay clip editing, camera parameters that have been created are re-operated and modified to improve them. When modifying the created camera parameters, only the values ​​of the specified operation axes of the virtual camera are changed, and the previous values ​​of the other operation axes are maintained (retained). This makes it possible to reduce the number of operation axes to be operated in a single operation (modification), i.e., to improve the degree of freedom in virtual viewpoint operation, thereby reducing the burden of operation and reducing the number of operators.

[0052] The camera parameters of the virtual camera are recorded on the timeline 607 for each frame as the match progresses. The virtual camera parameters are recorded on the timeline 607 as follows: When the information processing device 103 is used to generate a live virtual viewpoint image, the camera parameters of the virtual camera at that time are recorded on the timeline 607. When the information processing device 103 is used to generate a replay virtual viewpoint image, the camera parameters of the virtual camera are not recorded on the timeline 607.

[0053] When a point on the timeline 607 is clicked, a jump occurs to the clicked time, and the camera parameter values ​​of the frame corresponding to that time are set for the virtual camera. If camera parameters are not recorded in the corresponding frame, the camera parameters of the virtual camera at the time of the click are applied. In other words, in this case, the position and orientation remain the same, and only the time jumps.

[0054] 8 is a diagram illustrating the three-axis controller 202. One three-axis controller 202 is capable of controlling three axes, and includes six knobs 801 to 806, six buttons 807 to 812, a seesaw switch 813, and a three-axis joystick 814. In this embodiment, two three-axis controllers 202 are used and six-axis control is assigned. Hereinafter, one three-axis controller 202a is assigned to control the three axes of X, Y, and Z, and the other three-axis controller 202b is assigned to control the three axes of pan, tilt, and roll.

[0055] Next, the functions assigned to the knobs, buttons, seesaw switch, and three-axis joystick of the three-axis controller 202a will be described. In the three-axis controller 202a, button 807 is assigned to turn the X axis on / off, button 808 is assigned to turn the Y axis on / off, and button 809 is assigned to turn the Z axis on / off. Also, the seesaw switch 813 is assigned the Z axis, and the three-axis joystick 814 is assigned the X axis for tilting left and right, and the Y axis for tilting forward and backward. Note that twisting left and right is not assigned in the three-axis controller 202a.

[0056] Similarly, the functions assigned to the knobs, buttons, seesaw switch, and three-axis joystick of three-axis controller 202b will be described. In three-axis controller 202b, pan On / Off is assigned to button 807, tilt On / Off is assigned to button 808, and roll On / Off is assigned to button 809. In addition, three-axis joystick 814 is assigned such that tilting left and right is pan, tilting forward and backward is tilt, and twisting left and right is roll. In three-axis controller 202b, no predetermined function is assigned to seesaw switch 813.

[0057] As a supplementary note, a joystick that can control more than three axes can also be used. For example, translation of the joystick left and right can be assigned to the X axis, translation forward and backward to the Y axis, pulling up / pushing down to the Z axis, tilting left and right to pan, tilting forward and backward to tilt, and twisting left and right to roll.

[0058] 9 is a diagram illustrating the zoom controller 203. The zoom controller 203 includes two knobs 901 and 902, four buttons 903 to 906, a dial 907, and a seesaw switch 908. In the zoom controller 203, the button 903 is assigned to turn zoom on and off, the dial 907 is assigned to focus, and the seesaw switch 908 is assigned to zoom.

[0059] 10 is a diagram illustrating the replay controller 204. The replay controller 204 includes thirteen buttons 1001 to 1013, a slider 1014, a jog wheel 1015, and a finger wheel 1016. Below, the functions that can be assigned to the replay controller 204 will be described.

[0060] REC is assigned to button 1010, and when this button 1010 is pressed, creation of a replay clip begins, and thereafter, the parameters of the virtual camera operated by the user are recorded as the replay clip. PLAY / PAUSE is assigned to button 1011, and this button 1011 is used to play and pause a scene. Note that while paused, the time of the virtual camera is stopped. Furthermore, REVIEW is assigned to button 1012, and when this button 1012 is pressed, playback of the replay clip begins.

[0061] The slider 1014 is a slider for setting the scene playback speed. When the scene playback speed is set with the slider 1014, the time of the virtual camera is changed according to the set scene playback speed. The jog wheel 1015 is for setting the time. The jog wheel 1015 is used to rewind, fast-forward, etc. the time. The finger wheel 1016 is for setting the time in frame units. The finger wheel 1016 is used to rewind, fast-forward, etc. the time in frame units. In other words, the finger wheel 1016 allows for more precise control of the time than the jog wheel 1015.

[0062] Note that the knobs, buttons, switches, etc. on the controllers 202, 203, and 204 other than those described above are not relevant to this embodiment, and therefore their description will be omitted. Also, functions other than those described above, such as depth of field adjustment and focus adjustment, can be assigned. Each embodiment will be described below with reference to Figs. 11 to 16. Embodiment 1

[0063] FIG. 11 is a diagram illustrating the operation axes of the virtual camera. Of the operation axes, X-axis 703, Y-axis 704, and Z-axis 705 correspond to the movement operation directions of the virtual camera. Note that X-axis 703, Y-axis 704, and Z-axis 705 are perpendicular to one another, and Z-axis 705 is always perpendicular to ground 702 regardless of the attitude of virtual camera 701. Therefore, X-axis 703 and Y-axis 704 are always parallel to ground 702 regardless of the attitude of the camera. Furthermore, the directions of X-axis 703 and Y-axis 704 change depending on the attitude of virtual camera 701. The direction of the optical axis of virtual camera 701 projected onto a plane parallel to ground 702 is Y-axis 704, and the direction perpendicular to the optical axis of virtual camera 701 is X-axis 703. Of the operation axes, the Pan axis 706 corresponds to the rotation operation direction about the Z axis 705, the Tilt axis 707 corresponds to the X axis 703, and the Roll axis 708 corresponds to the Y axis 704.

[0064] 11(a) and 11(b) are views of the virtual camera 701 as viewed from the Z-axis direction. When the virtual camera is panned from the state shown in FIG. 11(a) (i.e., rotated around the Z-axis as the axis of rotation), the state shown in FIG. 11(b) is obtained. FIGS. 11(c) and 11(d) are views of the virtual camera 701 as viewed from the Y-axis direction. When the virtual camera is rolled from the state shown in FIG. 11(c) (i.e., rotated around the Y-axis as the axis of rotation), the state shown in FIG. 11(d) is obtained. FIGS. 11(e) and 11(f) are views of the virtual camera 701 as viewed from the X-axis direction. When the virtual camera is tilted from the state shown in FIG. 11(e) (i.e., rotated around the X-axis as the axis of rotation), the state shown in FIG. 11(f) is obtained.

[0065] Note that how each camera parameter changes in response to an operation on each operation axis depends on the setting of slide bar 513 in the pilot window when the camera control tab in FIG. 6 is selected. Also, there is not necessarily a one-to-one correspondence between the operation axes and the camera parameter items. For example, the X-axis of the operation axes is the left-right direction relative to the virtual camera, and the Y-axis is the front-to-back direction relative to the virtual camera. However, the X-axis of the camera parameters may be the long side direction of the ground, and the Y-axis may be the short side direction of the ground. In this case, the X-axis and Y-axis values ​​of the camera parameters after editing are determined by combining the current X-axis and Y-axis values ​​of the camera parameters with the X-axis and Y-axis values ​​of the operation axes. Specifically, for example, if the X-axis is set to Off (uneditable) and the Y-axis is set to On (editable), moving the virtual camera forward or backward will change both the X-axis and Y-axis values ​​of the camera parameters if the movement is diagonal to the ground.

[0066] 12 is a diagram showing functional blocks of the information processing device 103. The input / output unit 120 is a function realized by the input / output unit 115, which is composed of the above-mentioned controllers 202a-204 and display units 201a-201c, and displays a plurality of controllers for controlling the virtual cameras and the status of the virtual cameras, etc.

[0067] The setting unit 121 sets various camera parameters in response to the operation of the controller. For example, this corresponds to various settings in the pilot window when the camera control tab in Fig. 6 is selected, or settings using various buttons on the three-axis controller in Fig. 8. The values ​​set in the setting unit 121 are stored in the setting storage unit 122.

[0068] The camera parameter editing unit 123 edits the camera parameters in response to operations from the user via the input / output unit 120. Specifically, the camera parameter editing unit 123 edits the camera parameters by setting the position and field of view of the virtual camera in each frame from the start of a scene of the video to the end of the scene in response to operations via the input / output unit 120. The camera parameter storage unit 124 stores the camera parameters edited by the camera parameter editing unit 123 in association with the passage of time of the three-dimensional object data.

[0069] The window control unit 125 controls the display of various windows such as the camera view window, pilot window, and replay window. In the camera view window, the window control unit 125 reads out the camera parameters for each frame from the camera parameter storage unit 124 and passes them to the virtual viewpoint image generation server 102. Furthermore, the window control unit 125 displays the virtual viewpoint image generated by the virtual viewpoint image generation server 102 in the camera view window on the display unit 201a of the input / output unit 115. In addition, in the pilot window, the window control unit 125 reads out the camera parameters for each frame from the camera parameter storage unit 124 and draws a virtual camera angle of view display 412 according to the position and attitude of the camera parameters.

[0070] 13 is a flowchart showing the processing procedure in the information processing device 103. In the following description of the flowchart, the symbol "S" represents a step. That is, here, the processing steps S101 to S114 of the flowchart will be abbreviated as S101 to S114.

[0071] In S101, the camera parameter editing unit 123 reads out one camera parameter designated by the operator from among the camera parameters already stored in the camera parameter storage unit 124. Specifically, the replay clip list 604 displayed in the replay window shown in Fig. 7 is displayed, and the operator selects (designates) one replay clip 605 from the list.

[0072] In S102, the camera parameter editing unit 123 sets the start frame of the scene as a processing target. In S103, the camera parameter editing unit 123 reads out the virtual camera parameters in the start frame of the scene. In S104, the window control unit 125 reads out the virtual camera parameters from the camera parameter storage unit 124 and updates the camera view window and the pilot window. In updating the camera view window, the virtual camera parameters are passed to the virtual viewpoint image generation server 102, and the virtual viewpoint image generated by the virtual viewpoint image generation server 102 is displayed in the camera view window (camera view). In updating the pilot window, the virtual camera angle of view display 412 is drawn according to the position and attitude of the camera parameters. Note that the processes from S104 to S114 are processes executed for each frame of the video scene.

[0073] In S105, the camera parameter editing unit 123 acquires operations for each operation axis of the controller. In S106, the camera parameter editing unit 123 processes the first operation axis of the camera parameters. Specifically, for example, when processing is performed in the order of the X axis, Y axis, Z axis, Pan axis, Tilt axis, Roll axis, and Zoom axis, the X axis is first processed.

[0074] In S107, the camera parameter editing unit 123 refers to the information (setting values) stored in the setting storage unit 122 and determines whether the relevant operating axis (the operating axis to be processed) is designated as editable (On). That is, it determines whether editing is possible. If the relevant operating axis is designated as editable (S107 Yes), the information processing device 103 shifts the processing to S108, and if the relevant operating axis is not designated as editable (S107 No), the information processing device 103 shifts the processing to S109. Note that the processing from S107 to S112 is executed for each operating axis.

[0075] In S108, the camera parameter editing unit 123 derives (calculates) the camera parameter value of the operation axis based on the controller operation acquired in S105. Meanwhile, in S109, the camera parameter editing unit 123 reads out the parameter value of the operation axis of the frame of the existing camera parameters specified in S101 from the camera parameter storage unit 124.

[0076] In S110, the camera parameter editing unit 123 stores the value calculated in S108 or the value acquired in S109 in the camera parameter storage unit 124 as the value of the operation axis of the frame of the camera parameters being edited.

[0077] In S111, the camera parameter editing unit 123 determines whether the operation axis (the operation axis to be processed) is the last operation axis in the processing order. If the operation axis is the last operation axis in the processing order (S111 Yes), the information processing device 103 shifts the processing to S113, and if the operation axis is not the last operation axis in the processing order (S111 No), the information processing device 103 shifts the processing to S112.

[0078] In S112, the camera parameter editing unit 123 sets the next operation axis as the processing target, and then returns the process to S 107. Specifically, for example, when processing is performed in the order of X axis, Y axis, Z axis, Pan axis, Tilt axis, Roll axis, and Zoom axis, if the current processing target is the X axis, the Y axis is newly set as the next processing target (operation axis).

[0079] In S113, the camera parameter editing unit 123 determines whether the current frame is the last frame (final frame) of the scene. If it is the last frame, the information processing device 103 ends the processing of the flowchart shown in Fig. 13, and if it is not the last frame, the processing proceeds to S114. In S114, the camera parameter editing unit 123 sets the frame next to the current frame as the frame to be edited, and then returns the processing to S104.

[0080] Fig. 14 is a diagram showing the results of editing camera parameters. In Fig. 14, object 1402 is moving along trajectory 1405, and the operator is moving virtual camera 1401 along trajectory 1404 so as to follow object 1402. It is also assumed here that operation of the X-axis, Y-axis, and Z-axis is assigned to 3-axis controller 202a, operation of the Pan-axis, Tilt-axis, and Roll-axis is assigned to 3-axis controller 202b, and operation of the Zoom-axis is assigned to zoom controller 203.

[0081] In the first operation, as shown in FIG. 6( a), all of the X-axis, Y-axis, Z-axis, Pan-axis, Tilt-axis, Roll-axis, and Zoom-axis are set to an editable state (On). While primarily viewing the context view 408 in the pilot window, the operator operates the three-axis controller 202a with one hand to adjust the values ​​of the X-axis, Y-axis, and Z-axis, accurately moving the virtual camera so that it traces a desired trajectory. At the same time, the operator operates the three-axis controller 202b with the other hand to adjust the values ​​of the Pan-axis, Tilt-axis, and Roll-axis. However, since the values ​​of these operation axes will be accurately adjusted in the second operation, they do not need to be accurately operated (adjusted) in the first operation. Furthermore, adjustment of the Zoom axis using the zoom controller 203 is performed in the second operation and is not performed in the first operation. The camera parameters output in the first operation are displayed as a replay clip 605 in the replay window of FIG. 7.

[0082] In the first operation, as shown in Fig. 14(a), the orientation of the virtual camera 1401 is slightly off from the desired orientation (the orientation toward the object), and therefore the desired composition is not fully obtained. Therefore, in the second operation, the replay clip edit button 606 in the replay window is pressed to start editing the replay clip 605. Also, in the second operation, the X-axis, Y-axis, and Z-axis are set to an uneditable state (Off), as shown in Fig. 6(b).

[0083] In the second operation, the operator operates the three-axis controller 202b with one hand while mainly viewing the camera view 301 in the camera view window, to precisely adjust the values ​​of the Pan, Tilt, and Roll axes. At the same time, the operator operates the zoom controller 203 with the other hand to precisely adjust the value of the Zoom axis. By making these adjustments, the position and orientation of the virtual camera are set to the desired state, as shown in FIG. 14(b). Therefore, the desired composition can be obtained. Note that, if any of the above-mentioned operation axes require further fine adjustment, it is possible to set only that operation axis to an editable state (On state) and edit it again in the same way. Depending on the operation axis to be edited, it is also possible to operate it by checking either the pilot window (position and orientation of the virtual camera) or the camera view window.

[0084] As explained above, by reducing the number of operation axes to be controlled simultaneously and dividing the operation into multiple operations for each operation axis, it is possible to reduce the burden of operation and the number of operators. Furthermore, by operating in this manner, camera parameters can be adjusted accurately and easily for each operation axis. In particular, as described above, by performing operations while viewing the context view and operations while viewing the camera view separately, the operator can concentrate on a specific view, making it easy to perform accurate operation. Furthermore, when fine-tuning camera parameters is desired, there is no need to redo the operations of all operation axes, and only the operation axis to be fine-tuned can be adjusted (changed), thereby reducing the burden of adjustment. Embodiment 2

[0085] Next, the second embodiment will be described with reference to Fig. 15. Fig. 15 is a diagram for explaining the operation axes of the virtual camera. Of the operation axes, X-axis 703 corresponds to the side direction of virtual camera 701, Y-axis 704 corresponds to the front direction (optical axis direction) of virtual camera 701, and Z-axis 705 corresponds to the movement operation direction toward the top surface of virtual camera 701, and X-axis 703, Y-axis 704, and Z-axis 705 are perpendicular to one another. Also, of the operation axes, Pan axis 706 corresponds to the Z-axis 705, Tilt axis 707 corresponds to the X-axis 703, and Roll axis 708 corresponds to the rotation operation direction about Y-axis 704.

[0086] In the above-described first embodiment, when an operation to move the virtual camera forward (i.e., movement in the Y-axis direction) is performed using the three-axis controller 202a, the virtual camera always moves horizontally relative to the ground 702, regardless of the up-down orientation of the virtual camera lens (i.e., Tilt). On the other hand, in this embodiment (second embodiment), when an operation to move the virtual camera forward (i.e., movement in the Y-axis direction) is similarly performed using the three-axis controller 202a, the virtual camera moves in the direction of the virtual camera's optical axis. Therefore, if the virtual camera is pointing downward relative to the horizontal (i.e., the Tilt value is negative), the virtual camera will approach the ground (i.e., the Z value will be smaller). Conversely, if the virtual camera is pointing upward relative to the horizontal (i.e., the Tilt value is positive), the virtual camera will move away from the ground (i.e., the Z value will be larger). Third embodiment

[0087] Next, a third embodiment will be described with reference to Fig. 16. Fig. 16 is a diagram illustrating the operation axes of the virtual camera. Of the operation axes, X-axis 703, Y-axis 704, and Z-axis 705 always correspond to a fixed direction, independent of the position or orientation of the virtual camera. Specifically, for example, in the case of a facility such as a soccer field, X-axis 703 corresponds to the long side direction of the field, Y-axis 704 corresponds to the short side direction of the field, and Z-axis 705 corresponds to the movement operation direction perpendicular to the surface of the field (i.e., each operation axis corresponds to the shape of the facility).

[0088] The X-axis 703, Y-axis 704, and Z-axis 705 are perpendicular to one another. In this case, the Pan-axis 706, Tilt-axis 707, and Roll-axis 708 among the operation axes can correspond to the rotation operation directions with the Z-axis, X-axis, and Y-axis as the rotation axes, as in the first and second embodiments. However, this is not limiting. The Pan-axis 706, Tilt-axis 707, and Roll-axis 708 may be set so that the X-axis 703 and Pan-axis 706, the Y-axis 704 and Tilt-axis 707, and the Z-axis 705 and Roll-axis 708 do not coincide with each other. For example, even if the X-axis 703 is a coordinate axis based on the long side direction of the soccer field and the Y-axis 704 is a coordinate axis based on the short side direction of the soccer field, the rotation axes of the roll and tilt may be set based on the direction of the optical axis of the virtual camera. Fourth Embodiment

[0089] Next, a fourth embodiment will be described. In the above-described embodiments, an example has been described in which replay footage viewed from various viewpoints is provided by editing camera parameters for captured images, but here, an example will be described in which camera parameters are generated in real time while capturing images to provide live footage.

[0090] For example, there are cases where the scenario is predetermined, such as in music videos or theatrical videos. In such cases, a rehearsal is filmed and camera parameters for the rehearsal are created in advance. Then, during the actual performance, the camera parameters for the rehearsal are edited in synchronization with the progress of the scenario, and camera parameters for the actual performance are created (generated). Since there are subtle differences in the movements of the performers between the rehearsal and the actual performance, camera parameters for the actual performance are created while adjusting the direction of the virtual camera (for example, adjusting only the pan, tilt, and roll) accordingly to achieve the desired composition. Furthermore, the footage is broadcast or distributed as live footage (live images). Embodiment 5

[0091] Next, a fifth embodiment will be described. As described above, one of the operation axes is the time axis (more specifically, the time axis that controls the playback speed) of the replay controller 204. When editing camera parameters including virtual camera operations such as slow playback and pause, it may be possible to switch whether or not the playback speed can be adjusted (edited) by the replay controller 204.

[0092] When editing is enabled, the replay speed is adjusted by operating the replay controller 204, regardless of the playback speed of the existing camera parameters, and any operation axis is adjusted by operating the three-axis controller 202 or the zoom controller 203. When editing is disabled, any operation axis is further adjusted in accordance with changes in the playback speed of the existing camera parameters.

[0093] As described above in the first to fifth embodiments, the operational burden can be reduced by improving the degree of freedom in virtual viewpoint operation in generating a virtual viewpoint image. Also, the definition of the operation axis is not necessarily limited to the definition described in the first to fifth embodiments. Other embodiments

[0094] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0095] 103 Information processing equipment 121 Settings 123 Camera Parameter Editorial Department 124 Camera parameter memory unit

Claims

1. An information processing device that controls a virtual camera corresponding to a virtual viewpoint image, a first setting means for performing a first setting in accordance with a user operation to set either a mode that allows a change in the position of the virtual camera along a first axis corresponding to a first movement direction of the virtual camera or a mode that prohibits a change in the position of the virtual camera along the first axis; a second setting means for performing a second setting in accordance with a user operation to set either a mode that allows a change in the position of the virtual camera along a second axis corresponding to a second movement direction different from the first movement direction, or a mode that prohibits a change in the position of the virtual camera along the second axis; a display control means for controlling the display of first information indicating whether a mode allowing a change in position on the first axis or a mode prohibiting a change in position on the first axis is set, and second information indicating whether a mode allowing a change in position on the second axis or a mode prohibiting a change in position on the second axis is set; and The first setting and the second setting can be performed independently of each other; An information processing device characterized by:

2. a third setting means for performing a third setting in accordance with a user operation to set either a mode that allows a change in the position of the virtual camera on a third axis corresponding to a third movement direction different from the first movement direction and the second movement direction, or a mode that prohibits a change in the position of the virtual camera on the third axis; and the display control means controls to display the first information, the second information, and third information indicating whether a mode that allows a change of position on the third axis or a mode that prohibits a change of position on the third axis is set; 2. The information processing device according to claim 1,

3. the first information and the second information are displayed on a display device; 3. The information processing device according to claim 1, wherein:

4. An information processing method for controlling a virtual camera corresponding to a virtual viewpoint image, comprising: a first setting step of performing a first setting in accordance with a user operation to set either a mode that allows a change in position along a first axis corresponding to a first movement direction of the virtual camera or a mode that prohibits a change in position along the first axis; a second setting step of performing a second setting in accordance with a user operation to set either a mode that allows a change in the position of the virtual camera along a second axis corresponding to a second movement direction different from the first movement direction of the virtual camera, or a mode that prohibits a change in the position of the virtual camera along the second axis; a display control step of controlling the display of first information indicating whether a mode allowing a change in position on the first axis or a mode prohibiting a change in position on the first axis is set, and second information indicating whether a mode allowing a change in position on the second axis or a mode prohibiting a change in position on the second axis is set; Including, The first setting and the second setting can be performed independently of each other; An information processing method comprising:

5. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 3.

Images