Image processing device, image processing method, and program

Abstract

The goal is to allow users to easily configure the ideal camera path for virtual viewpoint images. [Solution] Based on controller signals derived from user operations via a controller for controlling a virtual viewpoint in a virtual space, time-series data of virtual viewpoint information is stored, with virtual viewpoint parameters and time points associated with each other. Then, based on the stored time-series data, a range of values ​​to be specified by user operations is set for the virtual viewpoint parameters.

Description

Technical Field

[0001] The present disclosure relates to a technique for controlling a virtual viewpoint for generating a virtual viewpoint image.

Background Art

[0002] There is a technique of installing a plurality of imaging devices at different positions and performing synchronous imaging, and using the plurality of imaging images obtained by the imaging to generate an image (virtual viewpoint image) representing the view from a virtual viewpoint specified by a user's operation. When generating a virtual viewpoint image, the user operates a controller such as a joystick to continuously set the position and orientation of the virtual viewpoint in time series while referring to a generated virtual viewpoint image corresponding to a non-existent camera (virtual camera) that captures the desired subject's pose. The movement path of the virtual viewpoint (virtual camera) thus set is generally called a "camera path".

[0003] In recent years, virtual viewpoint images have been used in sports broadcasts and the like. In such usage scenarios, for example, during rehearsal, the director conveys an image of an ideal camera path to the user (operator) in advance, and the operator practices operating the controller in advance so that the camera path can be realized. Then, during the actual performance, while following the movements of the players and the progress of the game, the operator operates the controller aiming at the instructed ideal camera path to set the position and orientation of the virtual viewpoint. However, it is difficult to reproduce the same camera path course as during rehearsal in the first place, and high operation skills are required. Moreover, for example, the operator or director may not be able to participate in the rehearsal, or the operator or director may change after the rehearsal. Also, the movements of the players and the progress of the game may be different from what was assumed during rehearsal. In such cases, the operator faces the difficulty of operating the controller under a situation where sufficient practice has not been done and aiming to realize the ideal camera path desired by the director. With respect to such problems, a certain solution can be achieved by providing restrictions on the position and orientation of the virtual viewpoint that can be set (see Patent Document 1). [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2007-195091 [Overview of the project] [Problems that the invention aims to solve]

[0005] Patent Document 1 is a technology aimed at preventing the viewer from looking at undesirable areas, such as the back of a stage set, or from setting a virtual viewpoint in an unnatural position, such as below ground level. In other words, the technology in Patent Document 1 only prevents the virtual viewpoint from being placed in an unnatural area or pointed in a direction that should not be seen. Therefore, it was completely insufficient as a technology for achieving an ideal camera path in situations where the operator has not had sufficient training. This disclosure has been made in view of the above, and aims to enable users to easily set an ideal camera path for a virtual viewpoint image. [Means for solving the problem]

[0006] The information processing device according to this disclosure is characterized by comprising: generation means for generating virtual viewpoint information, in which parameters of the virtual viewpoint and time are associated, based on controller signals based on user operations via a controller for controlling a virtual viewpoint in a virtual space; storage means for storing time-series data of the generated virtual viewpoint information; and setting means for setting a range of values ​​to be specified by the user operation for the parameters of the virtual viewpoint, based on the stored time-series data. [Effects of the Invention]

[0007] According to this disclosure, it will be possible for users to easily set an ideal camera path for virtual viewpoint images. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example configuration of an image processing system. [Figure 2] A diagram showing an example of the installation of an imaging device. [Figure 3] A diagram showing the hardware configuration of the control unit. [Figure 4] A functional block diagram showing the software configuration (logical configuration) of the control device according to Embodiment 1. [Figure 5] A flowchart showing the flow of the preparation process for the pre-camera path according to Embodiment 1. [Figure 6] A diagram showing an example of a UI screen. [Figure 7] An explanatory diagram of the pre-camera path according to Embodiment 1. [Figure 8] A flowchart illustrating the process for generating virtual viewpoint information based on a pre-set camera path according to Embodiment 1. [Figure 9] (a) is a diagram showing an example of a movement range set based on a pre-set camera path, and (b) is a diagram showing an example of the changes in the position of the virtual camera specified by the operator superimposed on the movement range on the UI screen. [Figure 10] A diagram illustrating the correction of the virtual viewpoint position. [Figure 11] An explanatory diagram of a pre-camera path according to a modified example of Embodiment 1. [Figure 12] A functional block diagram showing the software configuration (logical configuration) of the control device according to Embodiment 2. [Figure 13] A flowchart showing the flow of the pre-camera path preparation process according to Embodiment 2. [Figure 14] A flowchart illustrating the process for generating virtual viewpoint information based on a pre-set camera path according to Embodiment 2. [Figure 15] A conceptual diagram showing relative positions. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings. However, the following embodiments are not necessarily limiting to this disclosure. Furthermore, not all combinations of features described in each embodiment are essential to the solutions of this disclosure.

[0010] [Embodiment 1] <System Configuration> Figure 1 shows an example of the configuration of an image processing system according to this embodiment. The image processing system 100 shown in Figure 1 has an imaging device group consisting of multiple imaging devices 110, an image generation device 120, a control device 130, and a display 160. The imaging device group, the image generation device 120, and the control device 130 are connected via a communication cable such as a LAN (Local Area Network) cable. The image generation device 120 and the display 160 are connected via a video signal transmission cable. The imaging device 110 is a digital camera capable of capturing still images and moving images. Each imaging device 110 constituting the imaging device group is installed to surround a specific area in a stadium or the like, and synchronously captures subjects (objects) within the area. Multiple captured images from different viewpoints obtained by synchronous imaging (hereinafter referred to as "multi-view images") are transmitted to the image generation device 120.

[0011] The image generation device 120 stores multi-view images obtained by the imaging device group, and when virtual viewpoint information is input from the control device 130, it generates a virtual viewpoint image based on the multi-view images and the virtual viewpoint information. Here, the virtual viewpoint information is linked to a time code and contains information indicating the three-dimensional position (position of the virtual camera) and orientation (pose of the virtual camera) of the virtual viewpoint in the virtual space constructed from the captured images. The time code is information indicating the time when the image was captured, for example, the time when the target match started expressed as 00:00:00:00 frames. By specifying an arbitrary start / end time in the time code, the operator can generate a virtual viewpoint image corresponding to, for example, a highlight scene during a competition. For example, the number of frames per second is set to 60 frames. In addition to the position and orientation of the virtual viewpoint, the virtual viewpoint information may also include other information such as zoom (focal length), point of focus, and distance from one virtual viewpoint to the other.

[0012] The image generation device 120 is, for example, a server device and has a database function and an image processing function. By means of the database function, for example, captured images of scenes in a state where no athlete is present in the arena (such as before the start of a competition) are acquired by the imaging device group and stored as background images. Also, in a scene where an athlete is present after the start of the competition, foreground-background separation processing using the difference from the background image is performed on each captured image constituting the multi-viewpoint image acquired by the imaging device group to generate a foreground image. Note that the objects serving as the foreground include not only people but also, for example, balls. The image generation device 120 uses, for example, model-based rendering (MBR). MBR is a method of generating a virtual viewpoint image using three-dimensional shape data (3D model) of an object appearing in a shooting scene. The 3D model is obtained, for example, by restoring (modeling) the three-dimensional shape such as the volume intersection method, Multi-View-Stereo (MVS). The image generation device 120 arranges a 3D model of a pre-prepared CG (computer graphics) background and a 3D model of the foreground obtained by a method such as the above-mentioned volume intersection method in a virtual three-dimensional space, and generates an image representing the view from the virtual viewpoint indicated by the virtual viewpoint information. Note that the method of generating the virtual viewpoint image may use a rendering method other than MBR, for example, image-based rendering (IBR) that generates a virtual viewpoint image by transforming and synthesizing a plurality of captured images without performing modeling. The generated virtual viewpoint image is transmitted to the display 160 via a video signal transmission cable.

[0013] The control device 130 is an information processing device such as a PC (Personal Computer) or a tablet. The viewpoint controller 131 is an operation device for specifying the position and orientation of the virtual viewpoint (the position and posture of the virtual camera), and is composed of, for example, a 6-axis joystick, a mouse, a keyboard, a touch panel, etc. Further, the viewpoint controller 131 includes, for example, a turntable for specifying an arbitrary time by the above-mentioned time code. The control device 130 converts the controller signal input according to the user operation of the viewpoint controller 131 into the above-mentioned virtual viewpoint information and transmits it to the image generation device 120. When specifying the position and orientation of the virtual viewpoint, it is also possible to specify an instantaneous movement to a preset position such as the front position, the back position, or the position looking down from above of the object of interest in the virtual space. It is also possible to specify an instantaneous movement to that time by presetting a specific time in advance. The control device 130 can store the time-series data of the virtual viewpoint information thus set in the storage device as information (camera path) indicating the movement path of the virtual camera, or read out and use the stored camera path.

[0014] <Description of the imaging space> FIG. 2 is a diagram showing a state in which a plurality of imaging devices 110 are installed on a bank 101 which is a bicycle track arena according to the present embodiment. The bank 101 is a circular track composed of a straight section and a curved section. The bank 101 is angled to the track so that a bicycle can travel at high speed, and is designed to have a maximum angle in the curved section and a gradually looser angle over the straight section. In the example of FIG. 2, a plurality of imaging devices 110 are installed so as to surround the bank 101. <000009​​​Figure 3 shows the hardware configuration of the control device 130. The control device 130 includes a CPU 301, a ROM 302, a RAM 303, an HDD 304, a display unit 305, an input unit 306, and a communication unit 307. The CPU 301 reads the control program stored in the ROM 302 and executes various processes. The RAM 303 is used as the main memory and temporary storage area for the CPU 301, such as a work area. The HDD 304 stores various data and programs. The display unit 305 displays various information. The input unit 306 can be connected to a keyboard, mouse, 6-axis controller, etc., and accepts various operations from the user. The communication unit 307 performs communication processing with external devices via a network. Ethernet (registered trademark) is one example of such a network. Alternatively, the communication unit 307 may communicate with external devices wirelessly.

[0016] The various functions of the control device 130 are realized by the CPU 301 reading a program stored in the ROM 302 or HDD 304 and executing this program. The hardware configuration of the image generation device 120 is the same as that of the control device 130.

[0017] <Control device software configuration> Figure 4 is a functional block diagram showing the software configuration (logical configuration) of the control device 130 according to this embodiment. The control device 130 includes a controller signal acquisition unit 132, a virtual viewpoint information generation unit 133, an instruction reception unit 134, a camera path storage unit 135, a transmission unit 136, and a range setting unit 137.

[0018] The controller signal acquisition unit 132 acquires the controller signals output by the viewpoint controller 131. These controller signals include signals indicating the amount of change in the position and orientation of the virtual camera according to the amount of operation of a joystick, etc., and signals indicating the time of a desired moment according to the amount of operation of a rotating disc, etc. The controller signals input from the viewpoint controller 131 are output to the virtual viewpoint information generation unit 133.

[0019] The virtual viewpoint information generation unit 133 determines the position and orientation of the virtual viewpoint that will serve as the reference for rendering by the image generation device 120 based on the input controller signal, and generates virtual viewpoint information linked to the time code of the corresponding time. The virtual viewpoint information in this embodiment includes parameters such as zoom (focal length) and time, in addition to the position and orientation of the virtual viewpoint (position and orientation of the virtual camera). In this case, the position of the virtual camera is indicated by three-dimensional coordinate information according to a three-axis orthogonal coordinate system of the X, Y, and Z axes. The origin in this case is an arbitrary position in the three-dimensional space of the image target (for example, the center of link 101). The orientation of the virtual camera (orientation of the virtual viewpoint) is indicated by the angles it makes with the three axes of pan, tilt, and roll. Pan is the angle of rotation parallel to the ground surface, tilt is the angle of rotation perpendicular to the ground surface, and roll is the angle of rotation around the optical axis of the virtual camera. In this embodiment, the unit of angle is degrees, and the range of values ​​is from -180 to +180 degrees. Furthermore, the time is indicated by a time code. The time code is expressed, for example, as [hours;minutes;seconds;frames]. The range of the frame number is, for example, 0 to 60. Zoom and time are each one-axis parameters. In other words, the virtual viewpoint at a given moment is identified by [X, Y, Z] (unit [m]), [Pan, Tilt, Roll] (unit [degrees]), [Zoom] (unit [mm]), and the time code [HH.MM.SS.FF]. The controller signal input from the viewpoint controller 131 represents the change in the position, orientation, and zoom of the virtual viewpoint between frames [ΔX, ΔY, ΔZ, ΔPan, ΔTilt, ΔRoll, ΔZoom]. Note that parameters other than the above 8 axes may be included, or not all of the above 8 axis parameters may be included. The virtual viewpoint information thus generated is output to the transmission unit 136.

[0020] Furthermore, if the position of the virtual viewpoint, which is identified by the input controller signal, is outside the movement range of the virtual camera set by the range setting unit 137 (described later), the virtual viewpoint information generation unit 133 of this embodiment performs a correction process to bring it within the range.

[0021] The instruction receiving unit 134 receives various instructions from the user via a keyboard, mouse, or other device (not shown) connected to the input unit 306. In this embodiment, it receives instructions to start / stop recording virtual viewpoint information, instructions to set the movement range of the virtual camera, and instructions to select a generated camera path to refer to when setting it.

[0022] In response to the instruction receiving unit 134 receiving a recording start instruction, the camera path storage unit 135 begins recording the virtual viewpoint information generated by the virtual viewpoint information generation unit 133 to the HDD 304. Furthermore, in response to the instruction receiving unit 134 receiving a recording end instruction, the camera path storage unit 135 terminates the recording of the virtual viewpoint information generated by the virtual viewpoint information generation unit 133 to the HDD 304. The resulting collection of chronologically continuous virtual viewpoint information is then stored as a camera path.

[0023] The transmission unit 136 transmits the virtual viewpoint information input from the virtual viewpoint information generation unit 133 to the image generation device 120 via the communication unit 307.

[0024] The range setting unit 137, in response to the instruction receiving unit 134 receiving an instruction to set the movement range of the virtual camera, sets the movement range of the virtual camera that can be operated by the operator based on a pre-generated and saved camera path (hereinafter referred to as the "pre-camera path"). The information of the set virtual camera movement range is output to the virtual viewpoint information generation unit 133.

[0025] Each of the functions described above is realized by the CPU 301 reading a program stored in the ROM 302 or HDD 304, loading it into the RAM 303, and executing it. Although it has been explained here that the CPU 301 performs the execution, it is not limited to this. For example, if the control device has hardware such as an ASIC, DSP, or FPGA in addition to the CPU 301, the ASIC or DSP may perform some or all of the processing that the CPU 301 would normally do.

[0026] <Preparing your camera pass in advance> Next, the pre-camera pass preparation process in the control device 130 according to this embodiment will be described with reference to the flowchart in Figure 5. The series of processes shown in the flowchart in Figure 5 are executed, for example, during rehearsals. Each step is executed every 1 / 60th of a second, for example, if the number of frames is 60 frames per second. In the following description, the symbol "S" means step.

[0027] In S501, the next process to be executed is determined by whether or not a new controller signal from the viewpoint controller 131 has been input to the controller signal acquisition unit 132. For example, the operator operates the viewpoint controller 131 while referring to the virtual viewpoint image of the previous frame displayed on the display 160. Then, the operator specifies the frame to be processed (current frame) and the position and orientation of the virtual camera. A controller signal corresponding to the operator's operation is then input to the signal acquisition unit 132. The virtual viewpoint image of the previous frame may also be captured on the user interface screen (UI screen) shown in Figure 6 below, which is displayed on the display unit 305. If a new controller signal corresponding to the operator's operation has been input, S502 is executed next. On the other hand, if no new controller signal has been input, S502 is skipped.

[0028] In S502, the virtual viewpoint information generation unit 133 determines the virtual viewpoint information for the current frame based on the input controller signal. In the following S503, the next process to be executed is determined based on whether or not the process of recording virtual viewpoint information is ongoing. If the process of recording virtual viewpoint information is ongoing, S507 is executed next; if the recording process is not ongoing, S507 is executed next.

[0029] In S504, the next process to be executed is determined by whether or not the instruction receiving unit 134 has received an instruction to start the process of recording virtual viewpoint information. Figure 6 is an example of a UI screen displayed on the display unit 305. In the UI screen 601 of Figure 6, button 602 is a button for instructing the start of recording virtual viewpoint information, and button 603 is a button for instructing the end of recording virtual viewpoint information. Button 604 is a button for instructing the setting of the range in which the virtual camera can be moved. In addition, the UI screen 601 displays a time code representing the time of the current frame in field 605. The operator instructs the start of recording by pressing button 602. If an instruction to start recording virtual viewpoint information is received via such a UI screen, S505 is executed next. On the other hand, if an instruction to start recording virtual viewpoint information is not received, S505 is skipped, and S506 is executed next.

[0030] In S505, the camera path storage unit 135 records the virtual viewpoint information of the current frame, determined in S502, into the HDD 304. At this time, for example, all of the values ​​of the eight axes parameters mentioned above may be recorded, or only some of the parameters necessary as components of the pre-camera path (for example, the position and orientation of the virtual camera and the time code) may be recorded.

[0031] In S506, the transmitter 136 transmits the virtual viewpoint information determined in S502 to the image generator 120. If no new controller signal is input and S502 is skipped (No in S501), then the virtual viewpoint information determined in the most recent previous frame should be transmitted.

[0032] In S507, the next process to be executed is determined by whether or not the instruction receiving unit 134 has received an instruction to end the process of recording virtual viewpoint information. If the instruction to end the recording of virtual viewpoint information is received, for example by detecting the pressing of button 603 via the aforementioned UI screen 601, then S508 is executed. On the other hand, if the instruction to end the recording process of virtual viewpoint information has not been received, then S505 is executed. That is, the camera path storage unit 135 records the virtual viewpoint information determined in S502 to the HDD 304.

[0033] In S508, the camera path storage unit 135 finishes the process of recording virtual viewpoint information and saves the set of virtual viewpoint information recorded in a time series from the start instruction to the end instruction as a pre-camera path. Figure 7 is an explanatory diagram of the pre-camera path according to this embodiment. In Figure 7, the dashed arrow 701 shows the trajectory (movement path) of the position of the virtual camera 111 in the virtual space. In Figure 7, the starting point of the arrow 701 corresponds to the start time (start frame) of the virtual viewpoint, and the tip (end point) of the arrow 701 corresponds to the end time (end frame) of the virtual viewpoint. The time codes indicating the start and end times of the saved pre-camera path are displayed in the clip list 606 of the UI screen 601 in Figure 6. The operator can select any pre-camera path from the clip list 606 using checkboxes. The operator can also control the playback time of the selected pre-camera path using the adjustment bar 607.

[0034] The above describes the preparation process for the pre-camera path according to this embodiment. Note that the time-series data of virtual viewpoint information saved as a pre-camera path may have frames thinned out. That is, the frames in the time-series data may be discontinuous and may consist of virtual viewpoint information corresponding to so-called keyframes.

[0035] <Generation of virtual viewpoint information based on pre-set camera paths> Next, the process of generating virtual viewpoint information based on a pre-set camera path in the control device 130 according to this embodiment will be explained with reference to the flowchart in Figure 8. The series of processes shown in the flowchart in Figure 8 are executed in the actual game, for example, after the flow in Figure 5 described above has been performed in a rehearsal. Each step of the process is executed every 1 / 60th of a second, for example, if the number of frames is 60 frames per second. In the following explanation, the symbol "S" means step.

[0036] In S801, the instruction receiving unit 134 receives instructions to set the movement range of the virtual camera. In this embodiment, the operator makes this setting instruction by checking the checkbox of the pre-set camera path to be referenced from the clip list 606 on the UI screen 601 in Figure 6, and pressing the button 604.

[0037] In S802, the range setting unit 137 first reads and acquires time-series data from the HDD 304 as the selected pre-camera path related to the setting instruction. Then, the range setting unit 137 sets the movement range of the virtual camera based on the acquired pre-camera path. The movement range of the virtual camera set here is a continuous range that extends in three dimensions with the movement path indicated by the acquired pre-camera path as the central axis. Figure 9(a) shows a tubular movement range 901 set based on the pre-camera path 701 shown in Figure 7. The central axis of the tubular movement range 901 is the movement path indicated by the pre-camera path 701. Here, the diameter of the tubular movement range 901 is a predetermined size (for example, 1.0 [m]). The cross-sectional shape of the tubular movement range is arbitrary and may be a polygon, for example. The movement range only needs to be a continuous range that extends in three dimensions based on the pre-camera path, and may be a cylinder or a rectangular parallelepiped, for example. The information of the set movement range is output to the virtual viewpoint information generation unit 133.

[0038] S803 is the same process as S501 described above. That is, the next process to be executed is determined by whether or not the controller signal acquisition unit 132 has acquired a new controller signal from the viewpoint controller 131. If a new controller signal is input in response to the operator's operation, S804 is executed next. On the other hand, if no new controller signal is input, S804 is skipped.

[0039] S804 is the same process as S502 described above. That is, the virtual viewpoint information generation unit 133 determines the virtual viewpoint information for the current frame based on the input controller signal. In the next step, S805, the virtual viewpoint information generation unit 133 determines the next process to be executed depending on whether the position of the virtual camera in the virtual viewpoint information determined in S804 is inside the movement range determined in S802. If the determined position of the virtual camera is outside the movement range, S806 is executed next; if it is inside the movement range, S806 is skipped.

[0040] In S806, the virtual viewpoint information generation unit 133 changes the position of the virtual camera in the virtual viewpoint information determined in S804 so that it falls within the movement range set in S802. Figure 10 illustrates how the position of the virtual camera based on the controller signal is changed to fall within the movement range because it has fallen outside the tubular movement range shown in Figure 9(a). In Figure 10, the dashed line 901' shows a part of the tubular movement range 901 in Figure 9(a). The solid arrow 1001 shows the movement path of the virtual camera according to the input controller signal, and the dashed arrow 1002 shows the movement path of the virtual camera whose position has been changed in this step. The example in Figure 10 shows how, at a certain time, the height of the virtual camera corresponding to the controller signal is determined to be greater than the upper limit in the Z-axis direction (e.g., 1.5m) shown by the movement range 901', and how it is changed to fall within the upper limit. Thus, even if the virtual camera's height reaches the upper limit of the set movement range in the Z-axis direction, and a controller signal is input to move it further in the positive Z-axis direction (upward), the virtual camera's height is corrected so as not to exceed the upper limit. Although the explanation uses the height direction (Z-axis direction) as an example, the same process is performed for the X-axis and Y-axis as well.

[0041] S807 is the same process as S506 described above. That is, the transmission unit 136 transmits the virtual viewpoint information determined in S804 (or the virtual viewpoint information after the virtual camera position has been corrected in S806) to the image generation device 120.

[0042] In S808, the instruction receiving unit 134 determines whether or not it has received an instruction to terminate the virtual viewpoint image generation process. If the operator presses an exit button (not shown) or the like to receive an instruction to terminate generation, this process is terminated. On the other hand, if no instruction to terminate generation has been received, the process returns to S803 and continues. The above describes the content of the virtual viewpoint information generation process based on the pre-set camera path according to this embodiment.

[0043] <Example 1> In the above embodiment, an example was described in which the virtual camera's position is corrected so that it does not go outside the set range by setting a range within which the virtual camera's position can be moved based on a pre-set camera path. However, the correction is not limited to the virtual camera's position. For example, the pose of the virtual camera (direction of the virtual viewpoint) can also be corrected by setting a range that the operator can specify based on a pre-set camera path, so that it falls within the set range. For example, suppose that in the pre-set camera path, the value range for the Pan axis is -30 to 30, the value range for the Tilt axis is -45 to 45, and the value range for the Roll axis is 0 to 5. In this case, the range of the virtual camera's pose that the operator can control with the viewpoint controller 131 is set with a margin, for example, -35 to 35 for the Pan axis, -50 to 50 for the Tilt axis, and -5 to 10 for the Roll axis. Then, if the values ​​of the Pan axis, Tilt axis, and Roll axis of the virtual camera, which are obtained by operating the viewpoint controller, fall outside the pre-set range, they should be corrected to values ​​within the range.

[0044] <Modification 2> The system may notify the operator with a warning sound or message when the virtual camera's position and orientation, as controlled by the viewpoint controller, moves outside the set range (or is about to move outside the range). Alternatively, instead of correcting the virtual camera's position and orientation to prevent it from moving outside the range, the system may control the virtual camera's position and orientation so that the change is not smooth after it moves outside the range, making the operator aware that the current position and orientation of the virtual camera is inappropriate. For example, information indicating the controllable range of the virtual camera's position and orientation, based on a pre-set camera path, may be displayed on the UI screen when the operator operates the virtual camera. Furthermore, the system may overlay the changes in the virtual camera's position, as specified by the operator, onto the movement range set based on the pre-set camera path. Figure 9(b) is an example of this, showing the changes in the position of the virtual camera 111, as actually specified by the operator, with respect to the movement range 901 shown in Figure 9(a), indicated by the solid line 902. This allows the operator to concretely visualize the ideal camera path in the target scene, making it easier to reproduce the course taken during rehearsal. This modified version makes it possible to guide the operator's camera work to prioritize operations within the movement range based on the pre-defined camera path.

[0045] <Variation 3> To facilitate the reproduction of the course taken during rehearsal (when creating the pre-camera path), control of the virtual viewpoint may be initiated by referencing the positional information of the object at the start and end times of the pre-camera path. Specifically, control of Embodiment 1, including the modified version described above, is initiated when the object of interest reaches the same position as the position at the start time of the pre-camera path, based on image analysis such as object tracking. Then, control is terminated when the object of interest reaches the same position as the position at the end time of the pre-camera path. Note that this modified version is effective when the start and end positions of the pre-camera path are far apart. In other words, for example, in the case of a track where the object of interest (cyclist) circles the same place, such as in a keirin (bicycle racing) track, control like this modified version is unnecessary by creating a pre-camera path for one lap.

[0046] <Modification 4> In the embodiment described above, the movement range was set based on one pre-camera path, but the movement range may also be set based on multiple pre-camera paths. Figure 11 shows a tubular movement range 1102 set based on three pre-camera paths 1101a to 1101c. The movement range 1102 shown in Figure 11 can be obtained by deriving the movement range for each of the three pre-camera paths 1101a to 1101c using the method described in Embodiment 1, and then combining the three resulting movement ranges. When combining, the multiple movement ranges to be combined may be simply combined into one, or a movement range of sufficient size to encompass all of the multiple movement ranges may be reconstructed. Alternatively, a priority may be assigned to each of the multiple movement ranges to be combined, and the movement range with the highest priority may be combined to have the largest volume ratio. This makes it possible to realize an ideal camera path with flexible camera work that follows the actual flow of the game.

[0047] As described above, this embodiment, including the various modifications, allows for the operation of a virtual camera without deviating significantly from an ideal camera path by setting the movable range of the virtual camera based on the camera path obtained during prior practice such as rehearsals.

[0048] [Embodiment 2] Next, Embodiment 2 describes an embodiment in which the movement range of the virtual viewpoint is set based on the relative positional relationship with the object of interest. Note that explanations of aspects common to Embodiment 1 will be omitted, and the following explanation will focus on the differences.

[0049] <Control device software configuration> Figure 12 is a functional block diagram showing the software configuration (logical configuration) of the control device 130 according to this embodiment. The control device 130 includes a controller signal acquisition unit 132, a virtual viewpoint information generation unit 133, an instruction reception unit 134, a camera path storage unit 135, a transmission unit 136, a range setting unit 137, and a 3D model acquisition unit 138.

[0050] The 3D model acquisition unit 138 acquires a 3D model of the object of interest corresponding to the time (current frame) indicated by the controller signal from the image generation device 120, based on the instruction reception unit 134. The acquired 3D model is output to the camera path storage unit 135 and the range setting unit 137.

[0051] In this embodiment, the camera path storage unit 135 records the input 3D model, linking it to the virtual viewpoint information for the corresponding time. The range setting unit 137 of this embodiment uses the pre-set camera paths, each linked to a 3D model at a given time and stored by the camera path storage unit 135, to calculate the relative positional relationship between the 3D model of the object of interest and the pre-set camera path in virtual space. Details of the relative positional relationship will be described later. Based on the calculated relative positional relationship information, the range setting unit 137 of this embodiment sets the movement range of the virtual camera.

[0052] <Preparing your camera pass in advance> Next, the pre-camera path preparation process in the control device 130 according to this embodiment will be explained with reference to the flowchart in Figure 13. The following explanation will focus on the differences from the flowchart in Figure 5 relating to Embodiment 1.

[0053] Steps S501 to S504 are the same as in Embodiment 1, so their explanation will be omitted. If it is determined in S504 that a recording process start instruction has been received, then S1301 is executed. On the other hand, if a recording process start instruction has not been received, then S506 is executed.

[0054] In S1301, the 3D model acquisition unit 138 requests and acquires a 3D model of the object of interest corresponding to the current frame indicated by the input controller signal from the image generation device 120. The acquired 3D model is output to the camera path storage unit 135.

[0055] In S1302, the camera path storage unit 135 links the virtual viewpoint information of the current frame determined in S502 with the 3D model of the current frame acquired in S1301 and records it in the HDD 304. The following steps S506 to S508 are the same as in Embodiment 1, so their explanation is omitted. The above describes the preparation process for the pre-camera path according to this embodiment.

[0056] <Generation of virtual viewpoint information based on pre-set camera paths> Next, the process for generating virtual viewpoint information based on a pre-set camera path in the control device 130 according to this embodiment will be explained with reference to the flowchart in Figure 14. The following explanation will focus on the differences from the flowchart in Figure 8 relating to Embodiment 1.

[0057] S801 is the same as in Embodiment 1, so its explanation is omitted. In the following S1401, the virtual viewpoint information generation unit 133 first reads and acquires the pre-camera paths selected during the setting instruction, to which the 3D models at each time point are linked, from the HDD 304. Then, based on the acquired pre-camera paths, the virtual viewpoint information generation unit 133 calculates the relative position in virtual space between the 3D model of the object of interest and the pre-camera paths. Figure 15 is a conceptual diagram showing the relative position. Now, let (x1, y1, z1) be the three-dimensional position (centroid position) of the 3D model at a specific time Ta in the pre-camera path, and let (x2, y2, z2) be the position of the virtual camera 111 at the same time. At this time, the relative position (Xr, Yr, Zr) indicated by the thick arrow is expressed by the following equation (1).

[0058] (Xr,Yr,Zr)=(x2-x1,y2-y1,z2-z1)...Equation (1) The specific time Ta mentioned above can be, for example, the start time or end time of recording the pre-camera pass.

[0059] S803 and S804 are the same as in Embodiment 1, so their explanation is omitted. In S1402, the 3D model acquisition unit 138 requests the image generation device 120 to acquire a 3D model of the object of interest corresponding to the current frame indicated by the input controller signal. The acquired 3D model is output to the range setting unit 137.

[0060] In S1403, the range setting unit 137 sets the movement range of the virtual camera based on the relative position calculated in S1401 and the position of the 3D model of the current frame acquired in S1402. For example, suppose the centroid position of the 3D model of the object of interest at time Tb when the 3D model was generated was (x3, y3, z3). In this case, the position (x2', y2', z2') of the virtual camera 111 at the start time of recording the pre-camera pass after shifting to the relative position (Xr, Yr, Zr) is expressed by the following equation (2).

[0061] (x2',y2',z2')=(x3+Xr,y3+Yr,z3+Zr)...Equation (2) Then, the movement range is set based on the pre-camera path starting from the position (x2', y2', z2') of the virtual camera 111 after the shift. In other words, the movement range is set as a continuous range that extends in three dimensions with the movement path of the virtual camera indicated by the pre-camera path after the shift as its centroid.

[0062] The following steps S805 to S808 are the same as in Embodiment 1, so their explanation will be omitted. The above describes the process for generating virtual viewpoint information based on the pre-set camera path according to this embodiment.

[0063] Furthermore, the details of each modification described in Embodiment 1 are also applicable to this embodiment. In addition, regarding the generation of virtual viewpoint information based on a pre-set camera path, as described in Embodiments 1 and 2, for example, an operating mode may be provided so that the operator can switch the operation on or off at any time. Also, although Embodiments 1 and 2 were described assuming manual operation by an operator, they may also be applied to automated operation.

[0064] As described above, according to this embodiment, by setting the movable range of the virtual camera according to the actual position of the object during the actual game, it becomes possible to operate the virtual viewpoint without deviating significantly from the camera path used during pre-game practice, even if the game unfolds differently than expected.

[0065] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0066] Furthermore, this disclosure includes the following configurations and methods.

[0067] [Configuration 1] A generation means that generates virtual viewpoint information, which associates the parameters of the virtual viewpoint with the time, based on controller signals based on user operations via a controller for controlling a virtual viewpoint in a virtual space, A storage means for storing the time-series data of the generated virtual viewpoint information, A setting means that sets a range of values ​​for the parameters of the virtual viewpoint to be specified by the user operation, based on the saved time-series data, An information processing device characterized by having the following features.

[0068] [Configuration 2] When the range is set by the setting means, The information processing apparatus according to Configuration 1, characterized in that the generation means generates the virtual viewpoint information by correcting the value of the virtual viewpoint parameter so that it falls within the range when the value of the virtual viewpoint parameter indicated by the controller signal does not fall within the range.

[0069] [Configuration 3] The information processing device according to configuration 1, further comprising a notification means for notifying the user of a warning when the value of the virtual viewpoint parameter indicated by the controller signal does not fall within the range set by the setting means.

[0070] [Structure 4] The information processing apparatus according to any one of configurations 1 to 3, characterized in that the setting means sets the range based on a plurality of stored time-series data.

[0071] [Composition 5] The system further includes a display means for displaying a user interface screen for user operation, The user interface screen includes information indicating the range set by the setting means, An information processing device according to any one of configurations 1 to 4, characterized by the above.

[0072] [Composition 6] The information processing device according to configuration 5, characterized in that the user interface screen displays the changes in the position of the virtual viewpoint actually specified by the user overlaid on the range.

[0073] [Composition 7] The information processing device according to any one of configurations 1 to 6, characterized in that the parameter is a parameter that defines the position of a virtual viewpoint.

[0074] [Structure 8] The information processing device according to configuration 7, wherein the setting means sets a continuous range that extends in a three-dimensional direction with the movement path of the virtual viewpoint position indicated by the stored time-series data as the central axis.

[0075] [Composition 9] The information processing apparatus according to configuration 8, wherein, when setting the continuous range based on a plurality of stored time-series data, the setting means derives the continuous range from each of the plurality of time-series data and sets the range by combining the derived plurality of continuous ranges.

[0076] [Configuration 10] The storage means stores the time-series data and the three-dimensional shape data of the object of interest in a linked manner. The information processing apparatus according to configuration 8 or 9, characterized in that the setting means sets a continuous range in which the relative positional relationship with the object of interest is maintained, based on the stored time-series data and three-dimensional shape data and the three-dimensional shape data of the object of interest in the current frame.

[0077] [Composition 11] The information processing device according to any one of configurations 1 to 5, characterized in that the parameter is a parameter that defines the orientation of a virtual viewpoint.

[0078] [Composition 12] The information processing device according to configuration 11, wherein the setting means sets a range with a margin for each of the value ranges for Pan, Tilt, and Roll, which identify the orientation of the virtual viewpoint indicated by the stored time series data.

[0079] [Composition 13] An acquisition means for acquiring controller signals based on user operations via a controller for controlling a virtual viewpoint in a virtual space, A generation means that generates virtual viewpoint information in which the parameters of the virtual viewpoint and the time are associated based on the acquired controller signal, Display means for displaying the user interface screen for user operation, It has, The user interface screen includes information based on pre-generated time-series data of the virtual viewpoint information, which indicates the range of values ​​that should be specified by the user operation for the parameters of the virtual viewpoint. An information processing device characterized by the following:

[0080] [Composition 14] The information processing device according to configuration 13, characterized in that the user interface screen displays the changes in the position of the virtual viewpoint actually specified by the user overlaid on the range.

[0081] [Method 1] A generation step that generates virtual viewpoint information, which associates the parameters of the virtual viewpoint with the time, based on controller signals based on user operations via a controller for controlling a virtual viewpoint in a virtual space, A saving step of saving the time-series data of the generated virtual viewpoint information, A setting step of setting a range of values ​​to be specified by the user operation for the parameters of the virtual viewpoint based on the saved time-series data, An information processing method characterized by including

[0082] [Method 2] An acquisition step to acquire controller signals based on user operations via a controller for controlling a virtual viewpoint in a virtual space, A generation step that generates virtual viewpoint information in which the parameters of the virtual viewpoint and the time are associated based on the acquired controller signal, A display step of displaying the user interface screen for user operation on a display means, Includes, The user interface screen includes information based on pre-generated time-series data of the virtual viewpoint information, which indicates the range of values ​​that should be specified by the user operation for the parameters of the virtual viewpoint. An information processing method characterized by the following:

[0083] [Composition 17] A program that causes a computer to execute the information processing method described in Method 1.

[0084] [Composition 18] A program that causes a computer to perform the information processing method described in Method 2.

Claims

1. A generation means that generates virtual viewpoint information, which associates the parameters of the virtual viewpoint with the time, based on controller signals based on user operations via a controller for controlling a virtual viewpoint in a virtual space, A storage means for storing the time-series data of the generated virtual viewpoint information, A setting means that sets a range of values ​​for the parameters of the virtual viewpoint to be specified by the user operation, based on the saved time-series data, An information processing device characterized by having the following features.

2. When the range is set by the setting means, The information processing apparatus according to claim 1, characterized in that the generation means generates the virtual viewpoint information by correcting the value of the virtual viewpoint parameter so that it falls within the range when the value of the virtual viewpoint parameter indicated by the controller signal does not fall within the range.

3. The information processing apparatus according to claim 1, further comprising a notification means for notifying the user of a warning when the value of the virtual viewpoint parameter indicated by the controller signal does not fall within the range set by the setting means.

4. The information processing apparatus according to any one of claims 1 to 3, characterized in that the setting means sets the range based on a plurality of stored time-series data.

5. The system further includes a display means for displaying a user interface screen for user operation, The user interface screen includes information indicating the range set by the setting means, The information processing apparatus according to feature 1.

6. The information processing device according to claim 5, characterized in that the user interface screen displays the changes in the position of the virtual viewpoint actually specified by the user overlaid on the range.

7. The information processing apparatus according to any one of claims 1 to 3, characterized in that the parameter is a parameter that defines the position of a virtual viewpoint.

8. The information processing apparatus according to claim 7, wherein the setting means sets a continuous range that extends in a three-dimensional direction with the movement path of the virtual viewpoint position indicated by the stored time-series data as the central axis.

9. The information processing apparatus according to claim 8, wherein, when setting the continuous range based on a plurality of stored time-series data, the setting means derives the continuous range from each of the plurality of time-series data and sets the range by combining the derived plurality of continuous ranges.

10. The storage means stores the time-series data and the three-dimensional shape data of the object of interest in a linked manner. The information processing apparatus according to claim 8, characterized in that the setting means sets a continuous range in which the relative positional relationship with the object of interest is maintained, based on the stored time-series data and three-dimensional shape data and the three-dimensional shape data of the object of interest in the current frame.

11. The information processing apparatus according to any one of claims 1 to 3, characterized in that the parameter is a parameter that defines the orientation of the virtual viewpoint.

12. The information processing apparatus according to claim 11, wherein the setting means sets a range with a margin for each of the value ranges of Pan, Tilt, and Roll, which identify the orientation of the virtual viewpoint indicated by the stored time series data.

13. An acquisition means for acquiring controller signals based on user operations via a controller for controlling a virtual viewpoint in a virtual space, A generation means that generates virtual viewpoint information in which the parameters of the virtual viewpoint and the time are associated based on the acquired controller signal, Display means for displaying the user interface screen for user operation, It has, The user interface screen includes information based on pre-generated time-series data of the virtual viewpoint information, which indicates the range of values ​​that should be specified by the user operation for the parameters of the virtual viewpoint. An information processing device characterized by the following:

14. The information processing device according to claim 13, characterized in that the user interface screen displays the changes in the position of the virtual viewpoint actually specified by the user overlaid on the range.

15. A generation step that generates virtual viewpoint information, which associates the parameters of the virtual viewpoint with the time, based on controller signals based on user operations via a controller for controlling a virtual viewpoint in a virtual space, A saving step of saving the time-series data of the generated virtual viewpoint information, A setting step of setting a range of values ​​to be specified by the user operation for the parameters of the virtual viewpoint based on the saved time-series data, An information processing method characterized by including

16. An acquisition step to acquire controller signals based on user operations via a controller for controlling a virtual viewpoint in a virtual space, A generation step that generates virtual viewpoint information in which the parameters of the virtual viewpoint and the time are associated based on the acquired controller signal, A display step of displaying the user interface screen for user operation on a display means, Includes, The user interface screen includes information based on pre-generated time-series data of the virtual viewpoint information, which indicates the range of values ​​that should be specified by the user operation for the parameters of the virtual viewpoint. An information processing method characterized by the following:

17. A program for causing a computer to execute the information processing method described in claim 15.

18. A program for causing a computer to execute the information processing method described in claim 16.

Images