Information processing device control method and program

Abstract

To provide an information processing system for generating an appropriate camera path.SOLUTION: In an image processing system consisting of an imaging system, an image processing device and an information processing device: the information processing device is started by being triggered by user's depression of a key of a processing flow keyboard for generating a camera path for a clip video; an operation information acquisition unit transmits a signal to generate the camera path to a clip creation unit; the clip creation unit receives the signal, and acquires key-frames from a key-frame holding unit 504; the clip creation unit generates the camera path by performing interpolation between the key-frames in order of respective key-frame IDs; a clip holding unit acquires the camera path from the clip creation unit, and adds the acquired camera path to a clip list; and a key-frame holding unit clears the held key-frame list in accordance with the addition of the generated camera path to the clip list.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device that generates a camera path in a virtual viewpoint image. [Background technology]

[0002] In recent years, there has been an image processing system that can generate an image representing a view from a virtual viewpoint specified by a user from multiple images obtained by photographing a subject using multiple cameras positioned at different locations. Hereinafter, an image representing a view from a virtual viewpoint specified by a user is referred to as a virtual viewpoint image. Such an image processing system estimates the shape of a photographed subject, such as a person, to create a subject model and generates a virtual viewpoint image from which the viewpoint can be changed arbitrarily. It is also possible to generate virtual viewpoint images by continuously generating virtual viewpoint images while changing the position and direction of a virtual camera.

[0003] Such virtual viewpoint images are used to create a more realistic visual representation, particularly in live broadcasts of competitive sports (so-called sports coverage). In sports coverage, clipped video footage using virtual viewpoint images is sometimes generated as a replay of memorable scenes from the game. One method for generating a camera path to generate clipped video footage using virtual viewpoint images is the keyframe method. This method involves the user registering multiple virtual camera parameters for any time, position, and orientation as keyframes, and generating a camera path by interpolating these using a predetermined method.

[0004] Patent Document 1 describes generating a camera path by interpolating between designated key frames using spline interpolation. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-25979 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technique of Patent Document 1 may generate redundant camera paths when moving to a preset position.

[0007] In this disclosure, the objective is to generate an appropriate camera path. [Means for solving the problem]

[0008] The information processing device The system includes an acquisition means for acquiring a user operation, a setting means for setting the position and orientation of a virtual camera in a virtual viewpoint image generated from a plurality of images captured by a plurality of imaging devices based on the user operation acquired by the acquisition means, and a generation means for generating a camera path from the positions and orientations of the plurality of virtual cameras, wherein the generation means generates a camera path using the positions and orientations of the plurality of virtual cameras set by the setting means, current Virtual camera position and orientation, alignment Temporarily Fantasy Camera preset The procedure for generating a camera path is different between when generating a camera path using the position and orientation. The generating means generates a camera path such that the path is shorter when the camera path is generated using the preset position than when the camera path is generated using the position of a virtual camera for generating the virtual viewpoint image. . [Effects of the Invention]

[0009] According to the present invention, an appropriate camera path can be generated. [Brief explanation of the drawings]

[0010] [Figure 1] An example of interpolating a virtual camera between two points. [Figure 2] FIG. 1 is a diagram illustrating the overall configuration of an image processing system 20. [Figure 3] FIG. 3 is a diagram illustrating the information processing device 203 shown in FIG. 2 according to the first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram illustrating the hardware configuration of an information processing device 203. [Figure 5]FIG. 2 is a diagram showing an example of the functional configuration of an information processing device 203 according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a diagram illustrating the operation modes of the information processing device 203 and their state transitions. [Figure 7] A diagram explaining virtual camera keyframes and their interpolation. [Figure 8] A diagram explaining the path of the virtual camera moving to a preset position. [Figure 9] FIG. 10 is a diagram showing a processing flow for generating a camera path for a clip video in the first embodiment. [Figure 10] FIG. 10 is a diagram showing a processing flow for playing back a clip video in the first embodiment; [Figure 11] FIG. 10 is a diagram showing a processing flow when a preset movement button is pressed in the first embodiment. [Figure 12] FIG. 10 is a diagram showing a processing flow for moving to a preset in the first embodiment. [Figure 13] FIG. 3 is a diagram illustrating a GUI 308 of the information processing device 203 shown in FIG. 2 according to the second embodiment. [Figure 14] FIG. 10 is a diagram showing an example of the functional configuration of an information processing device 203 according to a second embodiment. [Figure 15] FIG. 10 is a diagram showing a processing flow for generating a camera path for a clip video in the second embodiment. [Figure 16] FIG. 10 is a diagram showing a processing flow when a preset movement button is pressed in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] [First embodiment] A first embodiment will be described in detail. In this embodiment, an image processing system has predetermined procedures for interpolating between key frames in a virtual viewpoint image and for interpolating from the current position of the virtual viewpoint to the preset position of the virtual camera, and generates a camera path according to the predetermined interpolation procedures.

[0012] Here, the virtual viewpoint is a viewpoint specified by the user in the three-dimensional space of the virtual space. In the following explanation, for convenience of explanation, a camera (virtual camera) is virtually placed at the position of the virtual viewpoint. That is, the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint correspond to the position and posture of the virtual camera, respectively. Furthermore, the field of view (field of view) from the virtual viewpoint corresponds to the angle of view of the virtual camera. This virtual viewpoint can be freely (arbitrarily) specified by the user.

[0013] Furthermore, the virtual viewpoint image in this embodiment is not limited to an image corresponding to a viewpoint freely (arbitrarily) designated by the user, and also includes, for example, an image corresponding to a viewpoint selected by the user from a plurality of candidates. Furthermore, although the present embodiment will mainly describe a case where the virtual viewpoint is designated by a user operation, the virtual viewpoint may also be designated automatically based on the results of image analysis, etc. Furthermore, the present embodiment will mainly describe a case where the virtual viewpoint image is a moving image. The virtual viewpoint image can be said to be an image captured by a virtual camera.

[0014] The image processing system in this embodiment has the function of switching between and outputting captured image (hereinafter also referred to as real camera image) from an imaging device (hereinafter referred to as real camera) for actually capturing images, such as a broadcast camera, and a virtual viewpoint image corresponding to a virtual viewpoint.

[0015] 2 is a diagram illustrating the overall configuration of an image processing system 20 that generates a virtual viewpoint image. The image processing system 20 is composed of an imaging system 201, an image processing device 202, and an information processing device 203.

[0016] The photography system 201 has multiple digital cameras (image capture devices) installed at different positions surrounding (around) a photography area, and captures images in a time-synchronized manner. Multiple images captured synchronously from multiple viewpoints are sent to the image processing device 202. At this time, the multiple images sent are transmitted via a communication medium such as a LAN cable. The photography area may be a photography studio where photography is performed to create virtual viewpoint images, a stadium where sports competitions are held, or a stage where performances are performed.

[0017] The image processing device 202 generates three-dimensional shape data of the subject based on multiple images captured synchronously from multiple cameras. The three-dimensional shape data can be generated, for example, using the Visual Hull method. As a result of this processing, a 3D point cloud (a set of points with three-dimensional coordinates) representing the three-dimensional shape of the subject is obtained. Note that the method for deriving the three-dimensional shape of the subject from the captured images is not limited to this. A virtual viewpoint image corresponding to the virtual camera is generated from this three-dimensional shape data and a specified background model. The background model is, for example, a CG (Computer Graphics) model of a stadium where the physical cameras of the imaging system 201 are installed. This background model is created in advance and stored in the image processing device 202 (for example, stored in the ROM 403 in FIG. 4, which will be described later). Furthermore, model-based rendering (MBR), for example, can be used to generate the virtual viewpoint image. This processing allows for the generation of an image of the three-dimensional shape data as seen from the position and direction of the virtual camera. Note that the method for generating the virtual viewpoint image is not limited to this. Information such as the position and direction of the virtual camera is expressed by camera parameters determined by the information processing device 203, which will be described later. In this embodiment, the image processing device 202 generates both the three-dimensional shape data of the subject and the virtual viewpoint image, but the generation of each data may be shared among multiple image processing devices.

[0018] FIG. 3 is a diagram illustrating an example of an information processing device 203. Joysticks 301a and 301b for controlling a virtual camera are connected to the information processing device 203. A user sets (determines) a camera path representing the viewpoint of the virtual camera by operating the joysticks 301a and 301b. In this embodiment, commands for controlling parameters (x, y, z) representing the position of the virtual camera in three-dimensional coordinates in virtual space are assigned to the multiple operating members of the joystick 301a. Furthermore, commands for controlling parameters (Pan, Tilt, Roll) in the pan, tilt, and roll directions representing the attitude of the virtual camera in virtual space are assigned to the multiple operating members of the joystick 301b.

[0019] A keyboard 302 is also connected to the information processing device 203. Commands for generating a virtual viewpoint image are assigned to each key on the keyboard 302. For example, a command for registering a key frame is assigned to key 303, and a command for generating a camera path for a clip video from the registered key frame is assigned to key 304. Also, a command for registering a preset is assigned to key 305, and a command for canceling playback of a clip video is assigned to key 306.

[0020] The application for generating a virtual viewpoint image stored in the information processing device 203 has a function for assigning any command to any operation member among the joystick 301 a, the joystick 301 b, and the keyboard 302. The user can use this function to set any command to any operation member.

[0021] Furthermore, in this embodiment, two or more display units are connected to the information processing device 203. In this embodiment, the information processing device 203 acquires a virtual viewpoint image generated by the image processing device 202 from the image processing device 202 and displays it on the display unit 307. The information processing device 203 also displays a GUI 308 on the display unit 309, which displays information such as a camera path for generating a clip video of the virtual viewpoint image and virtual camera parameters (presets, etc.). The GUI 308 corresponds to the GUI of an application for generating a virtual viewpoint image.

[0022] 3B, the GUI 308 displayed on the display unit 309 will be described in detail. The GUI 308 includes a clip list 310, a key frame list 311, and a preset list 312.

[0023] First, the operation method on the GUI 308 for registering key frames, generating clip video, and playing back will be described. The user sets the position and orientation of the virtual camera at any time (time code) in the virtual viewpoint image and presses the key 303. In response to this operation, the information processing device 203 adds a key frame having information about the key frame ID, the camera parameters of the virtual camera, and the time code to the key frame list 311. The key frame IDs are assigned consecutively in the order added by the user. However, the user can arbitrarily change the order of the key frame IDs. When the user presses the key 304 after one or more key frames have been set in the key frame list 311, the information processing device 203 generates a camera path from the key frames included in the key frame list 311. In this embodiment, the information processing device 203 generates a camera path by interpolating the position and orientation of the virtual camera using the camera parameters of the virtual camera in the order of the key frame IDs. This generated camera path is added to the clip list 310 each time it is generated. For example, when the user clicks (selects) any camera path in the clip list 310, a virtual viewpoint image (clip video) adapted to the clicked camera path is displayed on the display unit 307.

[0024] Next, a method for registering a preset and a method for applying a preset in the GUI 308 will be described. First, a method for registering a preset will be described. When the user sets the virtual camera at an arbitrary position and orientation and presses key 305, a preset having a preset ID and information on the camera parameters of the virtual camera is added to the preset list 312. This allows the user to record the arrangement of the virtual camera at a predetermined position and orientation in the information processing device 203. Next, a method for applying a preset will be described. When the user clicks (selects) an arbitrary preset in the preset list 312, the virtual camera transitions from its current state (position and orientation) to the preset state. In this embodiment, the information processing device 203 generates a camera path for transitioning from the current state (position and orientation) of the virtual camera to the preset state, and changes the position and orientation of the virtual camera according to the camera path (moves the virtual camera).

[0025] The information processing device 203 can assign a preset to any key on the keyboard 302. In this case, for example, when the user presses a key to which a preset position of the virtual camera is assigned, the information processing device 203 moves the virtual camera to the preset corresponding to the pressed key.

[0026] Then, the information processing device 203 transmits the camera path of the virtual camera selected by the user's operation to the image processing device 202. The image processing device 202 generates a virtual viewpoint image based on the camera path received from the information processing device 203.

[0027] Figure 1 shows an example of interpolation between two keyframes when generating a camera path. Figure 1 shows a method for interpolating the value p of one of the camera parameters at the start and end points. When the user sets the time and parameter at the start point as t1·p1, and the time and parameter at the end point as t2·p2, the value p is interpolated between t1 and t2 as shown in the graph in Figure 1. The virtual camera parameters that are interpolated in this way include the position coordinates (x, y, z), orientation (Pan, Tilt, Roll), and magnification rate Zoom.

[0028] The key frames registered at this time may allow the desired camera path to be generated with fewer key frames by setting unlimited upper and lower limits for parameters representing posture, such as Pan, Tilt, and Roll. For example, consider the case of setting the gaze point at an arbitrary position and generating a camera path that rotates twice while facing the gaze point. If the upper and lower limits of the rotation angle are unlimited, the path can be generated simply by setting two points: a start point parameter of 0° and an end point parameter of 720°. On the other hand, if the minimum value is set to 0° and the maximum value is set to 359°, a rotation angle of 360° or more cannot be set, so three or more key frames are required. In this embodiment, a camera path is generated using key frames with unlimited upper and lower limits. Hereinafter, the method of generating a camera path using key frames will also be referred to as the key frame method.

[0029] On the other hand, when generating camera paths for other purposes, using an interpolation procedure similar to that used to generate a camera path for clip video using the keyframe method described above may result in a camera path that is not desired by the user. For example, consider moving a virtual camera from its current position to a pre-registered position and orientation (preset position). One use case is live broadcasting of a baseball game. In live broadcasts, viewpoints such as those from the mound on the field or the catcher's position tend to produce images with a sense of realism. Real-time live broadcasts require frequent switching of viewpoints, so a function is required to register multiple preset positions and switch the virtual camera position to the desired preset position. However, if the virtual camera transitions from its current position to a preset position in an instant, even the user operating the virtual camera may need time to determine whether the switched position is the desired preset position. Therefore, it is necessary to devise a method for generating a camera path that moves the virtual camera from its current position to a preset position and displaying the movement. However, if you generate a camera path by interpolating between the current position and a preset using a method similar to the keyframe interpolation described above, you may end up with a redundant camera path that rotates unnecessarily, etc. Therefore, when generating a camera path that moves from the current position to a preset position, you need to generate the camera path taking into account the positional relationship.

[0030] 4 is a diagram showing an example of the hardware configuration of an information processing device 203 according to this embodiment that can generate and edit a camera path for a virtual viewpoint image. The information processing device 203 is composed of a CPU 401, a RAM 402, a ROM 403, and an input / output device unit 404. Note that in the virtual viewpoint image, a user such as a viewer or a camera operator can freely manipulate the position and posture of the virtual camera. Furthermore, the virtual viewpoint image may be a video or a still image.

[0031] The CPU 401 is a processor that uses the RAM 402 as a work memory, executes programs stored in the ROM 403, and performs overall control of each component of the information processing device 203. As a result, the CPU 401 executes various programs to realize the functions of each processing unit shown in Fig. 2, which will be described later.

[0032] The RAM 402 temporarily stores computer programs read from the ROM 403, intermediate calculation results, and the like.

[0033] The ROM 403 holds computer programs and data that do not require modification, and also stores data necessary for interpolating camera parameters, such as the interpolation method of the key frame method described above, and data necessary for determining the interpolation method.

[0034] The input / output device unit 404 has multiple controllers for controlling the virtual camera and multiple display units for displaying the status of the virtual camera, etc. The multiple controllers include general devices such as a keyboard and a mouse for the user to perform input operations, as well as a joystick, knob, jog dial, etc. for operating the virtual camera. The display unit is one or multiple display devices (hereinafter referred to as "monitors") for displaying information required by the user.

[0035] 5 is a block diagram showing an example of the functional configuration of the information processing device 203 according to the first embodiment. The information processing device 203 generates a camera path for a virtual camera in response to a user operation, and transmits the camera path to the virtual viewpoint image generation device.

[0036] Each function installed in the information processing device 203 will be explained in order.

[0037] The operation information acquisition unit 501 acquires the user's operation of the joystick 301 for each frame and converts it into parameters such as the position and orientation of the virtual camera according to the amount of operation. These parameters represent the amount of change in the virtual camera parameters for each frame. Furthermore, when there is a user operation (input) on the keyboard 302 or GUI 308, the operation information acquisition unit 501 transmits an input signal according to the operation to the operation mode determination unit 502, key frame holding unit 504, clip creation unit 505, and virtual camera parameter calculation unit 511.

[0038] The control mode determination unit 502 switches between a plurality of control modes in response to a user operation. The control modes include a manual control mode, a clip playback mode, and a preset movement mode. In the manual control mode, the information processing device 203 moves the virtual camera in response to a user operation on the joystick 301. In the clip playback mode, the information processing device 203 plays back a virtual viewpoint image generated using a camera path generated for the clip video. This virtual viewpoint image is generated by an external virtual viewpoint image generation device (not shown). In the preset movement mode, the information processing device 203 moves the position of the virtual camera to a preset position. Here, in the preset movement mode, the information processing device 203 generates a camera path for moving the position of the virtual camera to the preset position, and plays back on the display unit the virtual viewpoint image generated using the camera path.

[0039] The control mode holding unit 503 is a recording unit that holds (records) the current control mode. The held control mode is updated when a switching signal is input from the control mode determination unit 502. The control mode holding unit 503 also transmits the currently held control mode to the control mode determination unit 502.

[0040] FIG. 6 shows a state transition diagram of the three control modes. In this embodiment, the control mode is normally manual, and the user operates the virtual camera. When the user clicks on a camera path in the clip list 310, the control mode determination unit 502 switches the control mode to clip playback mode. Then, it transmits a signal notifying the control mode storage unit 503 and the virtual camera parameter calculation unit 511 that the mode has been switched. When the user clicks on any preset in the preset list 312, it switches the control mode to preset movement mode and transmits a signal notifying the control mode storage unit 503 and the virtual camera parameter calculation unit 511 that the mode has been switched. When the control mode is preset movement mode and movement to the preset is completed, the control mode determination unit 502 receives a signal for switching to manual control mode from the virtual camera parameter calculation unit 511, and switches to manual control mode.

[0041] When the user inputs a key 303 on the keyboard 302, the key frame holding unit 504 receives an input signal for registering a key frame from the operation information acquisition unit 501. A command for registering a key frame is assigned to the key 303. Upon receiving the input signal, the key frame holding unit 504 acquires the current virtual camera parameters from the virtual camera parameter calculation unit 511 and stores them as a key frame. The key frame holding unit 504 can hold multiple key frames. Each key frame holds, as parameters, a key frame ID, which is an integer value; external parameters such as the position and orientation of the virtual camera; internal parameters such as the optical center and focal length; and time information such as a time code. A key frame ID is assigned to each key frame in sequence when the key frame holding unit acquires a key frame. The key frame ID is a parameter that can be changed by the user. When the user inputs a key 304 on the keyboard 302, the held key frame is output to the clip creation unit 505 and then cleared. It should be noted that a command for generating a camera path for a clip video from a registered key frame is assigned to the key 304 .

[0042] When the user inputs an input to a key 304 of the keyboard 302, the clip creation unit 505 receives an input signal from the operation information acquisition unit 501. Upon receiving the input signal, the clip creation unit 505 acquires a plurality of key frames stored in the key frame storage unit 504 and generates a camera path by interpolating between the key frames.

[0043] 7(a) and 7(b) are diagrams illustrating an example of a method for interpolating between key frames for the x coordinate of the virtual camera's position coordinates. In FIG. 7(a), the positions and orientations of the virtual cameras are set as virtual cameras 701 to 704 at time codes t1 to t4. The graph in FIG. 7(a) shows the x coordinates of the virtual cameras at times t1 to t4. As shown in FIG. 7(b), the clip creation unit 505 generates a camera path 705 by interpolating between the virtual cameras 701 to 704 using an interpolation method that smooths the movement trajectory of the virtual camera. The graph in FIG. 7(b) shows the transition of the x coordinate of the generated camera path 705 between times t1 to t4. Here, the interpolation method can be, for example, spline interpolation. The clip creation unit 505 similarly interpolates the y coordinate and z coordinate of the virtual camera's position coordinates and the orientation information such as pan, tilt, and roll.

[0044] Clip storage unit 506 stores a camera path for a clip video generated by clip creation unit 505. When the user clicks (selects) any camera path in clip list 310, clip storage unit 506 transmits the selected camera path to virtual camera parameter calculation unit 511 in accordance with the input signal received from operation information acquisition unit 501.

[0045] When the user inputs a key 305 of the keyboard 302, the preset holding unit 507 acquires the camera parameters of the virtual camera from the virtual camera parameter calculation unit 511 and holds them as a preset position of the virtual camera. Note that a command for registering a preset is assigned to the key 305.

[0046] The preset movement time storage unit 508 stores the movement time from the virtual camera position to the preset. The movement time is registered (input) in advance by the user using the keyboard 302 or the like. This preset movement time is different from the time when the clip video is generated.

[0047] The preset movement remaining time holding unit 509 holds the remaining time to move to the preset. The preset movement remaining time holding unit 509 holds the preset movement time when the user clicks (selects) a preset in the preset list 312 of the GUI 308. Thereafter, the preset movement remaining time holding unit 509 also reduces the held remaining time by one frame each time one frame of the virtual viewpoint image is played back. The preset movement remaining time holding unit 509 transmits the remaining time to the virtual camera parameter calculation unit 511 for each frame.

[0048] The virtual camera parameter storage unit 510 acquires and stores the virtual camera parameters calculated by the virtual camera parameter calculation unit 511. The virtual camera parameter storage unit 510 also transmits the virtual camera parameters stored at the current time code to the virtual camera parameter calculation unit 511. Thereafter, the virtual camera parameters for the next frame calculated by the virtual camera parameter calculation unit 511 are acquired and the stored virtual camera parameters are updated.

[0049] When the control mode is updated, the virtual camera parameter calculation unit 511 receives a signal from the control mode determination unit 502 and calculates (calculates) the virtual camera parameters using a method that matches the control mode. Below, the calculation method of the virtual camera parameters in each control mode will be described.

[0050] In the manual operation mode, the virtual camera parameter calculation unit 511 adds the change amount of the virtual camera parameters from the operation information acquisition unit 501 and the camera parameters acquired from the virtual camera parameter storage unit 510, and calculates the virtual camera parameters for the next frame.

[0051] In the clip playback mode, the virtual camera parameter calculation unit 511 acquires the clip information selected by the user from the operation information acquisition unit 501, and acquires the corresponding camera path from the clip storage unit 506. The camera path acquired here is the camera path generated by the key frame method described above.

[0052] In the preset movement mode, the virtual camera parameter calculation unit 511 calculates a camera path indicating the path from the current position and orientation of the virtual camera to the preset. Specifically, when the frame rate is 60 fps, the virtual camera parameter calculation unit 511 performs calculation using the following Equation 1. [Formula 1] dp=(p_pre-p(t)) / f(t) p(t-1)=p(t)+dp f(t-1)=f(t)-1

[0053] Here, t is the time (remaining time) required for the virtual camera to move from the current position to the preset position, acquired from the remaining time storage unit 509. In this embodiment, t is expressed in the format of HH:MM:SS:FF, for example. p(t) is the virtual camera parameter at the current time acquired from the virtual camera parameter storage unit 510 at the remaining time t. p_pre is the virtual camera parameter at the preset position acquired from the preset storage unit 507. f(t) is the number of remaining frames in the remaining time t. The initial value of f(t) is calculated by multiplying the initial value of the remaining time t, converted into seconds, by the frame rate. dp is the amount of change in the virtual camera parameter per frame. Therefore, dp can be either positive or negative. At this time, when t = 0, the virtual camera parameter calculation unit 511 sends a signal to the operation mode determination unit 502 to switch to the manual operation mode. Note that if the frame rate is a non-integer, such as 59.94 fps, the virtual camera parameter calculation unit 511 performs calculations in the above formula, taking dropped frames into account.

[0054] Furthermore, in the preset movement mode, the virtual camera parameter calculation unit 511 acquires preset virtual camera parameters from the preset storage unit 507 and virtual camera parameters from the virtual camera parameter storage unit 510. Then, the virtual camera attitude parameters are corrected so that the path between two points is the shortest.

[0055] A method for correcting the attitude parameters of a virtual camera will be described with reference to FIG. 8 . For example, consider generating a camera path that moves from virtual camera position 801 in FIG. 8 to preset position 802. The Pan values ​​for virtual camera position 801 and preset position 802 when rotating while facing the gaze point 803 are set to 0° and 270°, respectively. In this case, if a path between virtual camera position 801 and preset position 802 is interpolated using a method similar to interpolation between key frames, the Pan value monotonically increases from 0° to 270°, resulting in a camera path of the virtual camera as path 804. However, in this embodiment, preset positions are used to quickly move the virtual camera to a desired position in real-time video. In other words, the camera path shown by path 804 is a detour from virtual camera position 801 to preset position 802 and is not an appropriate camera path. The path from virtual camera position 801 to preset position 802 is preferably path 805. Therefore, the virtual camera parameter calculation unit 511 corrects the values ​​of Pan, Tilt, and Roll, which are parameters that represent the orientation of the virtual camera, while keeping the position and orientation of the current virtual camera position 801 fixed. Specifically, if the absolute value of the difference between any of the Pan, Tilt, and Roll values ​​between the current virtual camera position and a preset position is greater than 180°, the virtual camera parameter calculation unit 511 corrects the value. For example, for a parameter whose difference from the preset position is greater than 180°, the virtual camera parameter calculation unit 511 corrects the orientation parameter of the current virtual camera position so that the absolute value of the difference is 180° or less. Specifically, if the value of the current virtual camera position is greater than 180° from the value of the preset position for that parameter, the virtual camera parameter calculation unit 511 subtracts a value that is an integer multiple of 360° from the value of the current camera path position so that the difference is 180° or less. Conversely, if the value of the current virtual camera position for that parameter is 180° or more smaller than the value of the preset position, the virtual camera parameter calculation unit 511 adds a value that is an integer multiple of 360° to the value of the current camera path position so that the difference is 180° or less.In this way, the virtual camera parameter calculation unit 511 can correct the orientation parameters of the virtual camera without changing the position and orientation of the virtual camera by adding or subtracting a multiple of 360° to or from the parameters to be corrected. In addition, by making the difference 180° or less as described above, a camera path that does not take a detour is generated.

[0056] The virtual camera parameter calculation unit 511 transmits the camera parameters that have been subjected to the above-mentioned correction processing to the virtual camera parameter provision unit 512 .

[0057] The virtual camera parameter providing unit 512 acquires the camera parameters for each frame from the virtual camera parameter calculating unit 511 and transmits them to the virtual viewpoint image generating device.

[0058] The time code holding unit 513 acquires the current time code from the time code calculation unit 514 and holds it.

[0059] The time code calculation unit 514 acquires the time code for each frame from the time code holding unit 513 , updates it, and transmits it to the time code holding unit 513 and the time code providing unit 515 .

[0060] The time code providing unit 515 acquires the time code from the time code calculation unit 514 and transmits it to the virtual viewpoint image generating device.

[0061] The display 516 displays the virtual viewpoint image generated by the virtual viewpoint image generating device, and is, for example, a liquid crystal display or an organic EL display.

[0062] The information processing device 203 communicates with the image processing device 202 using a communication unit (not shown). The communication unit performs communication according to a wired communication standard such as Ethernet or a wireless communication standard such as Wi-Fi (registered trademark).

[0063] Next, a processing flow for generating a camera path for a clip video in this embodiment will be described with reference to Fig. 9. This flowchart is triggered, for example, by the user pressing key 304 on keyboard 302. Note that key 304 is assigned a command for generating a camera path for a clip video from a registered key frame.

[0064] In step S901, the operation information acquisition unit 501 transmits a signal to the clip creation unit 505 to generate a camera path.

[0065] In step S902, the clip creation unit 505 receives a signal for generating a camera path from the operation information acquisition unit 501, and acquires key frames from the key frame storage unit 504. The clip creation unit 505 generates a camera path by interpolating between key frames in the order of their key frame IDs.

[0066] In step S 903 , the clip holding unit 506 acquires a camera path from the clip creation unit 505 and adds the acquired camera path to the clip list 310 .

[0067] In step S904, in response to the generated camera path being added to the clip list 310, the key frame holding unit 504 clears the held key frames (key frame list).

[0068] Next, a processing flow for playing back clip video in this embodiment will be described with reference to Fig. 10. This flowchart starts, for example, in response to the user clicking on an arbitrary camera path from clip list 310 on GUI 308. Note that this operation of the user clicking on an arbitrary camera path is an operation for playing back clip video.

[0069] In step S1001, the operation information acquisition unit 501 transmits to the operation mode determination unit 502 a signal for switching the operation mode to the clip playback mode.

[0070] In step S1002, the control mode determination unit 502 receives a signal from the operation information acquisition unit 501 and switches the control mode to the clip playback mode. The control mode determination unit 502 transmits a signal notifying that the control mode has been switched to the clip playback mode to the control mode storage unit 503 and the virtual camera parameter calculation unit 511.

[0071] In step S1003, the virtual camera parameter calculation unit 511 acquires the clip information selected by the user from the operation information acquisition unit 501, and acquires the corresponding camera path from the clip storage unit 506. The virtual camera parameter calculation unit 511 transmits the acquired camera path to the virtual camera parameter provision unit 512.

[0072] In step S1004, the virtual camera parameter providing unit 512 receives the camera path and transmits the camera path to the virtual viewpoint image generating device. Furthermore, the time code providing unit 515 acquires the time code from the time code calculation unit 514 and transmits it to the virtual viewpoint image generating device. A virtual viewpoint image is then generated in the virtual viewpoint image generating device. The information processing device 203 transmits the camera path and the time code via the communication unit.

[0073] In step S1005, the display 516 displays the virtual viewpoint image acquired from the virtual viewpoint image generating device. The displayed virtual viewpoint image corresponds to the clip video. Here, the information processing device 203 acquires (receives) the virtual viewpoint image via the communication unit.

[0074] Next, a process flow when the user selects (presses) an arbitrary preset position on the preset list 312 on the GUI 308 in this embodiment will be described with reference to Fig. 11. The process of this flowchart starts in response to the user selecting an arbitrary preset position.

[0075] In step S1101, the operation information acquisition unit 501 transmits to the operation mode determination unit 502 a signal for switching the operation mode to the preset movement mode.

[0076] In step S1102, when the control mode determination unit 502 receives a signal to switch the control mode to the preset movement mode from the operation information acquisition unit 501, it acquires information about the current control mode from the control mode storage unit 503. If the current control mode is the clip playback mode, the processes from S1103 onwards are not performed. If the current control mode is the manual control mode or the preset movement mode, the control mode determination unit 502 performs the process of step S1103.

[0077] In step S1103, if the control mode is the manual control mode, the control mode determination unit 502 switches the control mode to the preset movement mode. If the control mode is the preset movement mode, the control mode determination unit 502 maintains the control mode as the preset movement mode. The control mode determination unit 502 transmits a signal notifying that the control mode has been switched to the preset movement mode to the control mode determination unit 502 and the virtual camera parameter calculation unit 511.

[0078] In step S1104, virtual camera parameter calculation unit 511 receives a signal notifying that the mode has been switched to the preset movement mode from operation mode determination unit 502. Then, virtual camera parameter calculation unit 511 acquires camera parameters of the preset position selected by the user from preset storage unit 507. Also, virtual camera parameter calculation unit 511 acquires current camera parameters from virtual camera parameter storage unit 510. Virtual camera parameter calculation unit 511 calculates the absolute value of the difference for each acquired parameter (Pan, Tilt, Roll) related to each attitude, and determines whether the difference is greater than 180°. In the flowchart, for these parameters, the value of the current virtual camera position is referred to as the current attitude, and the value of the preset position is referred to as the preset attitude.

[0079] In step S1105, if there is a parameter whose absolute value of the difference is greater than 180° in step S1104, virtual camera parameter calculation unit 511 compares the value of the preset position with the value of the current virtual camera position for each of the parameters. If the value of the current virtual camera position for the parameter is greater than the value of the preset position, the process of step S1106 is executed. If the value of the current virtual camera position for the parameter is equal to or less than the value of the preset position, the process of step S1106 is executed.

[0080] In step S1106, the virtual camera parameter calculation unit 511 subtracts 360° from the value of the current virtual camera position for the parameter, and substitutes the result for the value of the virtual camera position held by the virtual camera parameter holding unit 510.

[0081] In step S1107, the virtual camera parameter calculation unit 511 adds 360° to the value of the current virtual camera position for the parameter, and substitutes the result for the value of the virtual camera position held by the virtual camera parameter holding unit 510.

[0082] The above processing of steps S1104 to S1107 is repeated for each of the Pan, Tilt, and Roll parameters until the difference between the value stored in the virtual camera parameter storage unit 510 and the preset value becomes 180° or less. As a result, for each of the Pan, Tilt, and Roll parameters, a value that is an integral multiple of 360° is added to or subtracted from the value of the camera path position.

[0083] In step S1108, the preset remaining movement time holding unit 509 acquires the preset movement time from the preset movement time holding unit 508, and sets it as the remaining time to move to the preset.

[0084] The above-described processing from step S1101 to step S1108 is processing that is performed before the current virtual camera is transitioned to the preset state.

[0085] The processing flow for transitioning the virtual camera to a preset state will be described using the flowchart in Fig. 12. The processing shown in this flowchart is executed following the flowchart shown in Fig. 11. Note that the following processing is executed for each frame.

[0086] In step S1201, the control mode determination unit 502 acquires information about the current control mode from the control mode storage unit 503. If the control mode is the clip playback mode or the manual control mode, the following processing is not performed and the process proceeds to the next frame. If the control mode is the preset movement mode, the process of step S1202 is executed.

[0087] In step S1202, the virtual camera parameter calculation unit 511 acquires the camera parameters of the preset position, the camera parameters of the current virtual camera position, and the remaining movement time, and calculates the movement amount in one frame.

[0088] In step S1203, the virtual camera parameter calculation unit 511 acquires the camera parameters of the current virtual camera position from the virtual camera parameter storage unit 510, and transmits to the virtual camera parameter storage unit 510 the new camera parameter position to which the movement amount has been added.

[0089] In step S1204, the preset remaining movement time holding unit 509 subtracts one frame from the remaining movement time and updates the held preset remaining movement time.

[0090] In step S1205, virtual camera parameter calculation unit 511 acquires the preset remaining movement time from preset remaining movement time storage unit 509. If the preset remaining movement time is 0, the process of step S1206 is executed. If the preset remaining movement time is not 0, the process of the next frame is executed.

[0091] In step S1206, if the preset remaining movement time acquired by virtual camera parameter calculation unit 511 is 0, virtual camera parameter calculation unit 511 transmits a signal to control mode determination unit 502 to switch the control mode to the manual control mode. Having received the signal, control mode determination unit 502 switches the control mode to the manual control mode. After switching the control mode to the manual control mode, control mode determination unit 502 transmits the current control mode (manual control mode) to control mode holding unit 503.

[0092] The above processing flow is performed for each frame, and the position of the virtual camera is transitioned to the preset position.

[0093] In this example, we have shown an example in which the interpolation procedure (camera path generation method) is changed when generating a camera path for a clip video and when moving to a preset. In other words, when moving to a preset position, by adding camera parameter correction processing so that the difference in posture parameters between two points is 180° or less, the camera path does not become redundant and a shorter route is generated.

[0094] [Second embodiment] Next, a second embodiment for carrying out the present invention will be described in detail.

[0095] In the first embodiment, as shown in Fig. 8, the information processing device 203 performed a process of correcting parameters in order to move the virtual camera from a current position 801 to a preset position 802 in a real-time video so as to shorten the route. However, there are situations in which the user wants the camera path to capture a certain area while moving, rather than simply following a short route. For example, if there is something that the user wants to display in area 806 of a subject at point of interest 803, the object will not be visible from the virtual camera following route 805, and therefore route 804 may be preferable for the user.

[0096] Conversely, when generating clip video, there are situations where a shorter path is preferable, such as when moving to a preset rather than the usual interpolation procedure.

[0097] Therefore, in the second embodiment, the user can select the interpolation procedure between key frames and the interpolation procedure between the current position and the preset, and the camera path is automatically generated according to the selected interpolation method.

[0098] An example of a GUI in the second embodiment is shown in Fig. 13. In Fig. 13, components that perform the same functions as those in the GUI in Fig. 3 are denoted by the same reference numerals, and only the differences from the first embodiment will be described.

[0099] Each camera path in the clip list 310 and each preset position in the preset list 312 are assigned a button 1301 or 1302 indicating whether or not to perform parameter correction to shorten the path. The user can switch parameter correction ON / OFF by clicking button 1301 or 1302. When parameter correction is ON, the information processing device 203, as in the first embodiment, performs processing to correct the parameters of the current virtual camera so that the values ​​of the parameters Pan, Tilt, and Roll are 180° or less between the preset and the current virtual camera. Then, the information processing device 203 generates a camera path using the corrected parameters. When parameter correction is OFF, the information processing device 203 generates a camera path without correcting the parameters.

[0100] Fig. 14 is a block diagram showing the functional configuration of an information processing device 203 in the second embodiment. The information processing device 203 generates a camera path of a virtual camera using a method selected by a user and transmits it to a virtual viewpoint image generation device. In Fig. 14, components that perform the same functions as those of the information processing device in Fig. 5 are denoted by the same reference numerals, and only differences from the first embodiment will be described.

[0101] When the operation information acquisition unit 501 acquires input information from the user to the button 1301 on the GUI 308, it transmits a signal for switching parameter correction ON / OFF to the interpolation switching determination unit 1401 and the virtual camera parameter calculation unit 511.

[0102] The interpolation switching determination unit 1401 receives a signal for switching ON / OFF of parameter correction from the operation information acquisition unit 501. Thereafter, the interpolation switching determination unit 1401 switches ON / OFF of parameter correction and transmits the setting after switching to the interpolation switching holding unit 1402. The interpolation switching determination unit 1401 also acquires current ON / OFF information from the interpolation switching holding unit 1402 and transmits it to the virtual camera parameter calculation unit 511.

[0103] The interpolation switching holding unit 1402 transmits ON / OFF information to the interpolation switching determination unit 1401. Furthermore, when the interpolation switching determination unit 1401 changes ON / OFF, the interpolation switching holding unit 1402 acquires the information from the interpolation switching determination unit 1401 and holds it.

[0104] When the virtual camera parameter calculation unit 511 receives a signal for switching parameter correction ON / OFF from the operation information acquisition unit 501, it acquires parameter correction ON / OFF information from the interpolation switching determination unit 1401. When the virtual camera parameter calculation unit 511 acquires information that parameter correction is ON, it corrects the attitude parameters of the virtual camera so that interpolation between key frames or from the current virtual camera parameters to the preset is performed in the shortest possible time, as described in the first embodiment. When the virtual camera parameter calculation unit 511 acquires information that parameter correction is OFF, it interpolates the path between key frames or from the current virtual viewpoint to the preset without correcting the attitude parameters.

[0105] Next, the processing flow for generating a camera path for a clip video in this embodiment will be described with reference to Fig. 15. In Fig. 15, the same reference numerals are used to denote the same functions as those in the processing flow of Fig. 9, and only the differences from the first embodiment will be described.

[0106] In step S1501, the interpolation switching determination unit 1401 acquires ON / OFF information of the current parameter correction setting from the interpolation switching holding unit 1402 and transmits it to the clip creation unit 505. If the parameter correction setting is ON, the process of step S1502 is executed. If the parameter correction setting is OFF, the process of step S902 is executed.

[0107] In step S1502, the clip creation unit 505 acquires a list of key frames from the key frame storage unit 504. The difference between the Pan, Tilt, and Roll values ​​of a key frame in the order of frame IDs and the value of the key frame with the next frame ID is calculated.

[0108] In step S1503, if the absolute value of the difference between the Pan, Tilt, and Roll values ​​of the key frame to be processed and the next key frame is greater than 180°, the clip creation unit 505 checks which of the parameters has the larger value. If the difference in absolute values ​​is 180° or less, the process proceeds to the next frame.

[0109] In step S1504, if the Pan, Tilt, and Roll values ​​of the key frame to be processed are larger, the clip creation unit 505 subtracts 360° from the values ​​of all key frames before the key frame to be processed.

[0110] In S1505, if the key frame to be processed has larger Pan, Tilt, and Roll values, the clip creation unit 505 adds 360° to the values ​​of all key frames before the key frame to be processed.

[0111] The above processing of steps S1502 to S1505 is repeated for each of the Pan, Tilt, and Roll parameters until the difference between the values ​​of the two key frames becomes 180° or less. As a result, for each of the Pan, Tilt, and Roll parameters, a value that is an integral multiple of 360° is added to or subtracted from the value of the key frame being processed.

[0112] Next, the processing flow when the user presses any preset on the preset list 312 on the GUI 308 in this embodiment will be described with reference to Fig. 16. In Fig. 16, the same reference numerals are used to denote processes that perform the same functions as those in the processing flow of Fig. 11, and only the differences from the first embodiment will be described.

[0113] In step S1601, the interpolation switching determination unit 1401 acquires ON / OFF information of the current parameter correction setting from the interpolation switching holding unit 1402 and transmits it to the virtual camera parameter calculation unit 511. If the parameter correction setting is ON, the process of step S1104 is executed. If the parameter correction setting is OFF, the process of step S1108 is executed.

[0114] As described above, in this embodiment, a button is provided that allows the user to select whether or not to correct the virtual camera parameters for the camera paths and presets for all clip images on the GUI 308, and a camera path can be generated using different interpolation procedures depending on the user's selection. This allows the user's desired camera path to be automatically generated, for example, when the user wants to interpolate so that the path between key frames is the shortest, or when the user wants to move to a preset position via a path that is not the shortest.

[0115] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0116] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0117] In the above embodiment, at least one of A and B may be only A, only B, or both A and B.

[0118] The disclosure of this embodiment includes the following configurations and methods.

[0119] (Configuration 1) An information processing device in an image processing system that captures images using a plurality of imaging devices arranged around a subject and generates a virtual viewpoint image from the captured images, An acquisition means for acquiring a user operation; a holding means for holding a position of a virtual camera in the virtual viewpoint image; generating means for generating a camera path from the positions of the plurality of virtual cameras held by the holding means; the holding means holds a position of a virtual camera for generating the virtual viewpoint image and a preset position for changing the virtual camera to a predetermined position; The generating means generates a camera path using different procedures depending on whether the camera path is generated using the position of a virtual camera for generating the virtual viewpoint image or the preset position. 1. An information processing device comprising:

[0120] (Configuration 2) The information processing device described in configuration 1, wherein the generation means generates a camera path so that the path is shorter when the camera path is generated using the preset position than when the camera path is generated using the position of a virtual camera for generating the virtual viewpoint image.

[0121] (Configuration 3) 3. The information processing device according to configuration 1 or 2, wherein the storage means stores information relating to the position of the virtual camera, as well as an ID, time information, and attitude information of the virtual camera.

[0122] (Configuration 4) 4. The information processing device according to configuration 3, wherein the generating means generates a camera path by interpolating the position of the virtual camera in the order of the IDs.

[0123] (Configuration 5) the storage means further stores a movement time, which is different from the time information, for generating a camera path using the preset position; When generating a camera path using a position of a virtual camera for generating the virtual viewpoint image, the generation means generates the camera path based on time information recorded together with the position of the virtual camera for generating the virtual viewpoint image, and when generating a camera path using the preset position, the generation means generates the camera path based on the movement time. 5. The information processing device according to configuration 3 or 4.

[0124] (Configuration 6) 6. The information processing device according to any one of configurations 1 to 5, wherein the generating means, when generating a camera path using the preset position, corrects the position of the virtual camera based on the preset position.

[0125] (Configuration 7) 7. The information processing device according to any one of configurations 1 to 6, further comprising a switching means for switching, in response to the user operation, between an operation mode for generating a camera path using a position of a virtual camera for generating the virtual viewpoint image and an operation mode for generating a camera path using the preset position.

[0126] (Configuration 8) An information processing device in an image processing system that captures images using a plurality of imaging devices arranged around a subject and generates a virtual viewpoint image from the captured images, An acquisition means for acquiring a user operation; a storage means for storing information relating to an ID, time information, and posture information of the virtual camera together with the position of the virtual camera in the virtual viewpoint image, and for storing a moving time different from the time information; a generating means for generating a camera path from the positions of the plurality of virtual cameras held in the holding means; The generating means has a switching means for switching between a mode in which the generating means generates a camera path based on time information recorded together with the position of the virtual camera for generating the virtual viewpoint image and a mode in which the camera path is generated using the movement time. 1. An information processing device comprising:

[0127] (Method 1) A control method for an information processing device in an image processing system that captures images using a plurality of imaging devices arranged around a subject and generates a virtual viewpoint image from the captured images, comprising: an acquisition step of acquiring a user operation; a holding step of holding the position of the virtual camera in the virtual viewpoint image; and generating a camera path from the positions of the plurality of virtual cameras held, The holding step holds a position of a virtual camera for generating the virtual viewpoint image and a preset position for changing the virtual camera to a predetermined position; In the generating step, a procedure for generating a camera path is made different depending on whether the camera path is generated using the position of a virtual camera for generating the virtual viewpoint image or the camera path is generated using the preset position. A control method comprising:

[0128] (Method 2) A control method for an information processing device in an image processing system that captures images using a plurality of imaging devices arranged around a subject and generates a virtual viewpoint image from the captured images, comprising: an acquisition step of acquiring a user operation; a storing step of storing information relating to an ID, time information, and attitude information of the virtual camera together with the position of the virtual camera in the virtual viewpoint image, and also storing a movement time different from the time information; a generating step of generating a camera path from the positions of the plurality of virtual cameras held; A switching step is performed to switch between a mode in which a camera path is generated based on time information recorded together with the position of a virtual camera for generating the virtual viewpoint image and a mode in which a camera path is generated using the movement time. A control method comprising:

Claims

1. An acquisition means for acquiring a user operation; a setting means for setting a position and an orientation of a virtual camera in a virtual viewpoint image generated from a plurality of images captured by a plurality of imaging devices, based on a user operation acquired by the acquisition means; generating means for generating a camera path from the positions and orientations of a plurality of virtual cameras; the generating means uses different procedures for generating a camera path depending on whether the camera path is generated using the positions and orientations of the plurality of virtual cameras set by the setting means or the camera path is generated using the current position and orientation of the virtual camera and a preset position and orientation of the virtual camera; The generating means generates a camera path such that the path is shorter when the camera path is generated using the preset position than when the camera path is generated using the position of a virtual camera for generating the virtual viewpoint image.

1. An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, further comprising a storage means for storing information relating to the position of the virtual camera, as well as an ID, time information, and attitude information of the virtual camera.

3. 3. The information processing apparatus according to claim 2, wherein the generating means generates a camera path by interpolating the position of a virtual camera in the order of the IDs.

4. the storage means further stores a movement time, which is different from the time information, for generating a camera path using the preset position; When generating a camera path using a position of a virtual camera for generating the virtual viewpoint image, the generation means generates the camera path based on time information recorded together with the position of the virtual camera for generating the virtual viewpoint image, and when generating a camera path using the preset position, the generation means generates the camera path based on the movement time.

3. The information processing apparatus according to claim 2, wherein:

5. The information processing apparatus according to claim 1 , wherein the generating means, when generating a camera path using the preset positions, corrects the position of the virtual camera based on the preset positions.

6. 2. The information processing device according to claim 1, further comprising a switching means for switching, in response to the user operation, between an operation mode for generating a camera path using the position of a virtual camera for generating the virtual viewpoint image and an operation mode for generating a camera path using the preset position.

7. An acquisition means for acquiring a user operation; a storage means for storing information relating to an ID, time information, and posture information of the virtual camera together with a position of the virtual camera in a virtual viewpoint image generated from a plurality of images captured by a plurality of imaging devices, and for storing a movement time different from the time information; a generating means for generating a camera path from the positions of the plurality of virtual cameras held in the holding means; a switching means for switching between a mode in which the generating means generates a camera path based on time information recorded together with the position of a virtual camera for generating the virtual viewpoint image and a mode in which the generating means generates a camera path using the movement time in response to a user operation acquired by the acquiring means; 1. An information processing device comprising:

8. an acquisition step of acquiring a user operation; a setting step of setting a position and an orientation of a virtual camera in a virtual viewpoint image generated from a plurality of images captured by a plurality of imaging devices, based on the user operation acquired in the acquisition step; generating a camera path from the positions and orientations of the plurality of virtual cameras; In the generating step, a procedure for generating a camera path is made different between a case where a camera path is generated using the positions and orientations of the plurality of virtual cameras set in the setting step and a case where a camera path is generated using the current position and orientation of the virtual camera and a preset position and orientation of the virtual camera; In the generating step, a camera path is generated so that the path is shorter when the camera path is generated using the preset position than when the camera path is generated using the position of a virtual camera for generating the virtual viewpoint image.

2. A method for controlling an information processing apparatus comprising:

9. an acquisition step of acquiring a user operation; a setting step of setting a position and an orientation of a virtual camera in a virtual viewpoint image generated from a plurality of images captured by a plurality of imaging devices, based on the user operation acquired in the acquisition step; generating a camera path from the positions and orientations of the plurality of virtual cameras; In the generating step, a procedure for generating a camera path is made different between a case where a camera path is generated using the positions and orientations of the plurality of virtual cameras set in the setting step and a case where a camera path is generated using the current position and orientation of the virtual camera and a preset position and orientation of the virtual camera; In the generating step, a camera path is generated so that the path is shorter when the camera path is generated using the preset position than when the camera path is generated using the position of a virtual camera for generating the virtual viewpoint image. A program for causing an information processing device to execute the control method.

10. an acquisition step of acquiring a user operation; a storing step of storing information relating to an ID, time information, and posture information of the virtual camera together with a position of the virtual camera in a virtual viewpoint image generated from a plurality of images captured by a plurality of imaging devices, and also storing a movement time different from the time information; a generating step of generating a camera path from the positions of the plurality of virtual cameras held; a switching step of switching between a mode in which a camera path is generated based on time information recorded together with the position of a virtual camera for generating the virtual viewpoint image and a mode in which a camera path is generated using the movement time in accordance with the user operation acquired in the acquisition step.

2. A method for controlling an information processing apparatus comprising:

11. An information processing device in an image processing system that arranges a plurality of imaging devices around a subject, captures images, and generates a virtual viewpoint image from the captured images, an acquisition step of acquiring a user operation; a storing step of storing information relating to an ID, time information, and posture information of the virtual camera together with a position of the virtual camera in a virtual viewpoint image generated from a plurality of images captured by a plurality of imaging devices, and also storing a movement time different from the time information; a generating step of generating a camera path from the positions of the plurality of virtual cameras held; a switching step of switching between a mode in which a camera path is generated based on time information recorded together with the position of a virtual camera for generating the virtual viewpoint image and a mode in which a camera path is generated using the movement time in accordance with a user operation acquired in the acquisition step; A program for causing an information processing device to execute the above.

Images