Information processing device, information processing method, and program

Abstract

To make it possible to efficiently correct a camera path of a virtual viewpoint video image.SOLUTION: An information processing device sets a partial section in a camera path of a virtual viewpoint video image as a correction object on the basis of a user instruction, performs a correction process on the basis of a keyframe after correction in the set partial section, and generates a partial camera path corresponding to the partial section. Then generates a virtual viewpoint corresponding to the generated partial camera path.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating and editing a camera path for a virtual viewpoint video. [Background technology]

[0002] A technology that captures images using imaging devices (real cameras) arranged at multiple different positions and generates a virtual viewpoint video that represents the view from a virtual camera that does not actually exist from the multiple captured images (generally called "multiple viewpoint images" or "multiple viewpoint images") has been attracting attention. When generating a virtual viewpoint video, a user operates a joystick or the like to change the position and attitude of the virtual camera in an imaging space reproduced by CG, and specifies the movement path of the virtual camera (generally called the "camera path"). One method for specifying this camera path is the key frame method. The key frame method is a method in which multiple reference frames (called "key frames") associated with arbitrary virtual viewpoints are set, and the camera path is obtained by interpolating between the multiple set key frames (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-144456 A Summary of the Invention [Problem to be solved by the invention]

[0004] When obtaining a camera path by the key frame method, the user first sets key frames from the start frame to the end frame in the video time to be generated, and specifies the position and orientation of the virtual camera at the time of each key frame. After that, an interpolation process is performed to determine the position and orientation of the virtual camera at the time between the set key frames, and the camera path is completed. Therefore, the user cannot know what kind of virtual viewpoint video will actually be created until the parameters of frames other than the key frames (intermediate frames) are determined by the interpolation process between all the set key frames. And even if the user does not like the video of a part of the completed virtual viewpoint video (between specific key frames), conventionally, the user has to make corrections to all the key frames from the start time to the end time of the virtual viewpoint video. Therefore, it takes a lot of time and effort to correct the camera path. [Means for solving the problem]

[0005] The information processing device according to the present disclosure includes a setting means for setting a plurality of key frames linking parameters representing a position and line of sight of the virtual viewpoint determined based on a user instruction for specifying a virtual viewpoint related to a virtual viewpoint video generated based on a plurality of captured images obtained by a plurality of imaging devices with frames representing specific times during a period in which imaging was performed by the plurality of imaging devices; a first generation means for acquiring parameters representing a position and line of sight of the virtual viewpoint in intermediate frames between key frames by an interpolation process based on the plurality of set key frames and linking the parameters to the intermediate frames to generate a camera path showing a transition of the virtual viewpoint; and a second generation means for generating a virtual viewpoint video corresponding to the camera path. When the generated virtual viewpoint video is to be modified, the setting means sets a partial section of the camera path as a modification target based on a user instruction, the first generating means performs the interpolation process based on modified key frames in the set partial section to generate a partial camera path corresponding to the partial section, and the second generating means generates a virtual viewpoint video corresponding to the generated partial camera path. Effect of the Invention

[0006] According to the present disclosure, it becomes possible to efficiently correct the camera path of a virtual viewpoint image. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing device. [Diagram 2] FIG. 2 is a block diagram showing an example of a software configuration of the information processing device. [Diagram 3] 13A to 13C are diagrams showing examples of UI screens. [Figure 4] 4 is a flowchart showing the flow of operations in an information processing device. [Diagram 5] 13A and 13B are diagrams showing an example of a UI screen. [Figure 6] FIG. 13 is a diagram showing an example of a UI screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the present invention will be described in detail based on preferred embodiments thereof with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings. Note that the same configurations will be described with the same reference numerals.

[0009] [Embodiment 1] <Hardware configuration> 1 is a block diagram showing an example of a hardware configuration of an information processing device 1 that generates a camera path based on a user operation and further generates a virtual viewpoint image based on the camera path. The information processing device 1 has a CPU 101, a ROM 102, a RAM 103, an auxiliary storage device 104, an operation unit 105, a communication I / F 106, a display control unit 208, and a bus 108.

[0010] The CPU 101 uses computer programs and data stored in the ROM 102 and the RAM 103 to control the entire information processing device 1 and realizes each functional unit shown in FIG. 2(b) described later. The information processing device 1 may have one or more dedicated hardware components different from the CPU 101, and at least a part of the processing by the CPU 101 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor). The ROM 102 stores programs that do not require modification. The RAM 103 temporarily stores programs and data supplied from the auxiliary storage device 104, and data supplied from the outside via the communication I / F 106. The auxiliary storage device 104 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

[0011] The operation unit 105 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and inputs various instructions to the CPU 101 in response to operations by a user. The communication I / F 106 is used for communication with an external device of the information processing device 1. For example, when the information processing device 1 is connected to an external device by wire, a communication cable is connected to the communication I / F 106. When the information processing device 1 has a function of wireless communication with an external device, the communication I / F 106 is provided with an antenna. The display control unit 208 is composed of, for example, a liquid crystal display, an LED, etc., and displays a GUI (Graphical User Interface) for a user to operate the information processing device 1. The CPU 101 operates as a display control unit that controls the display control unit 208 and an operation control unit that controls the operation unit 105. The bus 108 connects each unit of the information processing device 1 to transmit information. In this embodiment, the display unit 107 and the operation unit 105 are assumed to exist inside the information processing device 1, but at least one of the display unit 107 and the operation unit 105 may exist as a separate device outside the information processing device 1.

[0012] <Software configuration> 2 is a functional block diagram showing the software configuration (logical configuration) of the information processing device 1. The information processing device 1 is composed of a camera parameter generation unit 201, a key frame setting unit 202, a clip data storage unit 203, a camera path generation unit 204, a camera path management unit 205, a correction interval setting unit 206, a rendering unit 207, and a display control unit 208. Each functional unit will be described below.

[0013] The camera parameter generating unit 201 generates camera parameters of a virtual camera that does not actually exist in the imaging space based on an operation signal (hereinafter referred to as a "camera operation signal") that specifies the position and attitude of the virtual camera input by the user operating a joystick or the like. FIG. 3(a) is an example of a user interface screen (UI screen) used when the user operates a joystick or the like to specify the position and attitude of the virtual camera. An icon that two-dimensionally represents the position and attitude of the virtual camera is displayed on a CG image in a state in which a soccer field, which is the imaging space, is overlooked, and is associated with a start frame "20:30:45:00" as a key frame. Through such a UI screen, the user specifies the position and attitude of the virtual camera while associating it with the key frame. The camera parameters of this embodiment include parameters such as zoom and time in addition to the position and attitude of the virtual camera. In this case, the position of the virtual camera is indicated by three-dimensional coordinate information according to a Cartesian coordinate system of three axes, for example, the X axis, the Y axis, and the Z axis. The origin at that time is an arbitrary position in the three-dimensional space of the imaging target (for example, the center of the field of the stadium). The attitude of the virtual camera (the line of sight direction of the virtual viewpoint) is indicated by the angle it forms with the three axes of pan, tilt, and roll. The zoom is indicated by, for example, the focal length. The time is indicated by a time code. Zoom and time are each parameters of one axis. Therefore, the camera parameters in this embodiment have parameters of eight axes, namely, X coordinate, Y coordinate, Z coordinate, pan, tilt, roll, zoom, and time code. Note that the parameters may include parameters that define elements other than these eight axes, or may not include all of the parameters of these eight axes. The camera parameters determined based on the user's operation are sent to the key frame setting unit 202.

[0014] The key frame setting unit 202 associates the camera parameters input from the camera parameter generating unit 201 with a frame at a specific time designated by the user, and sets it as a key frame. Here, the frame at the specific time is a frame selected by the user from among a group of frames constituting a multi-viewpoint video during a period in which multiple imaging devices (not shown) perform synchronous imaging. Here, at least two key frames must be set, a start frame and an end frame corresponding to the start point and end point, respectively, of the target virtual viewpoint video. When a key frame is set based on a user instruction, information on the set key frame (key frame setting information) is output to the clip data storage unit 203.

[0015] The clip data storage unit 203 lists and stores a series of set key frames in a predetermined order based on the key frame setting information received from the key frame setting unit 202. Hereinafter, this listed data will be referred to as "clip data." When saving clip data, management information (clip ID), such as a number that can uniquely identify the data, is added. The time of each key frame from the start point to the end point in the clip data does not need to be unidirectional, and the key frames are read in the order in the list, allowing time rewinding and pausing. Furthermore, the order of the key frames in the list is not fixed, and can be rearranged.

[0016] The camera path generating unit 204 acquires clip data to be processed from the clip data storage unit 203, and derives the position and orientation of the virtual camera in frames (intermediate frames) between key frames included in the clip data by interpolation processing to generate a camera path. Representative examples of the interpolation method include, but are not limited to, spline interpolation and linear interpolation. The camera path is specified by a plurality of frames and a timeline, and camera parameters of the virtual camera are linked to each frame. The timeline is composed of a time code that expresses hours, minutes, seconds, and frames in the format of "hh:mm:ss:ff". For example, if the subject of imaging is a sports game such as soccer, the start time is expressed as "00:00:00:00". The number of frames included in the timeline is determined by the number of images to be played per second (frame rate). For example, if the frame rate is 60 fps (frames per second), 60 frames per second are included in the timeline. For each key frame indicated by the clip data among the frames constituting the camera path, information indicating that the frame is a key frame (e.g., a flag, etc.) is added, and the frame is distinguished from frames other than key frames. The generated camera path is sent to the camera path management unit 205.

[0017] The camera path management unit 205 manages the camera paths generated by the camera path generation unit 204. Specifically, the camera path management unit 205 assigns management information (path ID) such as a uniquely identifiable number to the generated camera path, and stores and manages the camera path by linking it with the clip ID of the clip data corresponding to the camera path.

[0018] The correction section setting unit 206 sets a key frame corresponding to a portion to be corrected in the generated virtual viewpoint video, based on a user selection.

[0019] The rendering unit 207 acquires the camera path to be processed from the camera path management unit 205, and performs rendering according to the camera path to generate a virtual viewpoint video. The generated virtual viewpoint video is displayed on the display unit 107 by the display control unit 208.

[0020] The functional units of the information processing device 1 described above may be distributed among a plurality of information processing devices, and the information processing device may be configured as a system made up of the plurality of information processing devices.

[0021] <Operation flow of information processing device> Next, the flow of operations of the information processing device according to this embodiment will be described with reference to the flowchart shown in Fig. 4. In the following description, the symbol "S" means step.

[0022] In S 401 , the camera parameter generating unit 201 generates the above-mentioned camera parameters based on a camera operation signal input from the operation unit 105 .

[0023] In S402, the key frame setting unit 202 associates the frame related to the frame designation information input from the operation unit 105 with the camera parameters generated in S401 and sets it as a key frame.

[0024] In S403, the key frame storage unit 203 adds the key frames set in S402 to a list in a predetermined order (for example, ascending order or descending order) to generate clip data (key frame list).

[0025] In S404, the process to be executed next is allocated depending on whether or not a key frame has been set as the end frame. If a user operation (e.g., an instruction to start generating a camera path) indicating that the setting of the end frame has been completed is detected, S405 is executed. A clip ID is assigned to the clip data at the time the end frame is set, and the clip data storage unit 203 stores the clip data in the RAM 103 or the like. On the other hand, if the setting of the end frame has not been completed, the process returns to S401 and the same process is continued. FIG. 3(b) shows a state in which a total of six frames from the start frame "20:30:45:00" to the end frame "20:30:54:30" are set as key frames on the UI screen of FIG. 3(a) described above. Now, it is assumed that the clip data consisting of these six frames is assigned a clip ID of "No. 1" and stored.

[0026] In S405, the camera path generating unit 204 reads out the saved clip data from the RAM 213 etc., and executes an interpolation process for the key frames indicated by the clip data to generate a camera path indicating the transition of the virtual camera. FIG. 3(c) is a diagram showing a state in which the flow line of the virtual camera, which is a result of the interpolation process for the six key frames shown in FIG. 3(b) described above, is superimposed and displayed on the UI screen by a dashed line 301. The camera path generated in this manner is stored and managed by the camera path managing unit 205. In this case, the camera path may be stored and managed together with the graphics data indicating the superimposed display state shown in FIG. 3(c).

[0027] In S406, the rendering unit 207 performs rendering based on the camera path generated in S405 to generate a virtual viewpoint video. Specifically, three-dimensional shape data of a foreground object is generated based on a foreground image obtained by performing foreground extraction from each of the multiple viewpoint images described above, and a coloring process is performed on the three-dimensional shape data according to the camera path to generate an image that represents the appearance from a virtual viewpoint. A method for generating three-dimensional shape data includes, for example, a volume intersection method, and is not particularly limited. The generated virtual viewpoint video is output to the display control unit 208.

[0028] In S407, the display control unit 208 displays the virtual viewpoint video, which is the rendering result, on the display unit 107. If the user who has viewed the virtual viewpoint video displayed on the display unit 107 decides that he or she wishes to modify the video, the user can continue to modify the virtual viewpoint video by, for example, pressing a modification button (not shown) on the UI screen.

[0029] In S408, processing is allocated depending on whether or not there is a user instruction to modify the virtual viewpoint video. If there is an instruction to modify, S409 is executed. On the other hand, if the user is satisfied with the completed virtual viewpoint video and presses an OK button (not shown) on the UI screen, generation of the virtual viewpoint video ends.

[0030] In S409, first, the display control unit 208 transitions the display of the UI screen to a state in which a user selection as to which section of the virtual viewpoint video is to be modified can be accepted. Then, the modification section setting unit 206 sets the section to be modified based on the user selection via the UI screen after the transition. FIGS. 5(a) and (b) are examples of UI screens for selecting the section to be modified. The UI screen in FIG. 5(a) has a two-pane configuration, and is composed of a clip data pane 501 that displays a list of clip data corresponding to the virtual viewpoint video to be modified, and a key frame pane 502 that displays a list of key frames included in the clip data. The UI screen in FIG. 5(b) has a three-pane configuration, and the above-mentioned clip data pane 501 is common, and two panes are provided in place of the key frame pane 502. Specifically, an overhead camera path pane 503 that expresses the movement line of the virtual camera in the camera path generated in S405 with graphics of an overhead angle, and a side camera path pane 504 that expresses the movement line of the virtual camera with graphics of a horizontal angle are provided. Now, in both of the UI screens of FIG. 5(a) and (b), the clip ID "No. 1" corresponding to the clip data of FIG. 3(b) is highlighted in the clip data pane 501. In the UI screen of FIG. 5(a), six key frames that are the contents of the highlighted clip data are displayed with check boxes in the key frame pane 502. In the UI screen of FIG. 5(b), six key frames that are the contents of the highlighted clip data are displayed with check boxes in the overhead camera path pane 503 and the side camera path pane 504. The user performs a selection operation to check the check box of the key frame that corresponds to the video portion that the user wants to correct via the UI screens shown in FIG. 5(a) and (b), and a correction target section consisting of one or more key frames is set based on the selection operation. In both of the UI screen examples of FIG. 5(a) and (b), three key frames, the second to the fourth, are selected, and these three consecutive frames are set as a correction target section.Then, the user modifies (edits) the key frames in the set section by performing the same operations as in S401 and S402 described above. Possible modifications include changing the position and orientation of the virtual camera associated with a specific key frame in the selected section, deleting a specific key frame in the selected section, and inserting a new key frame between specific key frames in the selected section. One or more key frames in the modified section are stored in the RAM 103 as provisional partial clip data. Note that two or more adjacent key frames are required for the interpolation process. Therefore, when only one key frame is set as the section to be modified and the position and orientation of the virtual camera are changed, or when only one key frame remains after modification, the multiple key frames including the one key frame and its adjacent key frame are stored as partial clip data.

[0031] In S410, the camera path generating unit 204 executes an interpolation process on the key frames included in the partial clip data saved in S409 to generate a partial camera path (hereinafter referred to as a "partial camera path") corresponding to the section to be corrected. FIG. 6 is an example of a UI screen showing the result of the interpolation process on the key frames included in the partial clip data. The UI screen in FIG. 6 has a two-pane configuration, and is composed of a preview pane 511 that displays a list of the key frames after correction, and a result pane 512 that displays the results of the interpolation process on the key frames after correction. With such a UI screen, the user can check only the partial camera path that corresponds to the section to be corrected out of the camera paths corresponding to the virtual viewpoint video generated in S406.

[0032] In S412, the rendering unit 207 performs rendering based on the partial camera path generated in S411 to generate a partial virtual viewpoint video corresponding to a part of the section after correction. After generation, the process returns to S407, and the display control unit 208 displays the generated partial virtual viewpoint video on the display unit 107. In this way, the user can check only the video corresponding to the correction portion. If the user is satisfied with the video, the user ends the generation of the virtual viewpoint video by pressing an OK button (not shown) on the UI screen. At this time, the corrected partial camera path is reflected in the camera path of the entire video managed by the camera path management unit 205. On the other hand, if the user decides that he or she wants to further correct the video, the user continues the work of correcting the virtual viewpoint video by pressing a correction button (not shown) on the UI screen.

[0033] The above is the flow of operations of the information processing device according to this embodiment. In the above-mentioned UI screen examples of Figs. 5(a) and (b), three key frames are selected by checking all of the second to fourth key frames. A similar selection may be realized, for example, by checking only the second and fourth key frames. That is, the user may select the start and end frames to be corrected, and the key frames in the range between these frames may be set as the section to be corrected.

[0034] In addition, in this embodiment, the partial camera path is generated using the corrected key frame, but the present invention is not limited to this. For example, a camera path may be generated based on a key frame of a section not set by the user and a corrected key frame, and only the correction target section set by the user may be extracted from the camera path to generate a partial camera path. This process is effective in an interpolation method in which the position and orientation of the virtual camera in the key frame outside the correction target section is used to calculate the camera path of the correction target section, such as spline interpolation. This process makes the virtual viewpoint video generated using the partial camera path corresponding to the correction target section and the video corresponding to the correction target section in the virtual viewpoint video generated using clip data including the corrected key frame the same video. Therefore, the effort of correcting a part of the section desired by the user can be eliminated.

[0035] As described above, according to the method of the present embodiment, only a portion of the completed virtual viewpoint video desired by the user is corrected and interpolated to be re-imaged, and the user can check only the image of the corrected portion. As a result, the time and effort required for generating a camera path can be reduced, and the burden on the user can be reduced.

[0036] (Other Examples) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0037] The present disclosure also includes the following configurations and methods.

[0038] [Configuration 1] a setting means for setting a key frame that links parameters representing a position and a line of sight of a virtual viewpoint, which are determined based on a user instruction for specifying a virtual viewpoint related to a virtual viewpoint video generated based on a plurality of captured images obtained by a plurality of imaging devices, with a frame that represents a specific time during a period in which imaging was performed by the plurality of imaging devices; a first generating means for acquiring parameters representing a position and a line of sight direction of the virtual viewpoint in an intermediate frame between the key frames by an interpolation process based on the set key frames, and linking the parameters to the intermediate frame to generate a camera path showing a transition of the virtual viewpoint; A second generating means for generating a virtual viewpoint image corresponding to the camera path; Equipped with When correcting the generated virtual viewpoint image, The setting means sets a section of the camera path as a correction target based on a user instruction, The first generating means performs the interpolation process based on the corrected key frames in the set partial section to generate a partial camera path corresponding to the partial section; The second generation means generates a virtual viewpoint image corresponding to the generated partial camera path. 23. An information processing apparatus comprising:

[0039] [Configuration 2] The information processing device according to configuration 1, further comprising a display control means for displaying a UI screen for accepting the user instruction on a display means.

[0040] [Configuration 3] When the second generation means generates the virtual viewpoint video, the display control means displays the generated virtual viewpoint video on the display means; The user instruction to select the partial section is given based on the displayed virtual viewpoint video. 3. The information processing device according to configuration 2.

[0041] [Configuration 4] a key frame included in the camera path is displayed on the UI screen for the user to specify the partial section; the setting means sets one or more key frames selected by a user from the displayed key frames; 4. The information processing device according to configuration 2 or 3.

[0042] [Configuration 5] the setting means sets a plurality of key frames included in a range between two key frames selected by a user as a start frame and an end frame from among the displayed key frames; 5. The information processing device according to configuration 4.

[0043] [Configuration 6] 6. The information processing device according to configuration 4 or 5, wherein the UI screen for a user to specify the partial section displays a list of key frames included in the camera path.

[0044] [Configuration 7] on the UI screen for a user to specify the partial section, keyframes included in the camera path are displayed in a graphical manner together with icons representing positions of virtual viewpoints and viewing directions specified by the parameters associated with each of the keyframes, the setting means sets one or more key frames selected by a user from the displayed key frames; 4. The information processing device according to configuration 2 or 3.

[0045] [Configuration 8] The information processing device according to any one of configurations 1 to 7, wherein the modification is to change a position and a line of sight direction of a virtual viewpoint associated with at least one of the set key frames of the partial section.

[0046] [Configuration 9] 8. The information processing device according to any one of configurations 1 to 7, wherein the modification is to delete at least one of the set key frames in the partial section.

[0047] [Configuration 10] 8. The information processing device according to any one of configurations 1 to 7, wherein the correction is to insert a new key frame between two adjacent key frames among the set key frames in the partial section.

[0048] [Configuration 11] The information processing device according to configuration 8 or 9, characterized in that, when the number of corrected key frames in the set partial section is one, the first generation means performs an interpolation process on a plurality of key frames including adjacent key frames of the one key frame to generate the partial camera path.

[0049] [Method 1] a setting step of setting a key frame that links parameters representing a position and a line of sight of a virtual viewpoint determined based on a user instruction for specifying a virtual viewpoint related to a virtual viewpoint video generated based on a plurality of captured images obtained by a plurality of imaging devices, and a frame that represents a specific time during a period in which imaging was performed by the plurality of imaging devices; a first generation step of acquiring parameters representing a position and a line of sight direction of the virtual viewpoint in an intermediate frame between the key frames by an interpolation process based on the set key frames, linking the parameters to the intermediate frame, and generating a camera path showing a transition of the virtual viewpoint; a second generation step of generating a virtual viewpoint image corresponding to the camera path; Equipped with When correcting the generated virtual viewpoint image, In the setting step, a part of the camera path is set as a correction target based on a user instruction; In the first generation step, the interpolation process is performed based on the corrected key frames in the set partial section to generate a partial camera path corresponding to the partial section; In the second generation step, a virtual viewpoint image corresponding to the generated partial camera path is generated. 23. An information processing method comprising:

[0050] [Configuration 12] A program for causing a computer to function as the information processing device according to any one of configurations 1 to 11.

Claims

1. a setting means for setting a plurality of key frames each linking a parameter representing a position and a line of sight of a virtual viewpoint determined based on a user instruction for specifying a virtual viewpoint related to a virtual viewpoint video generated based on a plurality of captured images obtained by a plurality of imaging devices with a frame representing a specific time during a period in which imaging was performed by the plurality of imaging devices; a first generating means for acquiring parameters representing a position and a line of sight direction of the virtual viewpoint in an intermediate frame between the key frames by an interpolation process based on a plurality of set key frames, and linking the parameters to the intermediate frame to generate a camera path indicating a transition of the virtual viewpoint; A second generating means for generating a virtual viewpoint image corresponding to the camera path; Equipped with When correcting the generated virtual viewpoint image, The setting means sets a section of the camera path as a correction target based on a user instruction, The first generation means performs the interpolation process based on the corrected key frames in the set partial section to generate a partial camera path corresponding to the partial section; The second generation means generates a virtual viewpoint image corresponding to the generated partial camera path.

23. An information processing apparatus comprising:

2. 2. The information processing apparatus according to claim 1, further comprising a display control unit for displaying a UI screen for accepting the user instruction on a display unit.

3. When the second generation means generates the virtual viewpoint video, the display control means displays the generated virtual viewpoint video on the display means; The user instruction to select the partial section is given based on the displayed virtual viewpoint video.

3. The information processing apparatus according to claim 2.

4. a key frame included in the camera path is displayed on the UI screen for a user to specify the partial section; the setting means sets one or more key frames selected by a user from the displayed key frames; 3. The information processing apparatus according to claim 2.

5. the setting means sets a plurality of key frames included in a range between two key frames selected by a user as a start frame and an end frame from among the displayed key frames; 5. The information processing apparatus according to claim 4.

6. The information processing device according to claim 4 , wherein the UI screen for the user to specify the partial section displays a list of key frames included in the camera path.

7. on the UI screen for a user to specify the partial section, keyframes included in the camera path are displayed in a graphical manner together with icons representing positions of virtual viewpoints and viewing directions specified by the parameters associated with each of the keyframes, the setting means sets one or more key frames selected by a user from the displayed key frames; 3. The information processing apparatus according to claim 2.

8. The information processing apparatus according to claim 1 , wherein the modification is a change in a position and a line of sight of a virtual viewpoint associated with at least one of the set key frames of the partial section.

9. The information processing apparatus according to claim 1 , wherein the modification is to delete at least one of the set key frames in the partial section.

10. The information processing apparatus according to claim 1 , wherein the correction is performed by inserting a new key frame between two adjacent key frames among the set key frames in the partial section.

11. The information processing device according to claim 8, characterized in that, when the number of corrected key frames in the set partial section is one, the first generation means performs an interpolation process on a plurality of key frames including adjacent key frames of the one key frame to generate the partial camera path.

12. a setting step of setting a key frame that links parameters representing a position and a line of sight of a virtual viewpoint determined based on a user instruction for specifying a virtual viewpoint related to a virtual viewpoint video generated based on a plurality of captured images obtained by a plurality of imaging devices, and a frame that represents a specific time during a period in which imaging was performed by the plurality of imaging devices; a first generation step of acquiring parameters representing a position and a line of sight direction of the virtual viewpoint in an intermediate frame between the key frames by an interpolation process based on the set key frames, linking the acquired parameters to the intermediate frame, and generating a camera path showing a transition of the virtual viewpoint; a second generation step of generating a virtual viewpoint image corresponding to the camera path; Equipped with When correcting the generated virtual viewpoint image, In the setting step, a part of the camera path is set as a correction target based on a user instruction; In the first generation step, the interpolation process is performed based on the corrected key frames in the set partial section to generate a partial camera path corresponding to the partial section; In the second generation step, a virtual viewpoint image corresponding to the generated partial camera path is generated.

23. An information processing method comprising:

13. A program for causing a computer to execute the information processing method according to claim 12.

Images