Information processing device, information processing method, and program

Abstract

To provide an easy setup of virtual viewpoint information relating to a reproduction of a virtual viewpoint image.SOLUTION: An image processing device 102 comprises a manually control unit 304 which includes a key frame setup section 401 for producing a key frame with a parameter and a time matched, where the parameter indicates a position of and a direction from the virtual viewpoint determined based on an operation by a user to designate a virtual viewpoint relating to a virtual viewpoint video made with a plurality of images taken by a plurality of imaging devices, and where the time is an imaging period by the plurality of imaging devices, a listing section 402 for determining a reproduction direction of a plurality of key frames, and a camera pass creating section 404 for producing virtual viewpoint information indicating a transition of the virtual viewpoint based on the plurality of key frames and the reproduction direction.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a technique for creating and editing camera path information for virtual viewpoint images and CG animation. [Background technology]

[0002] Recently, a technology that installs multiple imaging devices at different positions, captures images from multiple viewpoints synchronously, and generates a virtual viewpoint image using the multiple viewpoint images obtained by the capture has been attracting attention. The technology that generates a virtual viewpoint image from multiple viewpoint images can give a user a high sense of realism compared to normal images. For example, in a rugby match, it is possible to obtain an image that connects a cut that captures the scene where players carry the ball toward the goal from a bird's-eye view and a cut that captures the try scene from a specific player's subjective view (for example, passing by the side of the opponent player from the fullback's perspective). Furthermore, it is possible to play back the image from the bird's-eye view (forward playback), rewind the time and play it back (reverse playback), and then play back the image from the subjective view (forward playback). This makes it possible to provide viewers with easy-to-understand images and explanations that have not been available in the past. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-60280 A Summary of the Invention [Problem to be solved by the invention]

[0004] When generating a virtual viewpoint video, it is necessary to set virtual viewpoint information that represents the transition of a viewpoint (virtual viewpoint) that is virtually placed in a three-dimensional space. As a method for creating virtual viewpoint information when obtaining a virtual viewpoint video including the above-mentioned cut switching, for example, there is a key frame method. The key frame method is a method of setting a plurality of reference frames (called "key frames") to which any virtual viewpoint is associated, and obtaining camera path information by interpolating between the set plurality of key frames. In this key frame method, the set key frames and the virtual viewpoint are associated with each other using a time code so that the corresponding relationship between them can be specified. In this regard, Patent Document 1 discloses a technology for seamlessly progressing a CG animation that is rendered in real time in synchronization with a video material that employs a time code. However, the technology of Patent Document 1 does not assume that time will be rewound during the progress of the CG animation. Therefore, it is difficult to set virtual viewpoint information for generating a virtual viewpoint video in which forward playback and reverse playback are combined for the same scene as described above. In this way, depending on the playback method of the virtual viewpoint video, there is a risk that it may be difficult to set the virtual viewpoint information.

[0005] The technology disclosed herein aims to make it possible to easily set virtual viewpoint information related to the playback of virtual viewpoint video. [Means for solving the problem]

[0006] The information processing device according to the present disclosure generates a key frame in which parameters representing a position of a virtual viewpoint and a direction from the virtual viewpoint, which are determined based on a user operation for specifying a virtual viewpoint related to a virtual viewpoint video generated based on a plurality of captured images obtained by imaging performed by a plurality of imaging devices, are associated with a time during a period in which imaging is performed by the plurality of imaging devices. multiple Generate No. 1 A generating means; a second generating means for generating a first list in which a plurality of first key frame information corresponding to a plurality of first key frames, each of which has first information on a time axis set, among the plurality of key frames, is arranged in ascending order of time, and a second list in which a plurality of second key frame information corresponding to a plurality of second key frames, each of which has second information on a time axis set, among the plurality of key frames, is arranged in descending order of time; Based on the above, virtual viewpoint information representing a transition of the virtual viewpoint is generated. Third The present invention is characterized by having a generating means. Effect of the Invention

[0007] According to the technology of the present disclosure, it becomes possible to easily set virtual viewpoint information related to the playback of virtual viewpoint video. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system for generating a virtual viewpoint video. [Diagram 2] FIG. 1 illustrates a hardware configuration of an image processing apparatus. [Diagram 3] Block diagram showing the functional configuration of an image processing device [Figure 4] Block diagram showing the functional configuration related to creating and editing camera paths [Diagram 5] A diagram showing the process of creating a keyframe list [Figure 6] Diagram illustrating the drawbacks of creating a reverse video clip in timecode mode [Figure 7] A flowchart showing a control flow for creating camera path information taking into account the playback direction of a video clip. [Figure 8] A diagram showing an example of a GUI and controller for creating and editing camera path information. [Figure 9] (a) to (c) are diagrams showing the GUI for listing key frames. [Figure 10] Figure showing GUI for listing keyframes [Figure 11] Figure showing GUI for listing keyframes [Figure 12] Figure showing GUI for listing keyframes [Figure 13] Figure showing GUI for listing keyframes [Figure 14] Figure showing GUI for listing keyframes [Figure 15] Figure showing GUI for listing keyframes DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a mode for carrying out the present embodiment will be described with reference to the drawings, etc. Note that the following embodiment does not limit the technology of the present disclosure, and all configurations described in the following embodiment are not necessarily essential to the means for solving the problem.

[0010] [Embodiment 1] First, a brief overview of a virtual viewpoint image will be given. A virtual viewpoint image is an image that represents a view from a virtual camera viewpoint (virtual viewpoint) that is different from an actual camera viewpoint, and is also called a free viewpoint image. The virtual viewpoint is set by a method in which the user directly specifies the viewpoint by operating a controller, or selects from a plurality of virtual viewpoint candidates that are set in advance. Note that while the virtual viewpoint image includes both moving images and still images, the following embodiment will be described on the premise of moving images, and a virtual viewpoint image based on moving images will be expressed as a "virtual viewpoint video."

[0011] <System configuration> FIG. 1 is a diagram showing an example of the configuration of an image processing system for generating a virtual viewpoint video according to this embodiment. The image processing system 100 includes a plurality of imaging devices (cameras) 101, an image processing device 102, a viewpoint input device 103, and a display device 104. In the image processing system 100, the image processing device 102 generates a virtual viewpoint video based on a plurality of captured images (multiple viewpoint videos) obtained by synchronous shooting of the plurality of imaging devices 101 and a virtual viewpoint set by the viewpoint input device 103, and displays the generated video on the display device 104. The area subject to synchronous shooting by the plurality of imaging devices 101 is, for example, a field of a stadium where a sport such as rugby or soccer is played, or a stage of a concert hall where a play or the like is performed. For example, when generating a virtual viewpoint video for a rugby match held in a stadium, the imaging devices 101 are installed at different positions so as to surround the field in the stadium, and each imaging device 101 takes an image on the field in a time-synchronized manner. However, the plurality of imaging devices 101 do not necessarily have to be installed around the entire circumference of the area to be shot, and when there is a restriction on the installation location, the imaging devices 101 may be installed only in a direction of a part of the area to be shot. Each imaging device 101 is realized by, for example, a digital video imaging device equipped with a video signal interface such as a serial digital interface (SDI). Each imaging device 101 adds time information such as a time code to the video signal it outputs and transmits it to the image processing device 102.

[0012] The image processing device 102 generates a virtual viewpoint video based on images captured by multiple imaging devices 101 synchronously. The image processing device 102 generates a virtual viewpoint video by performing processes such as generating three-dimensional shape data of an object and rendering using the images captured by each imaging device 101 acquired via a network cable or the like. The image processing device 102 also sets a virtual viewpoint (creates camera path information) based on a user operation signal input from a viewpoint input device 103. The functions of the image processing device 102 will be described in detail later.

[0013] The viewpoint input device 103 is an input device having a controller such as a joystick. An operator performs various setting operations related to the virtual viewpoint via the viewpoint input device 103.

[0014] The display device 104 acquires and displays image data (UI screen data for a graphical user interface and virtual viewpoint image data) sent from the image processing device 102. The display device 104 is realized by, for example, a liquid crystal display, an LED, or the like.

[0015] In this specification, a series of explanations will be given taking the case of generating a virtual viewpoint video as an example, but the following contents can also be applied to generating and editing a camera path for a CG video. In that case, the image processing system includes a CG renderer instead of a module that generates a virtual viewpoint video. Then, a viewpoint that can be moved to any position within the CG space can be treated as the virtual viewpoint in this embodiment.

[0016] <Hardware configuration> 2 is a diagram showing a hardware configuration of the image processing device 102. The image processing device 102, which is an information processing device, includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, an operation unit 215, a communication I / F 216, and a bus 217.

[0017] The CPU 211 realizes each function of the image processing device 102 by controlling the entire image processing device 102 using computer programs and data stored in the ROM 212 or the RAM 213. The image processing device 102 may have one or more dedicated hardware different from the CPU 211 or a GPU (Graphics Processing Unit). At least a part of the processing by the CPU 211 may be performed by the GPU or the dedicated hardware. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0018] The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, and data supplied from the outside via the communication I / F 217. The auxiliary storage device 214 is constituted by, for example, a hard disk drive, and stores various data such as image data and volume data.

[0019] The display unit 215 is configured with, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) and the like for the user to operate the image processing device 102 . The operation unit 215 is composed of, for example, a keyboard, a mouse, etc., and inputs various instructions to the CPU 211 in response to operations by a user. The CPU 211 operates as a display control unit that controls the display device 104, and an operation control unit that controls the operation unit 215. The communication I / F 216 is used for communication with an external device of the image processing device 102. For example, when the image processing device 102 is connected to an external device by wire, a communication cable is connected to the communication I / F 216. When the image processing device 102 has a function of wirelessly communicating with an external device, the communication I / F 216 is equipped with an antenna.

[0020] The bus 217 transmits information by connecting each unit of the image processing device 102. Note that, in this embodiment, the viewpoint input device 103 and the display device 104 are provided as external devices, but either of them may be provided internally as one of the functional units of the image processing device 102.

[0021] <Functional configuration> 3 is a block diagram showing the functional configuration of the image processing device 102. The image processing device 102 includes an image acquisition unit 300, a foreground / background separation unit 301, a storage unit 302, a virtual viewpoint video generation unit 303, and a main control unit 304.

[0022] The image acquisition section 300 acquires data of images (multiple viewpoint images) obtained by the multiple image capture devices 101 capturing images synchronously.

[0023] The foreground / background separation unit 301 extracts specific objects (hereinafter referred to as "foreground objects") that are in the foreground, such as players and referees, from the video of each of the imaging devices 101 on a frame-by-frame basis. The foreground / background separation unit 301 stores an image showing the extracted foreground object as a foreground image in the storage unit 302. The method of extracting the foreground object is not limited. For example, a background difference method is a method of extracting a foreground object by comparing pixel values ​​of a frame (still image) in a video and a background image showing the background part of the frame to extract a foreground object in the frame. The background image is an image in a state in which no foreground object exists. For example, when shooting a rugby match, an image obtained by shooting before the start of the match when there are no players on the field is used. The foreground / background separation unit 301 stores the background image and the foreground image together with shooting position information indicating the three-dimensional position of the imaging device 101. The three-dimensional position is defined by the position in each direction corresponding to the height, width, and depth in the three-dimensional space of the subject to be shot. The shooting position information according to the world coordinate system can be obtained, for example, by shooting a marker or the like that has been set in advance and performing three-dimensional image measurement of the marker, etc. The background image and the foreground image are stored in the RAM 213 or the auxiliary storage device 214 by the storage unit 302.

[0024] The main control unit 304 creates camera path information based on a user operation via the viewpoint input device 103, and provides it to the virtual viewpoint video generation unit 303. Here, the camera path information is information indicating a transition of the viewpoint (virtual viewpoint) of a non-existent virtual camera corresponding to a virtual viewpoint video in which virtual viewpoint images in units of frames are arranged in a time series. The position and orientation of the camera in the camera path information correspond to the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint, respectively. Moreover, the camera path information is a parameter indicating the position and orientation of the camera, and is composed of a plurality of parameters that are continuous in time. In other words, the camera path information is virtual viewpoint information indicating the transition over time of the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint. The created camera path information is output to the virtual viewpoint video generation unit 303.

[0025] The virtual viewpoint video generator 303 generates three-dimensional shape data of the foreground object based on the foreground image, and performs coloring (texture pasting) on ​​the three-dimensional shape data according to the camera path information to generate an image that represents the view from the virtual viewpoint. The method of generating the three-dimensional shape data is not particularly limited, and may be, for example, a shape-from-silhouette method. When generating the three-dimensional shape data, the position and size of the foreground object in the shooting target space are also specified. The generated virtual viewpoint video is output to the display device 104.

[0026] Note that there may be a plurality of image processing devices 102. For example, the generation of the virtual viewpoint video may be performed in another image processing device having the functions of the image acquisition unit 300, the foreground / background separation unit 301, the storage unit 302, and the virtual viewpoint video generation unit 303 described above. In this case, the image processing device 102 provides the other image processing device with camera path information for generating the virtual viewpoint video. It is also possible to apply the image processing device 102 to editing the camera path of the CG video. In this case, a CG renderer may be provided instead of the virtual viewpoint video generation unit 303. In this case, the camera path created by the method described below represents the trajectory of the viewpoint in the CG image.

[0027] <Details about the main control unit> 4 is a block diagram showing an example of the functional configuration within the main control unit 304 related to creating and editing a camera path. The main control unit 304 is made up of a camera parameter generation unit 400, a key frame setting unit 401, a list creation unit 402, a list storage unit 403, a camera path creation unit 404, and a GUI control unit 405. Each unit will be described below.

[0028] The camera parameter generating unit 400 acquires from the viewpoint input device 103 an operation signal (hereinafter referred to as a "camera operation signal") that specifies the position and attitude of a virtual camera input by the user operating a joystick or the like. Then, based on the acquired camera operation signal, the camera parameters of a non-existent camera corresponding to the virtual viewpoint are set. The camera parameters in this embodiment include parameters such as the position and attitude of the virtual viewpoint, zoom, and time. In this case, the position of the virtual viewpoint is indicated by three-dimensional coordinate information according to a Cartesian coordinate system of three axes, for example, the X axis, the Y axis, and the Z axis. The origin at that time is an arbitrary position in the three-dimensional space of the subject to be photographed (the center of the field of the stadium in this embodiment). Also, the attitude of the camera corresponding to the virtual viewpoint is indicated by an angle formed with the three axes of pan, tilt, and roll. Also, the zoom is indicated by, for example, a focal length. Also, the time is indicated by a time code that sets the time when the game to be photographed started as 00:00:00:00 frame. The zoom and the time are each one-axis parameters. Therefore, the camera parameters in this embodiment have eight axis parameters, namely, X coordinate, Y coordinate, Z coordinate, pan, tilt, roll, zoom, and time code. Note that the camera parameters may include parameters defining elements other than these eight axes, and may not include all of these eight axis parameters. The camera parameters set based on the user's operation are sent to a key frame setting unit 401.

[0029] The key frame setting unit 401 generates a key frame in which the camera parameters input from the camera parameter generating unit 400 correspond to the time during which images are captured by the multiple image capturing devices 101, and sets the key frame in association with a frame designation signal. Here, the frame designation signal is a signal that instructs setting a specific frame selected by an operator from a group of frames constituting a multiple viewpoint video as a key frame, and is input from the viewpoint input device 103. At least two key frames must be set. Information on the set key frames is output to the listing unit 402.

[0030] The listing unit 402 arranges key frame information that can uniquely identify the key frames corresponding to the key frames set by the key frame setting unit 401 according to the arrangement mode set via the GUI, and creates a list in which the key frames are arranged in a predetermined order. The listing of key frames will be described later. The created key frame list is output to the list storage unit 403.

[0031] The list storage unit 403 stores the key frame list received from the list creation unit 402. The stored key frame list is read by a camera path creation unit 404 and used to create camera path information.

[0032] The camera path creation unit 404 creates camera path information based on the camera parameters input from the camera parameter generation unit 400 and the key frame list stored by the list storage unit 403. The camera path information is managed by a timeline consisting of a plurality of frames, and each frame is provided with information necessary for generating each image constituting the virtual viewpoint video, specifically, the time of the scene and the position and attitude of the virtual viewpoint. The time of the scene is expressed, for example, by a time code in which the time when the game to be filmed starts is set to 00:00:00:00. The position of the virtual viewpoint is expressed by three-dimensional coordinates based on the three axes of the X axis, Y axis, and Z axis described above. The attitude of the virtual viewpoint is expressed by the three angles of pan, tilt, and roll described above. The number of frames included in the timeline is determined by the number of images included per second (frame rate). For example, when the frame rate is 60 frames / second, 60 frames per second are included in the timeline. In setting and editing the camera path, frames are divided into two types: key frames and intermediate frames. A key frame is a frame in which the operator explicitly specifies the position, attitude, etc. of the virtual viewpoint. On the other hand, intermediate frames are frames that fill the gaps between key frames. The camera path creation unit 404 performs an interpolation process based on the key frames specified by the operator to determine the camera parameters corresponding to the intermediate frames. In this way, camera path information represented by a set of camera parameters that are continuous in time is obtained.

[0033] The GUI control unit 405 controls the graphical user interface and receives user instructions for creating and editing the above-mentioned camera path information.

[0034] The image processing device 102 including the above-mentioned units can also be used to create and edit a camera path for a CG image. In that case, a CG renderer should be provided instead of the virtual viewpoint image generating unit 303. In this case, the camera path created by the method described below represents the trajectory of the viewpoint in the CG image.

[0035] (Description of the problem to be solved) Here, the problem to be solved by the technology according to the present disclosure will be described in detail with reference to Fig. 5 and Fig. 6. Consider an operation mode (hereinafter referred to as "time code mode") in which key frames are automatically arranged in chronological ascending order along the time code as the key frame arrangement mode. In the following description, playback from the smallest time to the largest time code (ascending order) is defined as "forward playback", and playback from the largest time to the smallest time code (descending order) is defined as "reverse playback".

[0036] Fig. 5 is a diagram showing the process in which four key frames that have been set in sequence are automatically arranged in a time code mode to create a key frame list (hereinafter, simply referred to as a "list"). For ease of explanation, lists 501 to 504 shown in Fig. 5 only show the numbers given to each set key frame and the corresponding time codes.

[0037] List 501 shows the arrangement of key frame information in the virtual viewpoint video (video clip) to be finally created, and lists 502 to 504 show the arrangement of key frame information at each setting time point of key frames set sequentially. Also, the parts shown in bold in FIG. 5 show the key frame information of the key frames added at the second setting time point, the third setting time point, and the fourth setting time point, respectively. As described above, in the time code mode, the key frame information is arranged so as to be arranged in ascending order in chronological order along the time code. Therefore, when the third key frame with a time code of "20:05:53:15" is set, the key frame information of the key frame is automatically arranged at the position No. 2 (see list 503). Similarly, when the fourth key frame with a time code of "20:05:48:08" is set, the key frame information of the key frame is also arranged at the position No. 2 (see list 504). In this way, in the time code mode, the key frame information is arranged so as to be arranged from the smallest to the largest time code numbers from the top. Therefore, when creating a video clip with a forward playback direction, the operator does not need to manually rearrange the key frame information, which is convenient. However, when creating a video clip with a reverse playback direction, the time code mode is inconvenient. The following problem occurs, for example, when generating a camera path that has both forward and reverse playback. The reason for this is explained using Figure 6.

[0038] In FIG. 6, as in FIG. 5, a list 601 of video clips to be finally created is shown on the left side. In the video clip that the operator wants to create, No. 1 to 4 are video parts to be played forward, and No. 5 to 7 are video parts to be played backward. For No. 1 to 4 in forward playback, no problem occurs if key frames are arranged in ascending order along the time code (see list 602). However, if the fifth key frame information for reverse playback is arranged in ascending order, it will become like list 603, which does not match the arrangement of key frame information intended by the operator (see list 601). In this case, when the fifth key frame is set, the operator needs to rearrange the key frame information arranged at the position of No. 4 to the position of No. 5, as shown in list 604. Similarly, when the sixth key frame is set, the key frame information of the key frame is arranged at the position of No. 3 as shown in list 605, so the operator needs to rearrange it to the position of No. 6, as shown in list 606. Furthermore, when a seventh key frame is set, the key frame information for that key frame is placed at position No. 2 as shown in list 607, so the operator must reposition it at position No. 7 to achieve the desired state of list 601. The operator must recognize where in the list the key frame information corresponding to the newly added key frame is placed, and must repeatedly set key frames and rearrange the key frame information while being conscious of the correct placement, which places a heavy burden on the operator when creating a video clip.

[0039] The technology disclosed herein proposes a method for automatically and appropriately arranging information that identifies key frames that are set sequentially according to the playback direction of the desired video clip, in order to reduce the burden on the operator when creating camera path information.

[0040] (Creating camera path information) Next, the details of the process of creating camera path information in the main control unit 304 will be described. FIG. 7 is a flowchart showing a flow of control when creating camera path information considering the playback direction in a video clip to be created according to this embodiment. A series of processes shown in the flowchart in FIG. 7 is performed by the CPU 211 of the image processing device 102 expanding the program code stored in the ROM 212 into the RAM 213 and executing it. In addition, a part or all of the functions of the steps in this embodiment may be realized by hardware such as an ASIC or an electronic circuit. In addition, FIG. 8 is a diagram showing an example of a GUI and a controller for creating and editing camera path information. Two UI screens 801 and 802 are displayed on the display device 104, and a controller 803 constitutes an operation unit of the viewpoint input device 103. The UI screen 801 displays a multi-viewpoint image that is the basis of the virtual viewpoint video, and a virtual viewpoint video generated based on the camera path information. The UI screen 802 is used for setting the arrangement mode and key frames, checking the created camera path, and the like. For example, the operator can edit the camera path displayed on the UI screen 802 while checking an image representing the view from the virtual viewpoint in each frame being edited on the UI screen 801. In the following explanation, the symbol "S" means step.

[0041] In S701, a key frame arrangement mode is set according to an operation of an operator via a GUI. Figs. 9(a) to (c) are diagrams for explaining an example of a GUI. Figs. 9(a) and (b) show an example of a GUI for setting the listing of key frames. In Fig. 9(a), a "Key Order" field 901 in the upper left shows the name of the arrangement mode being set. When the operator clicks on field 901, a pull-down menu of options is displayed as shown in Fig. 9(b). The contents of each option are as follows: Time Code: Operation mode that arranges the time in ascending order according to the time code. Key Frame: Operation mode for arranging key frames in the order they are set. Play x Time Code: An operation mode that arranges the time sequence according to the time code while taking into account the playback direction.

[0042] The operator selects the desired placement mode from the above options displayed in the pull-down menu. Currently, the operator has selected "Play x Time Code" to create camera path information that takes the playback direction into consideration. The following item names are displayed under "Key Order". · "Sorting button (▲▼)" "Key frame number (Key No.) and time code (hh:mm:ss:ff)" ·"Play dir." "Virtual Camera Position"

[0043] Each time a key frame is set, information corresponding to each of these items is added to complete the key frame list. At this time, key frame information 902 corresponding to the set key frame is placed under field 901.

[0044] The "Sorting buttons (▲▼)" are used by the operator to manually sort the keyframe information in the list. Pressing the up button will swap the selected keyframe information with the keyframe information above. Similarly, pressing the down button will swap the selected keyframe information with the keyframe information below.

[0045] "Key frame number (Key No.)" indicates the arrangement order of the key frame information in the list, and is arranged in the order of playback in the created video clip. Below the key frame number is shown the time code corresponding to that key frame. Note that in the time code, "hh" represents hours, "mm" represents minutes, "ss" represents seconds, and "ff" represents the frame number. Currently, in Figure 9(a), only one key frame has been set with a time code of "58th frame, 8:05:46 p.m."

[0046] "Play direction" displays information about the play direction, specifically, whether forward play or reverse play.

[0047] "Camera position and orientation information (Camera Position)" indicates the position of the virtual viewpoint associated with the key frame using three coordinates, XYZ, and the orientation using three angles, Pan, Tilt, and Role.

[0048] The following description of steps S702 and after will be made on the assumption that "Play x Time Code" has been selected as the placement mode. Note that the placement control of key frames when "Time Code" or "Key Frame" is set as the placement mode is well known, and so a description thereof will be omitted.

[0049] In S702, it is determined whether the key frame to be set by the operator is to be added to an editing area of ​​interest (hereinafter referred to as a "current editing area") among the editing areas already set in the list. Here, the editing area means an area in the list in which a group of key frames having the same playback direction is one setting unit, and is specified by the operator (S710 described later). FIG. 10 is a diagram showing how the operator sets an editing area via a GUI. As shown in FIG. 10, the operator right-clicks with one or more pieces of key frame information selected to display a context menu 1001, and sets the editing area by selecting "Edit Area". This sets an editing area consisting of any key frame information selected by the user. In the example of FIG. 10, an editing area including four pieces of key frame information is set. FIG. 11 is a diagram showing how the operator selects and sets a current editing area from two already set editing areas. As shown in FIG. 11, the operator right-clicks on an area to be selected as the current editing area to display a context menu 1001, and this time selects "Current Edit Area". This sets the current editing area. In the example of FIG. 11, when the current editing area is set, the time code in the current editing area is displayed in bold so that it can be distinguished from other editing areas. However, as long as the operator can easily distinguish it, for example, the background color may be changed. Although not shown, the setting state of the current editing area can be canceled by performing a cancel operation on the area when the current editing area is set. In this step, if a selection operation of the current editing area shown in FIG. 11 is detected, the process proceeds to S711. On the other hand, if a selection operation of the current editing area is not detected, the process proceeds to S703. Note that immediately after the start of processing (when key frame information corresponding to the set key frame does not exist in the list), there is no set editing area in the first place, so the process always proceeds to S703.

[0050] In S703, the key frame setting unit 401 sets the playback direction for the key frame to be set based on a user instruction via the GUI. As shown in Fig. 9(c), the operator clicks on the "Play dir." portion and selects "Forward" meaning forward playback or "Reverse" meaning reverse playback from the options displayed in a pull-down menu. The playback direction is set according to the selection. In the example of Fig. 9(a), "Forward" meaning forward playback is currently set.

[0051] In S704, the camera parameter generating unit 400 generates the above-mentioned camera parameters based on the camera operation signal sent from the viewpoint input device 103. In the following S705, the key frame setting unit 401 sets a key frame based on the frame designation signal sent from the viewpoint input device 103. At this time, the camera parameters generated in S704 are associated with the key frame. Information on the key frame set in this manner is sent to the listing unit 402.

[0052] In the next step S706, the listing unit 402 determines whether the process of arranging the key frame information corresponding to the key frames set in S705 will be performed in ascending order of the time codes or in descending order of the time codes, based on the playback direction set in S703. If the set playback direction is "forward playback", the process proceeds to S707, and if it is "reverse playback", the process proceeds to S708.

[0053] In S707, the listing unit 402 arranges the key frame information of the key frames set in S705 in the list in ascending order of time codes. On the other hand, in S708, the listing unit 402 arranges the key frame information of the key frames set in S705 in the list in descending order of time codes.

[0054] In S709, it is determined whether to continue setting a key frame with the same playback direction or to set the above-mentioned editing area. If a new camera operation signal is subsequently input from viewpoint input device 103, the process returns to S704 to continue generating camera parameters and setting key frames. On the other hand, if a user operation to set the above-mentioned editing area is detected, the process proceeds to S710. Then, in S710, listing unit 402 sets the range specified by the user operation as the editing area. After the editing area is set, the process proceeds to S717.

[0055] In S717, it is determined whether or not to end the key frame setting for the video clip to be created. For example, if an operation by the operator to instruct saving of the key frame list via the GUI (for example, pressing a "Save" button, not shown) is detected, it is determined that the key frame setting work is ended, and the process proceeds to S718. On the other hand, if there is no instruction to save the key frame list, the process returns to S702 and continues.

[0056] In S711, if the selection of the current editing area is detected in the determination process of S702 to which the process has returned, similarly to S704, the camera parameter generation unit 400 generates the above-mentioned camera parameters based on the camera operation signal sent from the viewpoint input device 103. In the following S712, similarly to S705, the key frame setting unit 401 sets a key frame by associating the camera parameters generated in S711 with the frame designation signal sent from the viewpoint input device 103.

[0057] In S713, the listing unit 402 determines whether the arrangement process of the key frame information corresponding to the key frames set in S712 will be performed in ascending order of time codes or descending order of time codes based on the playback direction set for the current editing area. If the playback direction of the current editing area is "forward playback", the process proceeds to S714, and if it is "reverse playback", the process proceeds to S715.

[0058] S714 and S715 correspond to S707 and S708 described above, respectively. That is, in S714, the key frame information of the key frame set in S712 is arranged in the current editing area in ascending time code order, and in S715, the key frame information of the key frame set in S712 is arranged in the current editing area in descending time code order. Since arranged key frame information exists in the current editing area, the key frame information of the newly added key frame is inserted at an appropriate position according to the time code and playback direction. At this time, it is desirable to highlight the number and time code of the corresponding key frame, for example, by displaying them in reverse video (displaying them in white characters) so that the operator can understand where the key frame information of the added key frame has been arranged.

[0059] In S716, it is determined whether or not to continue adding key frames to the current editing area. If a new camera operation signal is input from the viewpoint input device 103, the process returns to S711 to continue generating camera parameters and setting additional key frames. On the other hand, if a user operation to cancel the setting of the current editing area is detected, the process proceeds to the above-mentioned S717, where it is determined whether or not to continue setting key frames.

[0060] In S718, the list storage unit 403 stores the key frame list created in the processing up to this point in the RAM 213. Then, in S719, the camera path creation unit 404 reads and acquires the key frame list stored by the list storage unit 403 from the RAM 213, and creates camera path information based on the key frame list.

[0061] The above is the flow of processing for creating camera path information taking into consideration the playback direction of a video clip to be created according to this embodiment.

[0062] <Modification> The above-mentioned flow chart of FIG. 7 does not assume the case where a key frame having the same time code as that of the key frame information already existing in the current editing area is added. Therefore, in order to deal with the case where a key frame having the same time code is added, the distance between virtual viewpoints and the similarity of the posture may be used as a reference for arrangement. FIG. 12 shows the result of arrangement based on the distance between virtual viewpoints when the area 1100 shown in FIG. 11 is set as the current editing area and two key frames 1201 and 1202 having the same time code as the key frame No. 3 are added. The virtual viewpoint position of the key frame No. 2 in the current editing area 1100 before the addition is (x, y, z)=(3.0, 1.0, 1.0). In the current editing area 1200 after the addition, a key frame 1201 associated with a virtual viewpoint position (x, y, z)=(4.0, 1.0, 1.0) closer to the virtual viewpoint position of the key frame No. 2 is newly arranged at the position of No. 3. The key frame 1202 associated with the virtual viewpoint position (x, y, z)=(6.0, 1.0, 1.0) is arranged at the position No. 5 since it is the furthest from the virtual viewpoint position in the key frame No. 2. In this way, by using the distance between the virtual viewpoints as a reference, even when a key frame with the same time code is added, the key frame information can be inserted at an appropriate position in the current editing area. If the distance between the virtual viewpoints is also the same, the key frame information of the key frame whose virtual viewpoint posture in both key frames is closer to the posture of the most recent virtual viewpoint may be arranged at a higher position. Furthermore, in response to a case where a key frame with the same time code is added, the key frames may be arranged in the setting order instead of the distance between the virtual viewpoints. FIG. 13 shows the result of arranging the key frames based on the setting order when the area 1110 shown in FIG. 11 is set as the current editing area and a key frame with the same time code "20:05:47:23" as the key frame No. 9 is added.In the current editing area 1110 before the addition, the virtual viewpoint position of the key frame placed at No. 8 is (x, y, z) = (4.0, 1.0, 1.0). In the current editing area 1300 after the addition, it can be seen that key frame 1301 with a virtual viewpoint position (x, y, z) = (3.0, 1.0, 1.0), which is closer to the virtual viewpoint position of the key frame at No. 8, is placed at No. 10 instead of No. 9.

[0063] Furthermore, the key frame information of a key frame added to the current editing area may be always placed at the top or bottom of the current editing area, regardless of the playback direction or time code. In this case, the operator needs to rearrange the key frame information every time a key frame is added, but by always placing the key frame information at the beginning or end of the current editing area, confusion during rearrangement can be reduced.

[0064] Also, the above-mentioned flowchart in FIG. 7 is premised on the assumption that "Play x Time Code" is selected as the arrangement mode from the beginning of the key frame setting work. However, the above-mentioned control contents can be applied in the same way even if the key frame setting work is initially performed in another arrangement mode and then switched to "Play x Time Code" halfway through. FIG. 14 shows an example of a key frame list obtained by performing key frame setting work under "Time Code". It can be seen that a total of 10 key frame information is arranged in ascending time code order. If an operator who has completed the key frame setting work up to this point changes the arrangement mode to "Play x Time Code" and rearranges the key frames in the desired order, it will be as follows. Now, suppose that the operator wants to play key frames No. 1, No. 5, No. 7, and No. 9 forward, then play key frames No. 3, No. 6, and No. 8 backward, and finally play key frames No. 2, No. 4, No. 7, and No. 10 forward. In this case, the operator first selects key frame information No.1, No.5, No.7, and No.9 to set the editing area, then selects it as the current editing area and sets the playback direction to forward playback. The operator then selects key frame information No.3, No.6, and No.8 to set the editing area, then selects it as the current editing area and sets the playback direction to reverse playback. Then, the operator selects key frame information No.2, No.4, No.7, and No.10 to set the editing area, then selects it as the current editing area and sets the playback direction to forward playback. By performing these operations, the desired key frame list can be obtained, as shown in Figure 15.

[0065] According to this embodiment, the key frame information of the key frames sequentially set by the operator is appropriately arranged according to the playback direction of the target video clip, thereby reducing the workload of the operator.

[0066] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0067] 100 Image Processing System 101 Imaging device 102 Image processing device 103 Viewpoint input device 104 Display device

Claims

1. a first generation means for generating a plurality of key frames in which parameters representing a position of a virtual viewpoint and a direction from the virtual viewpoint, the parameters being determined based on a user operation for specifying a virtual viewpoint related to a virtual viewpoint video generated based on a plurality of captured images obtained by imaging performed by a plurality of imaging devices, are associated with a time during which imaging is performed by the plurality of imaging devices; a second generating means for generating a first list in which a plurality of first keyframe information corresponding to a plurality of first keyframes, each having first information related to a time axis set, among the plurality of keyframes, is arranged in ascending order of time, and a second list in which a plurality of second keyframe information corresponding to a plurality of second keyframes, each having second information related to a time axis set, among the plurality of keyframes, is arranged in descending order of time; a third generating means for generating virtual viewpoint information representing a transition of the virtual viewpoint based on the first list and the second list; 13. An information processing device comprising:

2. The information processing device according to claim 1, characterized in that, when there are multiple pieces of first key frame information associated with the same time, the second generation means generates the first list in which the first key frame information corresponding to a key frame that was generated earlier is placed higher.

3. The information processing device described in claim 1 or 2, characterized in that the third generation means generates the virtual viewpoint information by interpolating the parameters of frames between a plurality of the first key frames corresponding to a plurality of the first key frame information included in the first list, and by interpolating the parameters of frames between a plurality of the second key frames corresponding to a plurality of the second key frame information included in the second list.

4. 4. The information processing apparatus according to claim 1, further comprising an image generating unit for generating a virtual viewpoint image based on the virtual viewpoint information.

5. A program for causing a computer to function as the information processing device according to any one of claims 1 to 4.

Images