Information processing apparatus, information processing method, and program

The information processing device efficiently generates multiple camera paths for virtual viewpoint videos by using keyframes linked to specific times and adding secondary parameters, addressing the inefficiency of traditional methods and enhancing scene diversity.

JP2024159067AInactive Publication Date: 2024-11-08CANON KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023074817
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-28
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The keyframe method for generating virtual viewpoint videos is time-consuming due to the need for repeated setting and adjustment of keyframes, making it inefficient for creating multiple camera paths for the same scene.

Method used

An information processing device that generates virtual viewpoint videos based on synchronous imaging by multiple cameras, using a first parameter set as a keyframe linked to a specific time and adding a second parameter to create multiple variations of camera paths, including sub-virtual cameras arranged in predefined modes.

Benefits of technology

Enables easy creation of multiple camera path variations for virtual viewpoint images, reducing the time required to generate diverse scenes and enhancing user interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159067000001_ABST
    Figure 2024159067000001_ABST
Patent Text Reader

Abstract

To easily generate a plurality of variations for camera paths for use in generating a virtual viewpoint image for the same scene.SOLUTION: A method according to the present invention has steps of, based on a user operation for designating a virtual viewpoint, setting a first parameter representing a position and a posture of the virtual viewpoint as a keyframe in association with a frame corresponding to a certain time during the synchronized image capturing and, based on the first parameter, adding, to the set keyframe, a second parameter which is a parameter representing the position and posture of the virtual viewpoint and is not based on the user operation.SELECTED DRAWING: Figure 7
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a technique for editing a camera path of a virtual viewpoint video. [Background technology]

[0002] A technology that uses multi-viewpoint images obtained by synchronously capturing images of the same subject (object) using multiple imaging devices installed at different positions to generate virtual viewpoint images, which are images that represent the view from a non-existent imaging device (virtual camera) virtually placed in a three-dimensional space, is attracting attention. Such a technology that generates virtual viewpoint images from multi-viewpoint images can provide viewers with a high sense of realism that is not possible with normal images. For example, in a rugby match, it is possible to obtain a highlight scene image that combines a cut that captures the players carrying the ball toward the goal from a bird's-eye view and a cut that captures the try scene from a specific player's subjective view (for example, the fullback's view of the ball passing by the side of the opposing player). This makes it possible to provide viewers with easy-to-understand images and commentary on the highlight scenes that have not been available before.

[0003] When generating a virtual viewpoint video, it is necessary to set virtual viewpoint information (generally called "camera path") that represents the transition of a viewpoint (virtual viewpoint) virtually placed in a three-dimensional space. For example, there is a key frame method as a method for creating this virtual viewpoint information. The key frame method is a method of setting a plurality of reference frames (called "key frames") to which any virtual viewpoint is associated, and obtaining path information indicating the path of the virtual camera by interpolating between the set plurality of key frames. In this key frame method, the set key frame and the virtual viewpoint are associated with each other using a time code so that the correspondence between the two can be specified. In this regard, Patent Document 1 discloses a technology for seamlessly progressing a CG animation that is rendered in real time in synchronization with a video material that employs a time code. Here, in a replay video to be used for commentary on a highlight scene of a sport, images from various angles are required, and it is required to quickly create a plurality of patterns of virtual viewpoint video for one scene. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-60280 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the keyframe method, the camerawork for a single image is completed by setting keyframes multiple times and repeatedly making adjustments, so generating multiple camera paths for the same scene requires a large amount of time in proportion to the number of paths.

[0006] The technology disclosed herein aims to easily create multiple variations of camera paths for generating virtual viewpoint videos of the same scene. [Means for solving the problem]

[0007] An information processing device according to the present disclosure is an information processing device for generating a virtual viewpoint video based on a plurality of moving images obtained by performing synchronized imaging by a plurality of imaging devices, and includes: a key frame setting means for linking a first parameter representing a position and attitude of the virtual viewpoint to a frame corresponding to a certain time during a period in which the synchronized imaging was performed, and setting it as a key frame, based on a user operation specifying a virtual viewpoint; an adding means for adding a second parameter representing the position and attitude of the virtual viewpoint, which is not based on the user operation, to the key frame set by the key frame setting means, based on the first parameter; and a generating means for generating a plurality of virtual viewpoint information representing changes in the virtual viewpoint, wherein the virtual viewpoint information is composed of a plurality of key frames, and at least one of the plurality of virtual viewpoint information has the second parameter linked to at least one of the plurality of key frames. Effect of the Invention

[0008] According to the present disclosure, it is possible to easily create multiple variations in the camera path for generating virtual viewpoint video of the same scene. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system. [Diagram 2] FIG. 2 is a diagram showing the hardware configuration of an image processing apparatus. [Diagram 3] FIG. 2 is a block diagram showing the functional configuration of the image processing apparatus. [Figure 4] FIG. 4 is a block diagram showing an example of a functional configuration in a main control unit. [Diagram 5] 4A to 4C are diagrams for explaining arrangement modes. [Figure 6] FIG. 13 is a diagram showing an example of a GUI and a controller for creating and editing virtual viewpoint information. [Figure 7] 10 is a flowchart showing a process for generating a virtual viewpoint video. [Figure 8] An example of a GUI that displays keyframes in a list format in ascending order. [Figure 9] An example of a GUI for displaying a list of camera paths. [Figure 10] An example of a GUI for post-setting the conditions of a sub virtual camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, a mode for carrying out the present embodiment will be described with reference to the drawings, etc. Note that the following embodiment does not limit the technology of the present disclosure, and all configurations described in the following embodiment are not necessarily essential to the means for solving the problem.

[0011] Before describing each embodiment, a brief overview of a virtual viewpoint image will be given. A virtual viewpoint image is an image that represents a view from a virtual camera viewpoint (virtual viewpoint) that is different from an actual camera viewpoint, and is also called a free viewpoint image. The virtual viewpoint is set by a method such as a user directly specifying the virtual viewpoint by operating a controller, or selecting from a plurality of virtual viewpoint candidates that are set in advance. The virtual viewpoint image includes both moving images and still images.

[0012] In the following embodiment, a description will be given on the premise of a video, and a virtual viewpoint image by the video will be expressed as a "virtual viewpoint video". Information indicating the transition of the viewpoint (virtual viewpoint) of a virtual camera that does not actually exist corresponding to the virtual viewpoint video will be called a "camera path". The camera path defines the movement of the virtual camera in a video created by sequentially playing a plurality of virtual viewpoint images or CG images. The camera path is managed by frames and a timeline. A frame holds information necessary for generating each image that constitutes a video. Specifically, the camera holds information on the time (time code) of a scene and the position and orientation of the camera. The time of a scene is expressed, for example, by a time code in which the time when the game to be filmed starts is 00:00:00:00 frame. The position of the camera is expressed, for example, by setting the origin at a predetermined position (for example, the center of the field) in the filming space, and is expressed by three coordinates, X, Y, and Z. The orientation of the camera is expressed, for example, by three angles, Pan, Tilt, and Roll. The timeline shows the time of the frames on a single time axis. The number of frames included in the timeline is determined by the number of images played per second (frame rate). For example, if the frame rate is 60 frames per second, the timeline will contain 60 frames per second. The reference frame is called a "key frame", and the time of the start key frame and the end key frame are indicated on the timeline. The video content creator creates a camera path by repeatedly determining the position and attitude of the virtual camera at an arbitrary time code and registering at least two or more key frames. When editing a camera path using key frames, frames are divided into two types: key frames and intermediate frames. For key frames, the user editing the camera path explicitly specifies the information. On the other hand, for intermediate frames, which are frames between key frames, the system determines the information by interpolating between the key frames.

[0013] [Embodiment 1] <System configuration> FIG. 1 is a diagram showing an example of the configuration of an image processing system for generating a virtual viewpoint video according to this embodiment. The image processing system 100 includes a plurality of imaging devices (cameras) 101, an image processing device 102, a viewpoint input device 103, and a display device 104. In the image processing system 100, the image processing device 102 generates a virtual viewpoint video based on a plurality of moving images (multiple viewpoint videos) obtained by synchronous imaging of the plurality of imaging devices 101 and a virtual viewpoint set by the viewpoint input device 103, and displays the generated virtual viewpoint video on the display device 104. The area subject to synchronous imaging by the plurality of imaging devices 101 is, for example, a field of a stadium where a sport such as rugby or soccer is played, or a stage of a concert hall where a play or the like is performed. For example, when generating a virtual viewpoint video for a rugby match held in a stadium, the imaging devices 101 are installed at different positions so as to surround the field in the stadium, and each imaging device 101 takes an image on the field in a time-synchronized manner. However, the plurality of imaging devices 101 do not necessarily have to be installed all around the periphery of the imaging target area, and when there is a restriction on the installation location, the imaging devices 101 may be installed only in a direction of a part of the imaging target area. Each imaging device 101 is realized by, for example, a digital video imaging device equipped with a video signal interface such as a serial digital interface (SDI). Each imaging device 101 adds time information such as a time code to the video signal it outputs and transmits it to the image processing device 102.

[0014] The image processing device 102 generates a virtual viewpoint video based on images captured by the multiple imaging devices 101 synchronously. The image processing device 102 generates a virtual viewpoint video by performing processes such as generating 3D shape data of an object and rendering using the images captured by each imaging device 101 acquired via a network cable or the like. The image processing device 102 also sets a virtual viewpoint (creates a camera path) based on a user operation signal input from a viewpoint input device 103. The functions of the image processing device 102 will be described in detail later.

[0015] The viewpoint input device 103 is an input device having a controller such as a joystick. An operator performs various setting operations related to the virtual viewpoint via the viewpoint input device 103.

[0016] The display device 104 is composed of, for example, a liquid crystal display or an LED, and acquires and displays image data (UI screen data for a graphical user interface and virtual viewpoint image data) sent from the image processing device 102.

[0017] In this specification, a series of explanations will be given taking the case of generating a virtual viewpoint video as an example, but the following contents can also be applied to generating and editing a camera path for a CG video. In that case, the image processing system includes a CG renderer instead of a module that generates a virtual viewpoint video. Then, a viewpoint that can be moved to any position within the CG space can be treated as the virtual viewpoint in this embodiment.

[0018] <Hardware configuration> 2 is a diagram showing a hardware configuration of the image processing device 102. The image processing device 102, which is an information processing device, includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, an operation unit 215, a communication I / F 216, and a bus 217.

[0019] The CPU 211 realizes each function of the image processing device 102 by controlling the entire image processing device 102 using computer programs and data stored in the ROM 212 or the RAM 213. The image processing device 102 may have one or more dedicated hardware different from the CPU 211 or a GPU (Graphics Processing Unit). At least a part of the processing by the CPU 211 may be performed by the GPU or the dedicated hardware. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0020] The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, and data supplied from the outside via the communication I / F 217. The auxiliary storage device 214 is constituted by, for example, a hard disk drive, and stores various data such as image data and volume data.

[0021] The operation unit 215 is composed of, for example, a keyboard, a mouse, etc., and inputs various instructions to the CPU 211 in response to operations by a user. The CPU 211 operates as a display control unit that controls the display device 104, and an operation control unit that controls the operation unit 215. The communication I / F 216 is used for communication with an external device of the image processing device 102. For example, when the image processing device 102 is connected to an external device by wire, a communication cable is connected to the communication I / F 216. When the image processing device 102 has a function of wirelessly communicating with an external device, the communication I / F 216 is equipped with an antenna.

[0022] The bus 217 transmits information by connecting each unit of the image processing device 102. Note that, in this embodiment, the viewpoint input device 103 and the display device 104 are provided as external devices, but either of them may be provided internally as one of the functional units of the image processing device 102.

[0023] <Functional configuration> 3 is a block diagram showing the functional configuration of the image processing device 102. The image processing device 102 includes an image acquisition unit 300, a foreground / background separation unit 301, a storage unit 302, a virtual viewpoint video generation unit 303, and a main control unit 304.

[0024] The image acquisition section 300 acquires data of images (multiple viewpoint images) obtained by the multiple imaging devices 101 capturing images synchronously.

[0025] The foreground / background separation unit 301 extracts specific objects (hereinafter referred to as "foreground objects") that are in the foreground, such as players and referees, from the video of each of the imaging devices 101 on a frame-by-frame basis. The foreground / background separation unit 301 stores an image showing the extracted foreground object as a foreground image in the storage unit 302. The method of extracting the foreground object is not limited. For example, a background difference method is a method of extracting a foreground object by comparing pixel values ​​of a frame (still image) in a video and a background image showing the background part of the frame to extract a foreground object in the frame. The background image is an image in a state where no foreground object exists. For example, when shooting a rugby match, an image obtained by shooting before the start of the game when there are no players on the field is used. The foreground / background separation unit 301 stores the background image and the foreground image together with shooting position information indicating the three-dimensional position of the imaging device 101. The three-dimensional position is defined by the position in each direction corresponding to the height, width, and depth in the three-dimensional space of the subject to be shot. The shooting position information according to the world coordinate system can be obtained, for example, by shooting a marker or the like that has been set in advance and performing a three-dimensional image measurement of the marker, etc. The background image and the foreground image are stored in the RAM 213 or the auxiliary storage device 214 by the storage unit 302.

[0026] The main control unit 304 creates the above-mentioned camera path based on a user operation via the viewpoint input device 103, and provides it to the virtual viewpoint video generation unit 303. The created camera path is output to the virtual viewpoint video generation unit 303.

[0027] The virtual viewpoint video generator 303 generates three-dimensional shape data of the foreground object based on the foreground image, and performs coloring (texture pasting) on ​​the three-dimensional shape data according to the camera path to generate an image that represents the view from the virtual viewpoint. The method of generating the three-dimensional shape data is, for example, the shape-from-silhouette method, and is not particularly limited. When generating the three-dimensional shape data, the position and size of the foreground object in the shooting target space are also specified. The generated virtual viewpoint video is output to the display device 104.

[0028] Note that there may be a plurality of image processing devices 102. For example, the generation of the virtual viewpoint video may be performed in another image processing device having the functions of the image acquisition unit 300, the foreground / background separation unit 301, the storage unit 302, and the virtual viewpoint video generation unit 303 described above. In this case, the image processing device 102 provides the other image processing device with a camera path for generating the virtual viewpoint video. It is also possible to apply the image processing device 102 to editing the camera path of the CG video. In this case, a CG renderer may be provided instead of the virtual viewpoint video generation unit 303. In this case, the camera path created by the method described below represents the trajectory of the viewpoint in the CG image.

[0029] <Details about the main control unit> 4 is a block diagram showing an example of a functional configuration in the main control unit 304 related to creating and editing a camera path. The main control unit 304 is made up of a camera parameter generation unit 400, a key frame setting unit 401, a sub virtual camera setting unit 402, a mode setting unit 403, a key frame list storage unit 404, a condition setting unit 405, a camera path generation unit 406, and a GUI control unit 407. Each unit will be described below.

[0030] The main control unit 304 creates a camera path based on user operations via the viewpoint input device 103 , and provides it to the virtual viewpoint video generation unit 303 .

[0031] The camera parameter generating unit 400 acquires from the viewpoint input device 103 an operation signal (hereinafter referred to as a "camera operation signal") that specifies the position and attitude of the virtual camera input by the user operating a joystick or the like. Then, based on the acquired camera operation signal, camera parameters of a non-existent virtual camera corresponding to the virtual viewpoint are generated. The camera parameters in this embodiment include parameters such as the position and attitude of the virtual viewpoint, zoom, and time. In this case, the position of the virtual viewpoint is indicated by three-dimensional coordinate information according to an orthogonal coordinate system of three axes, for example, the X axis, the Y axis, and the Z axis. The origin at that time is an arbitrary position in the three-dimensional space of the subject to be photographed (the center of the field of the stadium in this embodiment). The attitude of the virtual camera is indicated by an angle formed with the three axes of pan, tilt, and roll. The zoom is indicated by, for example, a focal length. The time is indicated by a time code that sets the time when the game to be photographed started as 00:00:00:00 frame. The zoom and time are each one-axis parameters. Therefore, the camera parameters in this embodiment have eight axis parameters, namely, X coordinate, Y coordinate, Z coordinate, pan, tilt, roll, zoom, and time code. Note that the camera parameters may include parameters defining elements other than these eight axes, and may not include all of these eight axis parameters. The camera parameters set based on the user's operation are sent to a key frame setting unit 401.

[0032] The key frame setting unit 401 sets, as a key frame, a frame corresponding to a certain time during a period in which synchronous imaging is performed, based on the camera parameters input from the camera parameter generation unit 400 and a frame designation signal input from the viewpoint input device 103. Here, the frame designation signal is a signal that instructs setting, as a key frame, a specific frame selected by the operator from among a group of frames constituting the multiple viewpoint video. At least two key frames must be set. Information on the set key frame (key frame setting information) is output to the sub virtual camera setting unit 402 and the key frame list storage unit 404.

[0033] Based on the arrangement mode set by mode setting unit 403 (described later), sub virtual camera setting unit 402 sets a virtual camera that is a variation of the virtual camera specified by the user for the key frame set by key frame setting unit 401. In the following description, the virtual camera specified by the user is referred to as the "main virtual camera," and the virtual camera that is a variation set for the main virtual camera is referred to as the "sub virtual camera."

[0034] The mode setting unit 403 sets an operation mode that specifies what kind of sub virtual camera is to be placed in the imaging space by the sub virtual camera setting unit 402 with respect to the key frame set by the key frame setting unit 401. In this embodiment, three types of settable placement modes are prepared: "circle mode", "angle mode", and "straight line mode". FIGS. 5(a) to 5(c) are diagrams for explaining the above three types of placement modes. Now, a virtual camera (main virtual camera) 501 capturing an object 500 from the right direction is placed in a three-dimensional space by an operator. FIG. 5(a) is an example of placement in the "circle mode". In this example, three sub virtual cameras 502a to 502c are placed at equal intervals on a circumference with the object 500 of interest at the center and the distance from the object 500 to the main virtual camera 501 as the radius. FIG. 5(b) is an example of placement in the "angle mode". In this example, four sub virtual cameras 512a to 512d are arranged at equal intervals on a sector-shaped arc having a central angle α with the object of interest 500 at the center and the distance from the object 500 to the main virtual camera 501 as the radius. Also, FIG. 5(c) is an example of an arrangement in the "straight line mode", in which two sub virtual cameras 522a and 522b are arranged at equal intervals on a straight line connecting the object of interest 500 and the main virtual camera 501. Note that the above three arrangement modes are merely examples, and various arrangement modes can be provided, such as a "concentric circle mode" in which the sub virtual cameras are arranged concentrically on multiple circumferences with different radii, and an "altitude mode" in which the sub virtual cameras are arranged at different heights.

[0035] The key frame list saving unit 404 makes a list of two or more key frames set by the key frame setting unit 401, associates it with a video ID (Video Clip No.), and saves it in the auxiliary storage device 214. At that time, for a key frame in which a sub virtual camera has been automatically added by the sub virtual camera setting unit 402, information about the sub virtual camera is also associated with information about the main virtual camera and saved as a key frame list. The saved key frame list is read out by the camera path generating unit 406 and used to generate a camera path.

[0036] The condition setting unit 405 sets the conditions when the sub virtual camera setting unit 402 sets the sub virtual camera and the conditions when the camera path generation unit 406 generates the camera path. Examples of the conditions include the following.

[0037] <Conditions for adding a sub virtual camera> Specify the number of additional sub virtual cameras to set. - Set a minimum distance between virtual cameras (virtual viewpoints) (e.g., less than 1m is not acceptable) To prevent the virtual camera from being too close or too far away from the object, a lower limit (e.g., not less than 30 cm) and an upper limit (e.g., not more than 100 m) are set for the distance between the object and the sub virtual camera. -Prohibits the addition of sub virtual cameras to specific keyframes (such as the start keyframe or a specific keyframe where a main virtual camera that captured a decisive moment is set).

[0038] <Camera path generation conditions> Specify the number of camera path variations (subpaths) to generate. To prevent the camera path from changing too suddenly, set an upper limit (for example, 120 degrees) on the amount of change in pan, tilt, and roll angles between keyframes. The end frame is fixed to an overhead image with the object at the center. - Generate a camera path for reverse playback based on the camera path for forward playback.

[0039] The camera path generating unit 406 generates a camera path based on the camera parameters output from the camera parameter generating unit 400 and the key frame list stored by the key frame list storing unit 404. Furthermore, when a sub virtual camera is set for any of the key frames in the stored key frame list, the camera path generating unit 406 generates various variations of camera paths combining the sub virtual cameras. In the following description, the camera path specified by the user is referred to as the "main path", and the camera path that is a variation of the main path is referred to as the "sub path".

[0040] The GUI control unit 407 controls a GUI (Graphical User Interface) displayed on the display device 104 for the user to operate the image processing device 102 and the viewpoint input device 103. FIG. 6 is a diagram showing an example of a GUI and a controller for creating and editing a camera path (virtual viewpoint information). Two UI screens 601 and 602 are displayed on the display device 104, and a controller 603 constitutes an operation unit of the viewpoint input device 103. The UI screen 601 displays a multi-viewpoint image that is the basis of the virtual viewpoint video, and a virtual viewpoint video generated based on the camera path information. The UI screen 602 is used for setting an arrangement mode and a key frame, checking the created camera path, and the like. For example, an operator can edit the camera path displayed on the UI screen 602 while checking an image representing the view from a virtual viewpoint in each frame being edited on the UI screen 601. Note that information on the frame being edited by the camera path generation unit 406 is constantly sent to the virtual viewpoint video generation unit 303, and the virtual viewpoint video generation unit 303 generates a virtual viewpoint video based on the received frame information. The generated virtual viewpoint image is displayed on the display device 104, and the user can edit the camera path while checking an image that represents the view from the virtual camera (virtual viewpoint) that the user has set for the frame to be edited.

[0041] <Operation flow of image processing device> Fig. 7 is a flowchart showing the flow of processing for generating a virtual viewpoint video in the image processing device 102 according to this embodiment. Below, with reference to the flowchart in Fig. 7, a processing flow will be explained, from automatically generating a sub-path based on a main path set by the user to creating a virtual viewpoint video corresponding to a camera path selected by the user. In the following explanation, the symbol "S" means a step.

[0042] In S701, the mode setting unit 403 sets a placement mode that specifies in what placement a sub-virtual camera that is a variation of the main virtual camera set by the user is to be added. In the case of this embodiment, one of the above-mentioned "circle mode", "angle mode", and "straight line mode" is set. In this case, for example, the setting may be based on a user operation via a UI screen (not shown) for mode selection, and if there is no user operation related to mode selection, a default placement mode may be set. In addition, if there are multiple objects in the target scene, the user may be allowed to specify an object of interest (reference position) that serves as a reference when adding and setting a sub-virtual camera in each placement mode via a GUI (not shown).

[0043] In S702, the condition setting unit 405 sets the conditions for additionally setting a sub virtual camera and the conditions for generating a camera path. At this time, for example, the conditions may be set based on a user operation via a UI screen (not shown) for setting conditions, and if there is no user operation, default conditions may be set. In addition, the setting of the arrangement mode performed in S701 may be regarded as one of the conditions for additionally setting a sub virtual camera, and may be performed together in this step.

[0044] In S703, the camera parameter generation unit 400 acquires a camera operation signal from the viewpoint input device 103, and generates camera parameters that specify the position, orientation, time (time code), etc. of the virtual camera (main virtual camera) corresponding to the acquired camera operation signal. The generated camera parameters are output to the key frame setting unit 401.

[0045] In S704, the key frame setting unit 401 sets a key frame by associating the camera parameters of the virtual camera input from the camera parameter generation unit 400 with the frame designation signal input from the viewpoint input device 103. Information on the set key frame is stored in the RAM 213 and is also output to the sub virtual camera setting unit 402.

[0046] In S705, the sub virtual camera setting unit 402 additionally sets a sub virtual camera for the key frame set in S704 based on the arrangement mode set in S701. At this time, the sub virtual camera is set randomly according to the "conditions for additional setting of sub virtual cameras" set in S702. For example, consider a case where the "circle mode" is set and the number of sub virtual cameras to be added is specified as "2". In this case, two sub virtual cameras are automatically additionally set at arbitrary positions on a circumference having a radius equal to the distance from the set main virtual camera 501 to the object. Note that, when the sub virtual cameras are arranged at equal intervals as in the example of FIG. 5(a) described above, it is sufficient to specify the arrangement at equal intervals in the "conditions for additional setting of sub virtual cameras".

[0047] In S706, the GUI control unit 407 displays one or more key frames set by the user in the processes up to this point. FIG. 8 is an example of a GUI that displays the key frames set by the user in a list format in ascending order. The GUI shown in FIG. 8 is composed of, from the left column, buttons (up / down) for rearranging key frames, a sub virtual camera number (Sub), a key frame number and time code (Key No. / hh:mm:ss:ff), and camera parameters (Camera Parameter) indicating the position and attitude of the virtual camera. The key frame rearrangement buttons are used when the user manually rearranges key frames. Pressing the upward button (▲up) replaces the key frame with the key frame above. Similarly, pressing the downward button (▼down) replaces the key frame with the key frame below. The key frame numbers (Key No.) indicate the order of the key frames in the list, and are arranged in the order of playback of the key frames in the target video clip. In this example, the camera parameters indicating the position and attitude of the virtual camera are the three coordinates XYZ indicating the camera position (distance from the origin expressed in meters) and the pan and tilt angles indicating the attitude of the camera. "Sub" indicating the sub virtual camera number is filled with a number equal to or less than the number of sub virtual cameras specified in the conditions for additional settings of the sub virtual camera set in S702, and if "-" instead of a number is filled, it means that the main virtual camera has been set by the user himself.

[0048] In S707, it is determined whether or not to end the key frame setting for the video clip to be created. If the GUI control unit 407 detects a user operation (e.g., pressing a "Save" button, not shown) to instruct saving of the key frame list via the GUI as described above, it determines that the key frame setting work is completed and proceeds to S707. On the other hand, if there is no instruction to save the key frame list, the process returns to S703 and continues.

[0049] In S708, the key frame list storage unit 404 stores all the key frames set in the processes up to this point in list format. The stored key frame list includes the camera parameters of the sub virtual camera that was automatically added and set, in addition to the camera parameters of the main virtual camera that the user set himself / herself.

[0050] In S709, the camera path generation unit 406 generates a main path and a sub path based on the key frame list stored in S708, in accordance with the camera path generation conditions set in S702.

[0051] In S710, the GUI control unit 407 displays a list of all the camera paths generated in S709, i.e., the main path and the sub-paths, on the GUI. FIG. 9 shows an example of a GUI displaying a list of camera paths. In FIG. 9, "Video Clip No." indicates the number of the video clip (virtual viewpoint image) realized by the corresponding camera path, and "No. 1-1" in the figure indicates the virtual viewpoint image realized by the main path, and "No. 1-1 to No. 1-5" indicate the virtual viewpoint images realized by the sub-paths. "hh:mm:ss:ff" under "Video Clip No." indicates the video codes of the start frame and the end frame. "Time" indicates the playback time when the target video clip is played. "Route" indicates the order of the key frames included in the corresponding camera path. In the example shown in the figure, the camera paths of "No.1-1 to No.1-4" are camera paths for forward playback that start with the key frame "K1", continue with the key frame "K2", the key frame "K3", and end with the key frame "K4". The camera path of "No.1-5" is a camera path for reverse playback that starts with the key frame "K4", continues with the key frame "K3", the key frame "K2", and end with the key frame "K1". In this case, for example, "K1" without a branch number indicates the first key frame set by the user. And "K1_s1", "K1_s2", and "K1_s3" with branch numbers indicate that a sub virtual camera additionally set for the first key frame "K1" is used. Although a description is omitted here, the same applies to "K2", "K3", and "K4". Although not included in the example of Fig. 9, a subpath may be further generated by adding a subvirtual camera to any of the key frames constituting the camera path for reverse playback generated as a subpath. "Playback Direction" indicates the playback direction, with "normal (forward)" indicating forward playback and "reverse" indicating reverse playback.The "Preview" button on the right side is a selection button used when the user previews (checks) the virtual viewpoint video for each camera path, and the check box on the left side is a check box for specifying the camera path the user wants to save. When the GUI control unit 407 detects that the user has selected the "Preview" button, it instructs the virtual viewpoint video generation unit 303 to generate a virtual viewpoint video based on the camera path related to the selection.

[0052] In S711, the virtual viewpoint video generation unit 303 generates a virtual viewpoint video corresponding to the camera path selected by the user, based on an instruction from the main control unit 304. The data of the generated virtual viewpoint video is output to the display device 104 by the main control unit 304 and displayed. If the user finds a camera path that he or she likes, he or she saves it by checking a check box in the GUI of FIG. 9 and pressing an OK button (not shown), for example.

[0053] The above is the flow of the virtual viewpoint video generation process in the image processing device 102.

[0054] <Modification> In the above embodiment, the conditions for the sub virtual camera and the camera path are set before the user starts setting the key frames, but they may be set afterwards. An example of a case where the conditions are set afterwards is when the user wants to obtain further sub paths that are variations based on a certain sub path after setting a series of key frames and previewing the virtual viewpoint video using the above-mentioned "Preview" button. Now, suppose that the user likes the sub virtual cameras set for the first and fourth key frames after previewing a certain automatically generated sub path, but wants to see other variations for the sub virtual cameras set for the second and third key frames. FIG. 10 shows an example of a GUI in which a user who thinks like this sets conditions for the sub virtual camera afterwards. Now, the user has specified a fixed "Sub Cam1" for the first key frame, "Auto" for the second and third key frames, and a fixed "Sub Cam3" for the fourth key frame as conditions for the sub virtual camera. Such conditions may be specified afterwards to perform the processing from S705 onwards. In the example of Fig. 10, for the second and third key frames where "Auto" is specified, the system will automatically set the sub virtual camera randomly according to the set placement mode, but for the first and fourth key frames, it will be fixed to "Sub Cam1" and "Sub Cam3" respectively. Similarly, the conditions regarding the camera path can also be set afterwards after checking the main path and the automatically generated sub path, and the camera path can be generated again.

[0055] As described above, according to this embodiment including the modified examples, the user can easily obtain variations in the camera path using key frames that he or she has set himself or herself.

[0056] <Other embodiments> The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0057] The present disclosure also includes the following configurations and methods.

[0058] [Configuration 1] An information processing device for generating a virtual viewpoint video based on a plurality of moving images obtained by performing synchronous imaging by a plurality of imaging devices, a key frame setting means for setting, as a key frame, a first parameter representing a position and an orientation of a virtual viewpoint, based on a user operation for designating the virtual viewpoint, in association with a frame corresponding to a certain time during a period in which the synchronized imaging is performed; an adding means for adding, to the key frame set by the key frame setting means, a second parameter that is not based on the user operation and that represents a position and an orientation of the virtual viewpoint, based on the first parameter; A generating means for generating a plurality of pieces of virtual viewpoint information representing a transition of the virtual viewpoint; having The virtual viewpoint information is composed of a plurality of key frames, At least one piece of virtual viewpoint information among the plurality of pieces of virtual viewpoint information is associated with at least one key frame among the plurality of key frames, and 23. An information processing apparatus comprising:

[0059] [Configuration 2] 2. The information processing apparatus according to configuration 1, further comprising condition setting means for setting an adding condition of the second parameter in the adding means.

[0060] [Configuration 3] The information processing device according to configuration 2, characterized in that the additional condition is an operation mode of the adding means that specifies how the virtual viewpoint represented by the second parameter is placed in an imaging space based on the first parameter.

[0061] [Configuration 4] The information processing device according to configuration 3, characterized in that the operation mode is an operation mode in which the virtual viewpoint represented by the second parameter is positioned on a circumference of a circle whose radius is the distance from an object of interest in the video to the virtual viewpoint represented by the first parameter.

[0062] [Configuration 5] The information processing device according to configuration 3, characterized in that the operating mode is an operating mode in which the virtual viewpoint represented by the second parameter is positioned on a sectorial arc having a central angle with a radius equal to the distance from the object to the virtual viewpoint represented by the first parameter, the sector being centered on an object of interest in the video.

[0063] [Configuration 6] The information processing device according to configuration 3, characterized in that the operation mode is an operation mode in which the virtual viewpoint represented by the second parameter is positioned on a straight line connecting an object of interest in the video and the virtual viewpoint represented by the first parameter.

[0064] [Configuration 7] 3. The information processing device according to configuration 2, wherein the additional condition is the number of the second parameters to be added.

[0065] [Configuration 8] 3. The information processing device according to configuration 2, wherein the additional condition is a lower limit for the distance between virtual viewpoints.

[0066] [Configuration 9] 3. The information processing device according to configuration 2, wherein the additional condition is a lower limit or an upper limit for a distance between an object of interest in the video and the virtual viewpoint represented by the second parameter.

[0067] [Configuration 10] 3. The information processing device according to configuration 2, wherein the addition condition is to prohibit addition to a specific key frame among the key frames set by the key frame setting means.

[0068] [Configuration 11] 3. The information processing apparatus according to configuration 2, wherein the condition setting means further sets conditions for generating the plurality of pieces of virtual viewpoint information in the generating means.

[0069] [Configuration 12] The information processing device according to configuration 11, characterized in that the generation condition is the number of pieces of virtual viewpoint information generated by the generation means, the virtual viewpoint information including at least one key frame to which the second parameter is linked.

[0070] [Configuration 13] The information processing device described in configuration 11, characterized in that the generation condition is an upper limit value for the amount of change in the virtual viewpoint between key frames of the virtual viewpoint information generated by the generation means, the virtual viewpoint information including at least one key frame to which the second parameter is linked.

[0071] [Configuration 14] 14. The information processing device according to configuration 13, wherein the amount of change in the virtual viewpoint is any one of an amount of change in a pan angle, a tilt angle, and a roll angle.

[0072] [Configuration 15] The information processing device described in configuration 11, characterized in that the generation condition fixes the second parameter in an end frame of the virtual viewpoint information generated by the generation means, the virtual viewpoint information including at least one key frame to which the second parameter is linked, to a specific content.

[0073] [Configuration 16] 16. The information processing device according to configuration 15, wherein the specific content is a parameter corresponding to an overhead image centered on an object of interest in the video.

[0074] [Configuration 17] The information processing device described in configuration 11, characterized in that the generation condition is to generate, as one of the plurality of virtual viewpoint information generated by the generation means, the virtual viewpoint information for reverse playback in which the order of the plurality of key frames set by the key frame setting means is reversed.

[0075] [Configuration 18] 18. The information processing device according to any one of configurations 1 to 17, further comprising a display control means for causing a display means to display a list showing a list of the plurality of pieces of virtual viewpoint information generated by the generation means.

[0076] [Configuration 19] 19. The information processing device according to configuration 18, further comprising an image generating means for generating a virtual viewpoint image corresponding to virtual viewpoint information selected by a user from the plurality of virtual viewpoint information included in the list.

[0077] [Configuration 20] 20. The information processing device according to configuration 19, wherein the display control means causes the display means to display the virtual viewpoint image generated by the image generation means.

[0078] [Method 1] 1. An information processing method for generating a virtual viewpoint video based on a plurality of moving images obtained by performing synchronous imaging by a plurality of imaging devices, comprising: a key frame setting step of linking a first parameter representing a position and an orientation of a virtual viewpoint to a frame corresponding to a certain time during a period in which the synchronized imaging is performed, as a key frame, based on a user operation of specifying the virtual viewpoint; an adding step of adding a second parameter, which is a parameter representing a position and an orientation of the virtual viewpoint and is not based on the user operation, to the key frame set by the key frame setting means based on the first parameter; a generating step of generating a plurality of pieces of virtual viewpoint information representing a transition of the virtual viewpoint; Including, The virtual viewpoint information is composed of a plurality of key frames, At least one piece of virtual viewpoint information among the plurality of pieces of virtual viewpoint information is associated with at least one key frame among the plurality of key frames, and 23. An information processing method comprising:

[0079] [Configuration 21] A program for causing a computer to function as the information processing device according to any one of configurations 1 to 20.

Claims

1. An information processing device for generating a virtual viewpoint video based on a plurality of moving images obtained by performing synchronous imaging by a plurality of imaging devices, a key frame setting means for setting, as a key frame, a first parameter representing a position and an orientation of a virtual viewpoint, based on a user operation for designating the virtual viewpoint, in association with a frame corresponding to a certain time during a period in which the synchronized imaging is performed; an adding means for adding, to the key frame set by the key frame setting means, a second parameter that represents a position and an orientation of the virtual viewpoint and is not based on the user operation, based on the first parameter; A generating means for generating a plurality of pieces of virtual viewpoint information representing a transition of the virtual viewpoint; having The virtual viewpoint information is composed of a plurality of key frames, At least one piece of virtual viewpoint information among the plurality of pieces of virtual viewpoint information is associated with at least one key frame among the plurality of key frames, and the second parameter is associated with at least one key frame among the plurality of key frames.

23. An information processing apparatus comprising:

2. 2. The information processing apparatus according to claim 1, further comprising a condition setting unit for setting an adding condition for the second parameter in the adding unit.

3. 3. The information processing apparatus according to claim 2, wherein the additional condition is an operation mode of the adding means that specifies how the virtual viewpoint represented by the second parameter is to be positioned in an imaging space based on the first parameter.

4. 4. The information processing device according to claim 3, characterized in that the operation mode is an operation mode in which the virtual viewpoint represented by the second parameter is positioned on a circumference of a circle whose radius is the distance from a target object in the moving image to the virtual viewpoint represented by the first parameter.

5. The information processing device according to claim 3, characterized in that the operating mode is an operating mode in which the virtual viewpoint represented by the second parameter is positioned on a sectorial arc having a central angle centered on an object of interest in the video and a radius equal to the distance from the object to the virtual viewpoint represented by the first parameter.

6. 4. The information processing device according to claim 3, characterized in that the operation mode is an operation mode in which the virtual viewpoint represented by the second parameter is positioned on a straight line connecting an object of interest in the moving image and the virtual viewpoint represented by the first parameter.

7. 3. The information processing apparatus according to claim 2, wherein the additional condition is the number of the second parameters to be added.

8. The information processing apparatus according to claim 2 , wherein the additional condition is a lower limit value for a distance between virtual viewpoints.

9. 3 . The information processing apparatus according to claim 2 , wherein the additional condition is a lower limit or an upper limit for a distance between an object of interest in the video and the virtual viewpoint represented by the second parameter.

10. 3. The information processing apparatus according to claim 2, wherein the addition condition is to prohibit addition to a specific key frame among the key frames set by the key frame setting means.

11. 3. The information processing apparatus according to claim 2, wherein said condition setting means further sets a generating condition for said plurality of pieces of virtual viewpoint information in said generating means.

12. 12. The information processing apparatus according to claim 11, wherein the generation condition is the number of pieces of virtual viewpoint information generated by the generation means, the pieces of virtual viewpoint information including at least one key frame to which the second parameter is linked.

13. The information processing device according to claim 11, characterized in that the generation condition is an upper limit value for an amount of change in the virtual viewpoint between key frames of the virtual viewpoint information generated by the generation means, the virtual viewpoint information including at least one key frame to which the second parameter is linked.

14. 14. The information processing apparatus according to claim 13, wherein the amount of change in the virtual viewpoint is any one of an amount of change in a pan angle, a tilt angle, and a roll angle.

15. The information processing device according to claim 11, characterized in that the generation condition fixes the second parameter in an end frame of the virtual viewpoint information generated by the generation means, the virtual viewpoint information including at least one key frame to which the second parameter is linked, to a specific content.

16. 16. The information processing apparatus according to claim 15, wherein the specific content is a parameter corresponding to an overhead image centered on an object of interest in the video.

17. 12. The information processing device according to claim 11, wherein the generation condition is to generate, as one of the plurality of pieces of virtual viewpoint information generated by the generation means, the virtual viewpoint information for reverse playback in which an order of the plurality of key frames set by the key frame setting means is reversed.

18. 2. The information processing apparatus according to claim 1, further comprising a display control means for causing a display means to display a list showing a list of the plurality of pieces of virtual viewpoint information generated by the generating means.

19. 19. The information processing apparatus according to claim 18, further comprising an image generating means for generating a virtual viewpoint image corresponding to virtual viewpoint information selected by a user from the plurality of virtual viewpoint information included in the list.

20. 20. The information processing apparatus according to claim 19, wherein the display control means causes the display means to display the virtual viewpoint video generated by the video generation means.

21. 1. An information processing method for generating a virtual viewpoint video based on a plurality of moving images obtained by performing synchronous imaging by a plurality of imaging devices, comprising: a key frame setting step of linking a first parameter representing a position and an orientation of a virtual viewpoint to a frame corresponding to a certain time during a period in which the synchronized imaging is performed, as a key frame, based on a user operation of designating the virtual viewpoint; an adding step of adding, to the key frame set by the key frame setting means, a second parameter that represents a position and an orientation of the virtual viewpoint and is not based on the user operation, based on the first parameter; a generating step of generating a plurality of pieces of virtual viewpoint information representing a transition of the virtual viewpoint; Including, The virtual viewpoint information is composed of a plurality of key frames, At least one piece of virtual viewpoint information among the plurality of pieces of virtual viewpoint information is associated with at least one key frame among the plurality of key frames, and the second parameter is associated with at least one key frame among the plurality of key frames.

23. An information processing method comprising:

22. A program for causing a computer to execute the information processing method according to claim 21.

Citation Information

Patent Citations

  • Information processor, information processing method, and program

    JP2012060280A