Information processing device, information processing method, and program.

Abstract

When we wanted to create a new camera path by modifying an existing camera path that corresponds to a specific time period, we were unable to easily create the desired camera path. [Solution] The virtual camera control device 110 acquires a first camera path, displays a virtual viewpoint image generated based on the first camera path, and, if it acquires input to operate the virtual camera during display, generates a second camera path based on the input from the time the input starts. It then generates a third camera path that includes a part of the first camera path corresponding to the time before the input started, and the second camera path.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus that controls a virtual camera corresponding to a virtual viewpoint image.

Background Art

[0002] There is a technique for generating a virtual viewpoint image corresponding to a virtual camera specified by a user using a plurality of captured images captured by a photographing system including a plurality of imaging devices. According to this technique, for example, in sports such as soccer and basketball, it is possible to provide a virtual viewpoint image captured from a position where a real imaging device cannot be arranged.

[0003] In generating a virtual viewpoint video corresponding to a predetermined time, an operator of a virtual camera sets a camera path indicating the trajectory of the virtual camera at the predetermined time. In Patent Document 1, a technique is described in which a camera path is corrected based on a second operation while playing back a virtual viewpoint video corresponding to a camera path indicating the trajectory of the virtual camera recorded based on the first operation.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when creating a new camera path by correcting a camera path corresponding to some of the existing camera paths corresponding to a predetermined time, it has not been possible to easily create a desired camera path.

[0006] An object of the present disclosure is to facilitate the generation of a desired camera path.

Means for Solving the Problems

[0007] An information processing apparatus according to one embodiment of the present disclosure has the following configuration: an acquisition means for acquiring first information which indicates the trajectory of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images, and information which indicates the position and orientation of the virtual camera corresponding to each of a plurality of time points; a display control means for performing first control to display a virtual viewpoint image generated based on the first information; and a generation means for generating third information which, when an input to operate the virtual camera is acquired during the first control, includes the first information corresponding to a time before the start of the input, and second information which indicates the position and orientation of the virtual camera corresponding to the input. [Effects of the Invention]

[0008] According to this disclosure, it is possible to easily generate a desired camera path. [Brief explanation of the drawing]

[0009] [Figure 1] This is a system configuration diagram of the image processing system 100 according to Example 1. [Figure 2] This is a hardware configuration diagram of the virtual camera control device 110 according to Embodiment 1. [Figure 3] This is a flowchart relating to the process by which the virtual camera control device 110 according to Embodiment 1 controls playback and recording. [Figure 4] This is a flowchart illustrating the processing of the image processing system 100 when the playback mode according to Example 1 is OFF. [Figure 5] This is a flowchart showing the processing of the image processing system 100 when the playback mode according to Example 1 is ON. [Figure 6] This figure shows an example of the UI of the virtual camera control device 110 according to Embodiment 1. [Figure 7] This figure shows an example of the UI of the virtual camera control device 110 according to Embodiment 1. [Figure 8]This figure shows an example of the UI of the virtual camera control device 110 according to Embodiment 1. [Figure 9] This figure shows an example of the UI of the virtual camera control device 110 according to Embodiment 1. [Figure 10] This figure shows an example of the controller 142 according to Example 1. [Figure 11] This figure shows an example of the UI of the virtual camera control device 110 according to Embodiment 1. [Figure 12] This is a diagram illustrating the modification of the camera path according to Example 1. [Figure 13] This figure shows an example of internal data for the camera path according to Example 1. [Figure 14] This figure shows an example of internal data for the camera path according to Example 1. [Figure 15] This figure shows an example of internal data for the camera path according to Example 1. [Figure 16] This figure shows an example of internal data for the camera path according to Example 1. [Figure 17] This figure shows an example of the UI of the virtual camera control device according to Embodiment 2. [Modes for carrying out the invention]

[0010] <Embodiment> According to a preferred embodiment of the present disclosure, an information processing apparatus has an acquisition unit that acquires first information indicating a trajectory of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images, and information indicating the position and orientation of the virtual camera corresponding to each of a plurality of times. Further, the information processing apparatus has a display control unit that performs first control to display a virtual viewpoint image generated based on the first information. Further, when the information processing apparatus acquires an input for operating the virtual camera during the first control, the information processing apparatus has a generation unit that generates second information indicating the position and orientation of the virtual camera corresponding to the input. The generation unit generates third information including a part of the first information corresponding to a time before the input and the second information. Further, the information processing apparatus has a recording unit that records the third information. The input is, for example, an input corresponding to an operation in which a user uses a joystick provided on a controller. The camera parameters of the virtual camera are parameters indicating the position and orientation of the virtual camera. The first information is a first camera path indicating a trajectory of the virtual camera generated in advance. The second information is a second camera path indicating a trajectory of the virtual camera generated based on a user input. The third information is a third camera path generated by combining a part of the first camera path and the second camera path.

[0011] With this aspect, an operator who operates the virtual camera can easily generate a third camera path obtained by modifying a part of the first camera path. For example, after setting the time corresponding to the virtual viewpoint image being displayed to a time desired to be modified in the first camera path, the third camera path can be generated by operating the virtual camera. Therefore, even without displaying virtual viewpoint images for all times corresponding to the first camera path, a third camera path obtained by easily modifying a part of the first camera path can be generated.

[0012] Further, the second information is generated based on the input and the first information. That is, the second information is generated by modifying the camera parameters of the virtual camera included in the first information based on the input for operating the virtual camera by the operator.

[0013] According to this aspect, the operator can easily create the third information by partially modifying the first information only by controlling the amount by which the operator wants to modify the camera parameters of the virtual camera in the first information. That is, since the operator does not need to operate the virtual camera so as to reproduce the camera parameters of the virtual camera included in the first information, it is useful when the operator wants to finely adjust the camera parameters of the virtual camera.

[0014] Further, the first information corresponds to a predetermined time. Further, the display control means displays a time bar indicating the predetermined time and the time corresponding to the virtual viewpoint image being displayed.

[0015] According to this aspect, the operator can adjust the time of the virtual viewpoint image being displayed according to the time to be modified using the time bar. Then, by performing an input for operating the virtual camera after the adjustment, the third information can be generated without displaying the virtual viewpoint image before the time to be modified.

[0016] Further, the display control means displays a marker indicating a specific time specified by a user operation on the time bar.

[0017] According to this aspect, for example, the operator can indicate a period that the operator considers to require modification in the first information, and can more efficiently generate the third information by partially modifying the first information.

[0018] According to another preferred embodiment of the present disclosure, the information processing method includes an acquisition step of acquiring first information, which includes the trajectory of a virtual camera corresponding to a virtual viewpoint image and camera parameters of the virtual camera corresponding to a plurality of time points. The information processing method also includes a display control step of controlling the display of a virtual viewpoint image generated based on the first information. The information processing method also includes a generation step of generating second information based on the input after the start of the input if an input for operating the virtual camera is acquired during the display. The generation step generates third information, which includes a portion of the first information corresponding to a time point before the start of the input and the second information. The information processing method also includes a recording step of recording the third information.

[0019] According to other preferred embodiments of the present disclosure, the program causes the computer to function as the information processing device described above.

[0020] <Examples> The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0021] <Example 1> In this example, when a user makes an input to operate the virtual camera while a virtual viewpoint image corresponding to a pre-generated camera path (corresponding to the first information and the first camera path) is displayed, a part of the camera path is modified to generate a new camera path (corresponding to the third information and the third camera path). Specifically, a new camera path is generated that includes some of the camera parameters included in the pre-generated camera path corresponding to the time when no user input has been received, and camera parameters based on the user input corresponding to the time when input has been received. This allows the user to easily generate a new camera path by operating the virtual camera while the virtual viewpoint image corresponding to the time they want to modify in the pre-generated camera path is displayed. For example, by displaying the virtual viewpoint image for the time they want to modify in the pre-generated camera path and making an input to operate the virtual camera, a new camera path including camera parameters corresponding to the time when no input has been received can be generated. This eliminates the need to display all the virtual viewpoint images for the time corresponding to the pre-generated camera path, making it possible to generate a new camera path more efficiently.

[0022] Figure 1 is a system configuration diagram of the image processing system 100 according to Embodiment 1 of this disclosure.

[0023] The image processing system 100 is a system that generates a virtual viewpoint image representing the view from a specified virtual viewpoint, based on multiple images captured by multiple imaging devices and a specified virtual viewpoint. The image processing system 100 consists of a camera group 101, a 3D model generation device 102, a 3D model storage device 103, a virtual viewpoint image generation device 104, a display 105, a virtual camera control device 110, a display 140, a mouse 141, and a controller 142. Note that the above configuration is just an example, and for example, display 105 and display 140 may be the same display. Also, display 140, mouse 141, and controller 142 may be included in the same device. Note that the mouse 141 is not required. In that case, the controller 142 is connected to the UI control unit 111. The virtual viewpoint image in this embodiment is also called a free viewpoint image, but it is not limited to images corresponding to a viewpoint freely (arbitrarily) specified by the user, and for example, images corresponding to a viewpoint selected by the user from multiple candidates are also included in the virtual viewpoint image. Furthermore, while this embodiment primarily describes the case where the virtual viewpoint is specified by user operation, the virtual viewpoint may also be specified automatically based on the results of image analysis, etc. Also, while this embodiment primarily describes the case where the virtual viewpoint image is a video, the virtual viewpoint image may also be a still image.

[0024] The viewpoint information used to generate a virtual viewpoint image is information indicating the position and orientation (viewing direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set that includes parameters representing the three-dimensional position of the virtual viewpoint and parameters representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. However, the content of the viewpoint information is not limited to the above. For example, the parameter set as viewpoint information may include a parameter representing the size of the field of view (angle of view) of the virtual viewpoint. Furthermore, the viewpoint information may have multiple parameter sets. For example, the viewpoint information may have multiple parameter sets corresponding to multiple frames that make up a video of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of multiple consecutive points in time.

[0025] The image processing system 100 has multiple imaging devices that capture images of the imaging area from multiple directions. The imaging area is, for example, a stadium where sports such as soccer or karate are played, or a stage where concerts or plays are performed. The multiple imaging devices are installed at different positions surrounding the imaging area and perform imaging synchronously. Note that the multiple imaging devices do not have to be installed around the entire circumference of the imaging area; depending on the constraints of the installation location, they may be installed only around a part of the perimeter of the imaging area. Also, the number of imaging devices is not limited to the example shown in the figure; for example, if the imaging area is a soccer stadium, about 30 imaging devices may be installed around the stadium. Furthermore, imaging devices with different functions, such as telephoto cameras and wide-angle cameras, may also be installed.

[0026] In this embodiment, each of the multiple imaging devices is assumed to be a camera with an independent housing capable of capturing images from a single viewpoint. However, this is not limited to this configuration, and two or more imaging devices may be configured within the same housing. For example, a single camera equipped with multiple lens groups and multiple sensors, capable of capturing images from multiple viewpoints, may be installed as multiple imaging devices.

[0027] A virtual viewpoint image is generated, for example, by the following method. First, multiple images (multiple viewpoint images) are obtained by capturing images from different directions using multiple imaging devices. Next, a foreground image is obtained by extracting the foreground region corresponding to a predetermined object such as a person or a ball, and a background image is obtained by extracting the background region other than the foreground region. In addition, a foreground model representing the three-dimensional shape of the predetermined object and texture data for coloring the foreground model are generated based on the foreground image, and texture data for coloring the background model representing the three-dimensional shape of the background, such as a stadium, is generated based on the background image. Finally, the texture data is mapped to the foreground model and background model, and rendering is performed according to the virtual viewpoint indicated by the viewpoint information to generate a virtual viewpoint image. However, the method of generating a virtual viewpoint image is not limited to this, and various methods can be used, such as a method of generating a virtual viewpoint image by projective transformation of captured images without using a three-dimensional model.

[0028] A virtual camera is a virtual camera distinct from the multiple imaging devices actually installed around the imaging area, and is a concept used to conveniently explain the virtual viewpoint involved in the generation of virtual viewpoint images. In other words, a virtual viewpoint image can be considered an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. The position and orientation of the virtual viewpoint in the said image can be represented as the position and orientation of the virtual camera. In other words, a virtual viewpoint image can be said to be an image that simulates the image obtained by a camera, assuming that the camera exists at the position of a virtual viewpoint set in space. In this embodiment, the content of the change in the virtual viewpoint over time is referred to as the camera path. That is, in this embodiment, the camera path of the virtual camera indicates the trajectory of the virtual camera. However, it is not essential to use the concept of a virtual camera to realize the configuration of this embodiment. In other words, it is sufficient that at least information representing a specific position in space and information representing orientation are set, and a virtual viewpoint image is generated according to the set information.

[0029] The camera group 101 synchronizes the timing of all cameras to capture images for the purpose of generating a 3D model of the subject. The cameras included in the camera group 101 are real-world cameras, unlike virtual cameras. The camera group 101 outputs multiple captured images to the 3D model generation device 102.

[0030] The 3D model generation device 102 extracts the subject as the foreground from the captured images acquired from the camera group 101 and generates a silhouette image of the foreground. Then, it generates a 3D model of the subject based on the generated silhouette image. The method for generating the 3D model may be the viewing volume cross-eyed method or photogrammetry. However, the 3D model generation means in this disclosure is not limited to these. The 3D model generation device 102 outputs the generated 3D model of the subject to the 3D model storage device 103.

[0031] The 3D model storage device 103 stores the 3D models acquired from the 3D model generation device 102.

[0032] The virtual viewpoint image generation device 104 generates a virtual viewpoint image based on the 3D model acquired from the 3D model storage device 103 and the camera parameters of the virtual camera acquired from the camera parameter transmission unit 117. The generated virtual viewpoint image is then output to the display 105. Here, the camera parameters of the virtual camera include information indicating the position, orientation, and field of view of the virtual camera. The output destination of the virtual viewpoint image does not necessarily have to be the display 105; for example, it may be transmitted to a distribution company or broadcasting station.

[0033] The display 105 displays a virtual viewpoint image acquired from the virtual viewpoint image generation device 104. The operator (user) controlling the virtual camera control device 110 can operate the virtual camera by operating the controller 142, etc., while viewing the virtual viewpoint image output to the display 105. The operator can operate the virtual camera via the controller 142.

[0034] The virtual camera control device 110 is a device that allows an operator to control a virtual camera for generating virtual viewpoint images. The virtual camera control device 110 consists of a UI control unit 111, a state holding unit 112, a camera parameter determination unit 115, a camera parameter modification unit 116, a camera parameter transmission unit 117, a playback unit 118, a recording unit 119, and a camera path holding unit 120. The virtual camera control device 110 is also referred to as an information processing device.

[0035] The UI control unit 111 creates a UI representing the internal state of the virtual camera control device 110 and a list of clips, and displays it on the display 140. In this embodiment, a clip is synonymous with a camera path managed by the camera path holding unit 120. Figure 6 shows an example of the UI generated by the UI control unit. The UI can also be operated with the mouse 141, and the operator can press the recording start button 603 on the virtual camera control device UI 601 with the mouse.

[0036] The state holding unit 112 holds the recording state 113 and the playback state 114. All states held by the state holding unit 112 are accessible and rewritable by all blocks constituting the virtual camera control device 110.

[0037] The recording status 113 indicates whether or not recording is currently in progress. While the specific means of maintaining the status are not limited, in this embodiment, it is maintained as a variable that can take two values: True or False. When True, it indicates that recording is in progress, and when False, it indicates that recording is not in progress. In other words, when recording status 113 is set to True, the recording mode is ON, and when recording status 113 is set to False, the recording mode is OFF.

[0038] The playback state 114 indicates whether the camera path held by the camera path holding unit 120 is currently being played back. The specific means of holding the state are not limited, but in this embodiment, it is held as a variable that can take two values, True and False, where True indicates that playback is in progress and False indicates that playback is not in progress. In other words, when the playback state is set to True, the playback mode is ON, and when the playback state is set to False, the playback mode is OFF. Playback in this disclosure means applying the camera parameters of the virtual camera corresponding to each of the multiple time points included in the camera path to the camera parameters of the virtual camera corresponding to the virtual viewpoint image to be generated, in order of frame number.

[0039] When the playback state 114 is False, the camera parameter determination unit 115 determines the camera parameters of the virtual camera, including information on the virtual camera's position, orientation, and field of view, based on the operation information of the controller 142 operated by the operator. For example, when the position of the virtual camera is expressed in three-dimensional coordinates [X, Y, Z], values ​​such as X=4.0, Y=9.0, Z=1.5 are calculated. In this embodiment, the unit is [m], and the origin position is assumed to be the center of the 3D model generation range. When the X-axis is specified to be parallel to the ground surface, the Y-axis is specified to be parallel to the ground surface and perpendicular to the X-axis, and the Z-axis is specified to be perpendicular to the ground surface. Furthermore, when the orientation of the virtual camera is expressed in three angles [Pan, Tilt, Roll], values ​​such as Pan=20.0, Tilt=10.0, Roll=2.0 are calculated. In this embodiment, the unit of angle is [degrees], and the value range is from -180 to 180. Pan is the angle of rotation parallel to the ground surface, Tilt is the angle of rotation perpendicular to the ground surface, and Roll is the angle of rotation relative to the optical axis of the virtual camera. When the field of view of the virtual camera is expressed in terms of focal length Zoom, a value such as Zoom=6.0 is calculated. In this embodiment, the unit of the field of view is [mm]. The camera parameter determination unit 115 can also store the camera parameters of the virtual camera from the previous frame and determine the camera parameters of the virtual camera based on the amount of change of each parameter calculated based on the operation information of the controller 142. Furthermore, when the playback state 114 is True, the camera parameters of the virtual camera are determined based on the camera parameters of the virtual camera obtained from the playback unit 118. That is, the camera parameters of the virtual camera are received from the playback unit 118 based on the frame number of the camera path being played back from the camera path held by the camera path holding unit 120, and the camera parameters of the virtual camera are determined.

[0040] When the playback state 114 is False, the camera parameter modification unit 116 does not modify the camera parameters of the virtual camera obtained from the camera parameter determination unit 115. When the playback state 114 is True, it modifies the camera parameters of the virtual camera obtained from the camera parameter determination unit 115 based on input from the controller 142. The specific means of modification input are not limited. For example, if the controller 142 is a controller equipped with two joysticks as shown in Figure 10, the position of the virtual camera may be modified by operating the left stick and the pose of the virtual camera may be modified by operating the right stick.

[0041] Figure 12 is a diagram illustrating the modification of the camera path according to Embodiment 1. Line 1202 represents the trajectory (camera path) of the virtual camera, point 1201 represents the starting point of the trajectory, and point 1203 represents the ending point of the trajectory. The virtual camera 1204 moves along the trajectory 1202. For example, if the left joystick 1003 shown in Figure 10 is tilted to the right, the camera position of the virtual camera 1204 is modified to the right with respect to the optical axis. The modified position of the virtual camera is shown by virtual camera 1205. Also, when the left joystick is tilted to the right, the camera position may be modified in the positive direction of the X axis instead of moving towards the optical axis, as shown by virtual camera 1206. Furthermore, if the playback state 114 is True and operation information for modifying the camera parameters of the virtual camera is received from the controller 142, and the recording state 113 is False, the recording state is changed to True and recording begins. The camera parameter modification unit 116 also transmits the camera parameters of the virtual camera to the camera parameter transmission unit 117. Furthermore, if the recording status 113 is True, the camera parameters of the virtual camera are also sent to the recording unit 119.

[0042] The camera parameter transmission unit 117 transmits the camera parameters of the virtual camera, obtained from the camera parameter modification unit 116, to the virtual viewpoint image generation device 104.

[0043] If the playback state 114 is True, the playback unit 118 acquires the camera path held by the camera path holding unit 120 and transmits the camera parameters of the virtual camera to the camera parameter determination unit 115 based on the playback time.

[0044] The recording unit 119 stores the camera parameters of the virtual camera obtained from the camera parameter modification unit 116 in the camera path holding unit 120. Also, when the playback state is True and the recording state is False, the camera path from the start of playback to the current frame is stored when the controller 142 sends operation information for modification to the camera parameter modification unit 116. In other words, using Figure 12 as an example, the camera path from the starting point 1201 to the virtual camera 1204 of the current frame is stored in the camera path holding unit 120 as a new camera path. To put it another way, the newly generated camera path is recorded in the camera path holding unit 12.

[0045] The camera path holding unit 120 holds the camera path. Figure 13 shows an example of the data to be stored, and in this embodiment, a dataset linking the time 1302 and the camera parameters 1303 of the virtual camera is stored in chronological order. However, the data structure of the camera path in this disclosure is not limited to this, and it is acceptable as long as it shows at least the trajectory of the camera parameters of the virtual camera.

[0046] The display 140 displays a UI as shown in Figure 6, generated by the UI control unit 111.

[0047] Mouse 141 controls the mouse pointer used to operate the UI shown in Figure 6, and enables the selection of buttons such as the recording start button 603.

[0048] The controller 142 is a device used by the operator to control the virtual camera. For example, it may be equipped with a joystick or a seesaw switch as shown in Figure 10.

[0049] Figure 2 shows the hardware configuration of the virtual camera control device 110 according to Embodiment 1. The 3D model generation device 102, 3D model storage device 103, and virtual viewpoint image generation device 104 have the same hardware configuration. The virtual camera control device 110 consists of a CPU 201, ROM 202, RAM 203, auxiliary storage device 204, display unit 205, operation unit 206, communication I / F 207, and system bus 208.

[0050] The CPU 201 controls the entire virtual camera control device 110 using computer programs and data stored in the ROM 202 and RAM 203, thereby realizing each function of the system shown in Figure 1. The virtual camera control device 110 may have one or more dedicated hardware components separate from the CPU 201, and at least a portion of the processing performed by the CPU 201 may be executed by the dedicated hardware. Examples of such dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0051] ROM202 stores programs and other data that do not require modification. RAM203 temporarily stores programs and data supplied from auxiliary storage device 204, as well as data supplied from external sources via communication interface 207. Auxiliary storage device 204 is composed of, for example, a hard disk drive and stores various types of data such as image data and audio data.

[0052] The display unit 205 is composed of, for example, an LCD display or LEDs, and displays a GUI (Graphical User Interface) for the user to give instructions to the virtual camera control device 110. The operation unit 206 is composed of, for example, a keyboard, mouse, joystick, touch panel, etc., and receives various instructions from the user and inputs them to the CPU 201. The CPU 201 operates as a display control unit that controls the display unit 205, and as an operation control unit that controls the operation unit 206.

[0053] The communication interface 207 is used for communication with external devices of the virtual camera control device 110, such as the camera group 101, the display 105, the joystick 130, the foot pedal 131, and the knob controller 132. If the virtual camera control device 110 has a function to communicate wirelessly with external devices, the communication interface 207 is equipped with an antenna.

[0054] The system bus 208 connects the various parts of the virtual camera control device 110 and transmits information.

[0055] In this embodiment, the display unit 205 and the operation unit 206 are assumed to be located inside the virtual camera control device 110, but at least one of the display unit 205 and the operation unit 206 may be located outside the virtual camera control device 110 as a separate device.

[0056] Figure 3 is a flowchart showing the process by which the virtual camera control device 110 according to Embodiment 1 controls playback and recording. The virtual camera control device 110 repeatedly executes the process from step S301 to step S310 for each frame.

[0057] In step S302, the virtual camera control device 110 determines whether a clip has been selected by user operation. If a clip has been selected, it proceeds to step S303; otherwise, it proceeds to step S304. In the example in Figure 6, it determines whether any of the clips 606 to 609 included in the clip list 605 have been selected. The camera path holding unit 120 has already acquired clips 606 to 609, and the UI control unit 111 displays clips 606 to 609 in the clip list 605.

[0058] In step S303, the virtual camera control device 110 transitions to a state in which the clip selected in step S302 is played by setting the playback state 114 to True.

[0059] In step S304, the virtual camera control device 110 determines whether the playback stop button has been pressed. If it has been pressed, it proceeds to step S305; otherwise, it proceeds to step S306. In the example in Figure 6, it determines whether the clip playback stop button 602 has been pressed.

[0060] In step S305, the virtual camera control device 110 stops playback of the clip by setting the playback state 114 to False.

[0061] In step S306, the virtual camera control device 110 determines whether the recording start button has been pressed. If it has been pressed, it proceeds to step S307; otherwise, it proceeds to step S310. In the example in Figure 6, it determines whether the recording start button 603 has been pressed.

[0062] In step S307, the virtual camera control device 110 determines the state of the recording state 113 for the frame to be processed. If the recording state 113 is True, the device proceeds to step S308; otherwise, the device proceeds to step S309.

[0063] In step S308, the virtual camera control device 110 sets the recording state 113 to False and stops the recording process.

[0064] In step S309, the virtual camera control device 110 sets the recording state 113 to True and starts the recording process.

[0065] In step S310, the current frame is incremented, and the process proceeds to step S301.

[0066] With the above configuration, the virtual camera control device 110 manages the playback state and the recording state by updating them every frame.

[0067] Figure 4 is a flowchart showing the processing of the virtual camera control device 110 when the playback mode according to Embodiment 1 is OFF. In other words, it is the flowchart for when the playback state 114 is False, and the operator freely operates the virtual camera and records and creates a clip. The processing from step S401 to step S407 is repeated for each frame.

[0068] In step S402, the camera parameter determination unit 115 acquires user operation information from the controller 142.

[0069] In step S403, the camera parameter determination unit 115 determines the camera parameters of the virtual camera, including information on the virtual camera's position, orientation, and field of view, based on the operation information of the controller 142 acquired in step S402. Alternatively, the camera parameter determination unit 115 may store the camera parameters of the virtual camera from the previous frame and determine the camera parameters of the virtual camera based on the amount of change in each parameter calculated based on the operation information of the controller 142.

[0070] In step S404, the state of recording state 113 is determined. If recording state 113 is True, the process proceeds to step S405. If recording state 113 is False, the process proceeds to step S406.

[0071] In step S405, the recording unit 119 acquires the camera parameters of the virtual camera determined in step S403 and stores them in the camera path holding unit 120.

[0072] In step S406, the camera parameter transmission unit 117 transmits the camera parameters of the virtual camera determined in step S403 to the virtual viewpoint image generation device 104.

[0073] Figure 5 is a flowchart showing the processing of the image processing system 100 when the playback mode according to Embodiment 1 is ON. That is, the flowchart in Figure 5 is executed when the playback state 114 is True. The processing from step S501 to step S514 is repeated for each frame. The clip to be played back is assumed to have been selected in step S302 as described in Figure 3.

[0074] Step S502 retrieves the frame number of the clip currently being played. The clip being played and the frame number are explained using Figures 9 and 13 as examples. Figure 9 shows the UI when clip 901 is selected from the clip list 605 and the virtual viewpoint image corresponding to clip 901 is being played. Figure 13 shows an example of the camera path held by the camera path holding unit 120, corresponding to clip 901. Figure 13 represents 601 frames of data for 10 seconds from time 16:50:20:00 to 16:50:30:00, with the camera parameters of the virtual camera stored in each frame. The frame number retrieved in step S502 is the corresponding value from frame number 1301. The frame number is 0 immediately after selecting clip 901 and is incremented by 1 with each frame processed. The increment process is performed in step S513, which will be described later.

[0075] In step S503, the camera parameters of the virtual camera corresponding to the frame number obtained in step S502 are acquired. In the example in Figure 13, the camera parameters 1303 of the virtual camera corresponding to frame number 1301 are acquired. For example, if the frame number value obtained in step S502 is 120, the value of the camera parameters 1303 of the virtual camera is acquired from the row where frame number 1301 in Figure 13 is 120. Specifically, the camera parameters 1303 of the virtual camera, which are "position [3.1m, 4.3m, 1.1m], orientation [45.9°, -11.4°, 0.0°], field of view 6.7mm", are acquired.

[0076] In step S504, it is determined whether or not there is an input from the controller 142. If there is an input, the process proceeds to step S505; otherwise, the process proceeds to step S510.

[0077] In step S505, the camera parameters of the virtual camera acquired in step S503 are modified based on the operation input from the controller 142. The specific means of modification are not limited. For example, if the controller has two joysticks as shown in Figure 10, the camera position may be modified by operating the left stick and the camera's pose may be modified by operating the right stick.

[0078] In step S506, the recording status 113 is determined. If it is True, the process proceeds to step S509; otherwise, the process proceeds to step S507.

[0079] In step S507, the recording status 113 is set to True, putting the device in a recording state. In other words, the recording mode is turned ON.

[0080] In step S508, the camera path from the beginning of the camera path to the previous frame is registered as a new camera path in the camera path holding unit 120. That is, a portion of the camera path up to the frame corresponding to the time the input from the controller was acquired is registered as a new camera path in the camera path holding unit 120. For example, if the frame No. value is 120 in step S502, the data from frame No. 0 to 119 is duplicated to create a camera path as shown in Figure 14.

[0081] In step S509, the camera parameters of the current frame after modification are recorded in the camera path holding unit 120. As an example, the process when 120 is obtained as the frame number in step S502 and the camera path shown in Figure 14 is created in step S508 will be described. The camera parameters of the virtual camera for frame No. 120 are registered as shown in Figure 15, by adding them to the end of the camera path in Figure 14. The values ​​of the camera parameters of the virtual camera at this time are those of the virtual camera that were modified in step S505. In the example in Figure 15, an example is shown in which the camera position X coordinate of the virtual camera corresponding to frame No. 120 in Figure 13 is modified from 3.1m to 3.2m and stored.

[0082] In step S510, the camera parameter transmission unit 117 transmits the camera parameters of the virtual camera to the virtual viewpoint image generation device 104. The camera parameters of the virtual camera transmitted here are those acquired in step S503 if there was no controller input in step S504. If there was controller input in step S504, the camera parameters transmitted are those of the virtual camera after modification in step S505.

[0083] In step S511, it is determined whether the frame number is the last frame. That is, if the camera path in Figure 13 is being played back, it is determined whether the frame number is 600, which is the last frame. If it is the last frame, the process proceeds to step S512; otherwise, the process proceeds to step S513.

[0084] In step S512, False is stored in the recording state 113, stopping the recording. That is, playback of the camera path in Figure 13 is completed, and as a result, the creation of a new camera path as shown in Figure 16 is completed. The difference between the camera paths in Figure 13 and Figure 16 is that the X coordinate of the virtual camera has been modified for camera parameters 1303 and 1403 of the virtual camera from frame No. 120 onwards. Although this explanation describes a newly created camera path when only the X coordinate of the virtual camera is modified, this disclosure is not limited to this, and any other parameter may be modified.

[0085] In step S513, the frame number is incremented. That is, if the frame number obtained in step S502 was 120, the frame number is incremented by 1, making the frame number 121.

[0086] As described above, when input corresponding to user operation of the virtual camera is obtained, a third camera path can be generated that includes a portion of the first camera path and a portion of the first camera path modified by the user operation (corresponding to the second information and the second camera path). Since the user operation of the virtual camera is the trigger, there is no need to record the camera path while playing back the first camera path from the beginning, and the third camera path can be generated efficiently.

[0087] In the flow shown in Figure 5, the camera parameters of the modified virtual camera are recorded in the camera path holding unit 120 for each frame in step S509, but this is not limited to this. For example, the camera parameters of the modified virtual camera may be recorded in the auxiliary recording device 204, and when recording is stopped in step S512, the camera parameters of the modified virtual camera for multiple frames may be recorded together in the camera path holding unit 120.

[0088] In the flow shown in Figure 5, step S505 involves modifying the camera parameters of the virtual camera acquired in step S503, but this is not limited to this configuration. For example, it may also involve modifying the camera parameters of the virtual camera in the previous frame. In that case, while the user is operating the virtual camera using the controller, they can operate it freely without being limited by the camera parameters of the virtual camera included in the clip selected by the user, thus increasing the degree of freedom.

[0089] Figures 6 to 9 show examples of the virtual camera control device UI. Each shows an example of the display of the virtual camera control device UI 601, and the display changes as shown in Figures 6 to 9 depending on various states.

[0090] Figure 6 shows an example of the UI when playback state 114 is False. In other words, it is the UI when clip playback is not being performed and the virtual camera is being freely controlled according to controller input.

[0091] The clip playback stop button 602 is a button to be pressed during clip playback, and will be explained later in Figure 9. The recording start button 603 is used to start recording, and pressing it changes the recording status 113 from False to True. When the recording status is True, the display changes to a recording stop button as shown in Figure 7, 701. Also, the display "Recording" appears as shown in Figure 7, 702.

[0092] Timecode 604 displays the time in the format HH;MM;SS;FF, and display 105 shows a virtual viewpoint image associated with the time indicated by timecode 604.

[0093] Clip list 605 is an area that displays all the clips you have created, from 606 to 609. Note that you do not necessarily have to display all the clips you have created; you can display only specific clips.

[0094] Clips 606-609 are clips created by pressing the record start button 603, and playback begins when clicked with the mouse. For example, the timecode "16;50;14;00-16;50;31;00" displayed on clip 606 indicates that it is a clip of approximately 17 seconds, from 16;50;14;00 to 16;50;31;00. These displays have the same meaning for clips 607-609.

[0095] The playback bar 610 is displayed grayed out as shown when the playback status 114 is False, i.e., when a clip is not being played. The playback bar 610 shows the time corresponding to the selected clip and the time corresponding to the virtual viewpoint image being played. The playback bar 610 is also referred to as the time bar.

[0096] Figure 7 shows an example of the UI of the virtual camera control device 110 when recording mode is ON. Compared to Figure 6, the recording stop button 701, the recording indicator 702, and the clip 703 have been changed in their display.

[0097] The recording stop button 701 stops recording when pressed and changes the recording status 113 from True to False. At the same time as stopping recording, the recording indicator 702 also disappears.

[0098] The recording indicator 702 is a display intended to clearly indicate to the operator that recording is in progress. It is displayed when the recording status 113 is True, and disappears when it is False.

[0099] Clip 703 is created immediately after recording begins and displays the timecode at the start of recording and the message "Recording in progress".

[0100] Figure 8 shows an example of the UI when recording is stopped by pressing the stop recording button 701 from the state shown in Figure 7. Specifically, the stop recording button 701 returns to the display of the start recording button 603, and the recording indicator 702 disappears. Also, the display of clip 703 changes to that of clip 801. In other words, the display switches to show the start timecode and end timecode at the moment the recording status 113 changes from True to False.

[0101] Figure 9 shows an example of the UI when clip 607 is clicked to start playback, compared to the state shown in Figure 8. Specifically, the display of clip 607 changes to a selected state, as shown in clip 901, indicating that it is playing. In addition, the display of the playback bar 610 becomes active, as shown in playback bar 902, and the operator can keep track of how far the played clip has progressed as the playback bar advances over time.

[0102] Additionally, clip playback can be stopped by pressing the clip playback stop button 602 while a clip is playing. That is, pressing it when the playback state 114 is True changes the playback state to False. At this time, the selection state of clip 901 returns to the display of clip 607, and the display of playback bar 902 becomes inactive, like playback bar 610. Note that the clip playback stop button 602 does not have to do anything when pressed when the playback state is False, nor does it have to be disabled when the playback state is False.

[0103] Figure 10 shows an example of a controller 142 for controlling a virtual camera. In this embodiment, two joystick controllers are used: a joystick controller 1001 located on the left and a joystick controller 1002 located on the right. However, this disclosure is not limited to this, and a device with two joysticks integrated into one, such as a gamepad, may also be used. The gamepad may also be equipped with a gyro sensor. In this case, the camera parameters of the virtual camera may be changed based on the angular acceleration of the gyro sensor. Alternatively, the position and orientation of the virtual camera may be corrected in the camera parameter correction unit 116 based on the angular acceleration of the gyro sensor.

[0104] The joystick 1003 can control the position of the virtual camera. Tilting the joystick 1003 left or right moves the virtual camera left or right, and tilting it forward or backward moves the virtual camera forward or backward. Additionally, twisting the joystick 1003 clockwise lowers the virtual camera, and twisting it counterclockwise raises it.

[0105] The joystick 1004 can control the orientation of the virtual camera. Tilting the joystick 1004 left or right controls the pan of the virtual camera, and tilting it forward or backward controls the tilt of the virtual camera. Additionally, twisting the joystick 1004 clockwise rotates the roll angle of the virtual camera clockwise, and twisting the joystick 1004 counterclockwise rotates the roll angle of the virtual camera counterclockwise.

[0106] The seesaw switch 1005 can control the field of view of the virtual camera. For example, tilting the seesaw switch 1005 backward may gradually zoom in. Alternatively, tilting the seesaw switch 1005 forward may gradually widen the field of view.

[0107] Figure 11 shows an example of the UI when the recording state is transitioned by operating the controller during clip playback as described in this disclosure. This is an example of the UI when the camera path is modified and the system automatically transitions to recording state by operating one or more of the multiple operation axes of the controller shown in Figure 10 while the clip is playing.

[0108] For example, when clip 901 is playing, tilting joystick 1003 to the right changes the record start button 603 to the record stop button 701. In other words, if the user operates controller 142 while clip 901 is playing, the recording mode is turned ON and a new clip (camera path) is generated. Since the recording status becomes True, the recording indicator 702 is displayed. Clip 1101 is the newly generated clip as the modified clip. This recording status continues until playback of clip 901 is complete, at which point the recording status becomes False and recording stops. The point at which playback is complete is when the time corresponding to clip 901 has elapsed. For example, clip 901 corresponds to the time 0:10, and the point at which playback is complete is when the elapsed time associated with the clip reaches 0:10. In other words, if the user inputs a virtual camera operation while a clip is playing, a new clip corresponding to the same time as the currently playing clip is created.

[0109] As described above, this embodiment describes a configuration in which recording starts when operation information from the controller 142 is acquired during clip playback.

[0110] This makes it easier for operators to record the edited camera pass while editing the camera pass during playback.

[0111] Furthermore, it is also acceptable to have a configuration where recording is canceled if the user rewinds or stops the recording after recording has started when operation information from controller 142 is acquired during clip playback. It is also acceptable to have a configuration where the clip being recorded is discarded at that time.

[0112] Furthermore, although this embodiment describes a configuration in which recording starts when operation information from the controller 142 is acquired during clip playback, the following configuration is also acceptable. That is, recording may start when the clip is played, and if operation information from the controller 142 is not acquired until playback is complete, the recording may be canceled. In this case, the clip being recorded may be discarded.

[0113] <Example 2> In Example 1, a configuration was described in which recording starts while the camera parameters of the virtual camera are modified by operating a virtual camera control controller during clip playback. However, there is a challenge in that it is difficult for the operator to time the modification. That is, the operator has to watch the virtual viewpoint video once, remember the timing of the modification, perform playback again, and then operate the controller to make the modification from the remembered timing, which is a difficult task. In Example 2, in order to solve the above problem, a UI is described in which the operator can add start and end markings at any location on the playback bar of the virtual camera control device UI.

[0114] Figure 17 shows an example of the virtual camera control device UI in Embodiment 2. Note that configurations similar to those described in Figures 6, 8, and 9 are given the same numbers and their descriptions are omitted.

[0115] The playback bar 1701 allows the operator to understand how far a clip has been played as time progresses. The operator can register a start marker 1702 and an end marker 1703 at any position. The start marker 1702 indicates the start time of the period for modifying the camera parameters of the virtual camera. The end marker 1703 indicates the end time of the period for modifying the camera parameters of the virtual camera. The method of registering the start marker 1702 and the end marker 1703 is not limited in this disclosure, but as an example, the start marker 1702 may be registered when the left mouse button is pressed at any point on the playback bar, and the end marker may be registered when the left mouse button is pressed at any point on the playback bar. In this embodiment, markers are displayed, but this is not limited to this, and tags, icons, etc., may be displayed. The start marker 1702 and the end marker 1703 are displayed adjacent to the playback bar, but this is not limited to this. For example, they may be displayed on the playback bar. Alternatively, instead of using markers, the line indicating the playback bar could be made thicker or its color changed, as long as the start and end times are identifiable.

[0116] As described above, this embodiment describes a configuration in which the operator registers a start marker 1702 and an end marker 1703 at any position on the playback bar. This makes it easier for the operator to make corrections from the appropriate timing during playback.

[0117] Alternatively, the system could be configured to automatically switch to slow-motion playback when the frame number on the playback bar passes the start marker, making corrections easier. In that case, the system could also be configured to cancel slow-motion playback and return to normal speed playback when the frame number exceeds the end marker.

[0118] Although this disclosure has been described above based on several embodiments, this disclosure is not limited to the above embodiments, and various modifications are possible in accordance with the spirit of this disclosure, and these modifications are not excluded from the scope of this disclosure.

[0119] Furthermore, in this embodiment, some or all of the control may be provided to an image processing system, etc., via a network or various storage media, by supplying a computer program that realizes the functions of the embodiment described above. The computer (or CPU, MPU, etc.) in the image processing system, etc., may then read and execute the program. In that case, the program and the storage medium storing the program constitute the present disclosure.

[0120] Furthermore, the disclosure of this embodiment includes the following configuration, method, and program.

[0121] (Composition 1) An acquisition means that acquires first information which is information indicating the position and orientation of the virtual camera corresponding to a virtual viewpoint image generated based on multiple captured images, and the trajectory of the virtual camera corresponding to each of multiple time points, A display control means that performs a first control to display a virtual viewpoint image generated based on the first information, When an input to operate the virtual camera is obtained during the first control, a generation means generates third information including first information corresponding to the time before the input start and second information which is information indicating the position and orientation of the virtual camera corresponding to the input, An information processing device characterized by having the following features.

[0122] (Configuration 2) The information processing device according to configuration 1, characterized in that the second camera path is generated based on the input and the first camera path.

[0123] (Composition 3) The information processing device according to configuration 2, characterized in that the second camera path is generated by modifying the first camera path based on the input.

[0124] (Composition 4) The aforementioned first camera path corresponds to a predetermined time, The information processing apparatus according to any one of configurations 1 to 3, characterized in that the display control means performs control to display a time bar indicating the predetermined time and the time corresponding to the virtual viewpoint image being displayed.

[0125] (Composition 5) The information processing apparatus according to configuration 4, characterized in that the display control means performs control to display a marker indicating a specific time specified by user operation on the time bar.

[0126] (Composition 6) The information processing apparatus according to any one of claims 1 to 5, characterized by having a recording means for recording the third information.

[0127] (Composition 7) The information processing device according to any one of configurations 1 to 6, characterized in that the input corresponds to an input made by a user using a joystick.

[0128] (Composition 8) The information processing apparatus according to any one of configurations 1 to 7, characterized in that the camera parameters of the virtual camera are parameters indicating the position and orientation of the virtual camera.

[0129] (method) An acquisition step involves acquiring first information which is information indicating the position and orientation of the virtual camera corresponding to a virtual viewpoint image generated based on multiple captured images, and the position and orientation of the virtual camera corresponding to each of multiple time points. A display control step that performs a first control to display a virtual viewpoint image generated based on the first information, If an input to operate the virtual camera is obtained during the first control, a generation step is made to generate third information including first information corresponding to the time before the input and second information which is information indicating the position and orientation of the virtual camera corresponding to the input. An information processing method characterized by having the following features.

[0130] (program) A program for causing a computer to function as an information processing device as described in any one of items 1 to 8. [Explanation of symbols]

[0131] 110 Virtual Camera Control Device 112 State holding unit 113 Recording status 114 Playback status 116 Virtual camera parameter modification unit 118 Playback Department 119 Recording Department 120 Camera path holding unit

Claims

1. An acquisition means that acquires first information which is information indicating the position and orientation of the virtual camera corresponding to a virtual viewpoint image generated based on multiple captured images, and the trajectory of the virtual camera corresponding to each of multiple time points, A display control means that performs a first control to display a virtual viewpoint image generated based on the first information, When an input to operate the virtual camera is obtained during the first control, a generation means generates third information including a part of the first information corresponding to the time before the input and second information which is information indicating the position and orientation of the virtual camera corresponding to the input. An information processing device characterized by having the following features.

2. The information processing apparatus according to claim 1, characterized in that the second information is generated based on the input and the first information.

3. The information processing apparatus according to claim 2, characterized in that the second information is generated by modifying the first information based on the input.

4. The aforementioned information 1 corresponds to a predetermined time, The information processing apparatus according to claim 1, characterized in that the display control means performs control to display a time bar indicating the predetermined time and the time corresponding to the virtual viewpoint image being displayed.

5. The information processing apparatus according to claim 4, characterized in that the display control means performs control to display a marker indicating a specific time specified by user operation on the time bar.

6. The information processing apparatus according to claim 1, characterized by having a recording means for recording the third information.

7. The information processing apparatus according to claim 1, characterized in that the input corresponds to an operation by a user using a joystick.

8. An acquisition step involves acquiring first information which is information indicating the position and orientation of the virtual camera corresponding to a virtual viewpoint image generated based on multiple captured images, and the position and orientation of the virtual camera corresponding to each of multiple time points. A display control step that performs a first control to display a virtual viewpoint image generated based on the first information, If an input to operate the virtual camera is obtained during the first control, a generation step is made to generate third information including first information corresponding to the time before the input and second information which is information indicating the position and orientation of the virtual camera corresponding to the input. An information processing method characterized by having the following features.

9. A program for causing a computer to function as an information processing device according to any one of claims 1 to 7.

Images