Information processing device, information processing method, and program

Abstract

This information processing device is provided with a display processing unit for performing display processing of, as a camera work designation screen that accepts a designation operation of camera work information indicating at least a movement locus of a viewpoint in a free view point image, a screen that filters a plurality of pieces of information to display, from thereamong, a piece of camera work information corresponding to information inputted by a user.

Description

[Technical field]

[0001] The present technology relates to an information processing device, a method thereof, and a program thereof, and in particular to a processing technology relating to a free viewpoint image that allows an imaged subject to be observed from any viewpoint in a three-dimensional space. [Background technology]

[0002] A technology is known that generates a free viewpoint image (also called a free viewpoint video, virtual viewpoint image (video), etc.) that corresponds to an observation image from any viewpoint in three-dimensional space, based on three-dimensional information that represents an image of an imaged subject in three-dimensional space. As a related prior art, the following Patent Document 1 can be mentioned. Patent Document 1 discloses a technology related to the generation of camerawork that can be said to be a movement trajectory of a viewpoint. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2018 / 030206 publication Summary of the Invention [Problem to be solved by the invention]

[0004] Free viewpoint images are also useful as broadcast content, and are used, for example, as replay images in sports broadcasts. For example, in broadcasts of soccer or basketball, a clip of a few seconds, such as a shooting scene, is created from images recorded in real time and broadcast as a replay image. In this disclosure, a "clip" refers to an image of a certain scene that is cut out from a recorded image and further processed.

[0005] Meanwhile, in broadcasting sites, especially in the case of live broadcasting, operators are required to quickly create clips for replay and broadcast them. For example, there is a demand to broadcast a replay 10 or so seconds after a certain play. This demand is also applicable to the creation of clips including free viewpoint images, and therefore there is a demand to create free viewpoint images quickly.

[0006] The present technology has been made in consideration of the above circumstances, and has an object to enable the task of creating a free viewpoint image to be performed quickly. [Means for solving the problem]

[0007] The information processing device related to the present technology is equipped with a display processing unit that performs display processing of a screen showing camera work information that corresponds to user input information among multiple camera work information, as a camera work designation screen that accepts an operation to designate camera work information, which is information that indicates at least the movement trajectory of the viewpoint in a free viewpoint image. By filtering and displaying camera work information according to information input by the user, it becomes possible to make it easier for the user to find the camera work information they desire, thereby shortening the time required to specify camera work information.

[0008] In the information processing device related to the above-mentioned present technology, the display processing unit can be configured to perform a process of filtering and displaying camera work information corresponding to keywords as the input information on the camera work designation screen. This makes it possible to perform appropriate filtering of camera work information that reflects the user's intentions.

[0009] In the information processing device related to the above-mentioned present technology, the camerawork designation screen displays filtering condition information indicating filtering conditions for camerawork information, and the display processing unit can be configured to perform processing to filter and display the camerawork information using the filtering conditions indicated by the filtering condition information selected as the input information. This makes it possible to reduce the operation required for filtering and displaying camera work information to just selecting filtering condition information.

[0010] In the information processing device related to the above-mentioned present technology, the display processing unit can be configured to perform processing to display information that visualizes the movement trajectory of the viewpoint on the camerawork designation screen. By displaying information that visualizes the movement trajectory of the viewpoint, it becomes easier for users to imagine what kind of camera work is being performed.

[0011] In the information processing device related to the above-mentioned present technology, the display processing unit can be configured to perform processing to display camera arrangement position information indicating the arrangement positions of multiple cameras that capture images for generating a free viewpoint image on the camerawork designation screen. By displaying information indicating the placement position of each camera, the user can easily imagine what kind of image should be generated as a free viewpoint image.

[0012] In the information processing device related to the above-mentioned present technology, the display processing unit can be configured to perform processing to display, on the camerawork designation screen, start point placement position information and end point placement position information indicating the respective positions of the camera that is the start point and end point of the movement of the viewpoint among the multiple cameras. This allows the user to understand from which camera position the movement of the viewpoint begins and ends.

[0013] In the information processing device related to the above-mentioned present technology, the display processing unit can be configured to perform processing to display the start point placement position information and the end point placement position information, and placement position information of the cameras other than the camera that is the start point of movement and the camera that is the end point of movement among the multiple cameras, in different manners. This allows the user to intuitively grasp at which camera position the movement of the viewpoint starts and ends.

[0014] In the information processing device related to the above-mentioned present technology, the display processing unit can be configured to perform processing to display information that visualizes the movement speed of the viewpoint on the camerawork designation screen. During the period in which the viewpoint is moved, the period in which the movement speed of the viewpoint is changed is considered to be an important element in creating a free viewpoint image.

[0015] In the information processing device related to the above-mentioned present technology, the display processing unit can be configured to perform processing to display information indicating a period during which the movement speed of the viewpoint decreases, as information that visualizes the movement speed of the viewpoint. During the period in which the viewpoint is moved, the period in which the movement speed of the viewpoint is reduced is considered to be an important factor in creating a free viewpoint image.

[0016] In the information processing device related to the above-mentioned present technology, the display processing unit can be configured to perform processing to display information that visualizes the field of view range from the viewpoint on the camerawork designation screen. By visually indicating the field of view, it becomes easier for the user to understand the camera work.

[0017] In the information processing device according to the present technology described above, the display processing unit can be configured to perform processing for displaying, on the camera work designation screen, a target that determines a line of sight direction from the viewpoint. This allows the user to easily grasp at what position in three-dimensional space the subject is targeted by the camera work.

[0018] In the information processing device related to the above-mentioned present technology, it is possible to configure it to include a camerawork editing processing unit that updates the information on the position of the target in the camerawork information in response to a change in the position of the target on the camerawork designation screen. This makes it possible to eliminate the need to launch software for generating camerawork information when editing the camerawork information at the stage of specifying the camerawork information to be used in generating a free viewpoint image.

[0019] In the information processing device according to the present technology described above, the display processing unit can be configured to perform processing for displaying an image of a three-dimensional space observed from the viewpoint on the camerawork designation screen. This makes it possible to provide a preview of an image similar to a free viewpoint image generated based on camera work information to the user, making it easier for the user to understand the camera work.

[0020] In the information processing device related to the above-mentioned present technology, the display processing unit can be configured to perform processing to display an image rendered from a virtual three-dimensional model of real space as an image of the three-dimensional space observed from the viewpoint. This makes it possible to realize a preview display of an observed image from a viewpoint without the need to perform rendering processing using a three-dimensional model generated from a captured image of the target real space.

[0021] In the information processing device related to the above-mentioned present technology, the display processing unit can be configured to perform display processing of information notifying a camera among the multiple cameras in which a change in the field of view has been detected. In generating a free viewpoint image, in order to accurately generate three-dimensional information from images captured by multiple cameras, it is necessary for each camera to maintain a pre-assumed position and orientation, and if a change in position or orientation occurs in any of the cameras, calibration of the parameters used to generate the three-dimensional information is required. By notifying the user of the camera in which a change in the field of view has been detected as described above, it is possible to notify the user of the camera that requires calibration.

[0022] The information processing method related to the present technology is an information processing method in which an information processing device performs display processing of a screen that shows camera work information that corresponds to user input information among multiple camera work information, as a camera work designation screen that accepts an operation to designate camera work information, which is information that indicates at least the movement trajectory of the viewpoint in a free viewpoint image. With such an information processing method according to the present technology, the same effect as that of the information processing device according to the present technology described above can be obtained.

[0023] The program related to the present technology is a program readable by a computer device, and causes the computer device to realize a function of performing display processing of a screen that shows camera work information that corresponds to user input information among multiple camera work information, as a camera work designation screen that accepts designation operations of camera work information, which is information that indicates at least the movement trajectory of the viewpoint in a free viewpoint image. Such a program realizes the information processing device according to the present technology described above. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is a block diagram of a system configuration according to an embodiment of the present technology. [Diagram 2] 1 is an explanatory diagram of an example of a camera arrangement for generating a free viewpoint image according to an embodiment. [Diagram 3] FIG. 1 is a block diagram of a hardware configuration of an information processing apparatus according to an embodiment. [Figure 4]FIG. 2 is an explanatory diagram of functions of an image creation controller according to the embodiment. [Diagram 5] FIG. 2 is an explanatory diagram of functions of a free viewpoint image server according to an embodiment. [Figure 6] FIG. 2 is an explanatory diagram of a viewpoint in a free viewpoint image according to the embodiment. [Figure 7] FIG. 2 is an explanatory diagram illustrating an overview of a camera work designation screen in the embodiment. [Figure 8] FIG. 2 is an explanatory diagram of an overview of a creation operation screen in the embodiment. [Figure 9] FIG. 4 is an explanatory diagram of an output clip according to the embodiment. [Figure 10] 11 is an explanatory diagram of an output clip including a still image FV clip according to an embodiment. FIG. [Figure 11] 11 is an explanatory diagram of an output clip including a moving image FV clip according to an embodiment. FIG. [Figure 12] 11A and 11B are diagrams illustrating an example of an image of an output clip according to the embodiment. [Figure 13] 1A to 1C are diagrams illustrating a procedure for creating a clip according to an embodiment. [Figure 14] 10A to 10C are diagrams illustrating a procedure for detecting camera fluctuations according to an embodiment. [Figure 15] FIG. 13 is a diagram illustrating an example of an initial screen of a creation operation screen according to the embodiment. [Figure 16] FIG. 11 is a diagram for explaining an example of an operation for acquiring a preset list of a camera. [Figure 17] FIG. 11 is an explanatory diagram of a change in a background 3D model. [Figure 18] FIG. 13 is an explanatory diagram of a change in a background 3D model. [Figure 19] FIG. 13 is an explanatory diagram of adding a camera work entry. [Figure 20] 11A and 11B are explanatory diagrams of an example of displaying the field of view and observed images of a specified camera. [Figure 21] FIG. 13 is an explanatory diagram of an example of displaying the viewing range and observed image of a specified camera. [Figure 22] FIG. 11 is an explanatory diagram of an example of an In-camera designation technique. [Diagram 23] FIG. 13 is a diagram showing an example of a screen display when an In camera is specified. [Figure 24] FIG. 11 is an explanatory diagram of an example of a method for specifying the Out camera. [Diagram 25] FIG. 13 is a diagram showing an example of a screen display when the Out camera is specified. [Figure 26] 11A to 11C are diagrams illustrating changes in screen display in response to operation of a seek bar in a timeline operation section. [Figure 27] 11 is an explanatory diagram of an example of a method for specifying a waypoint of a viewpoint. [Figure 28] FIG. 13 is an explanatory diagram of an example of a method for specifying a waypoint of the viewpoint. [Figure 29] FIG. 13 is an explanatory diagram of an example of a screen display when the designation of a waypoint of the viewpoint is completed; [Diagram 30] FIG. 13 is an explanatory diagram of an example of a screen display when a plurality of waypoints are specified. [Diagram 31] 11A and 11B are diagrams for explaining an example of a technique for specifying a shape type of a movement trajectory of a viewpoint. [Diagram 32] FIG. 13 is a diagram for explaining an example of a method for specifying a shape type of a movement trajectory of a viewpoint. [Diagram 33] 13 is an explanatory diagram of an example of a screen display when a shape type of a movement trajectory of a viewpoint is specified; FIG. [Diagram 34] 11A and 11B are diagrams illustrating an example of a technique for specifying a movement speed of a viewpoint. [Diagram 35] 13 is an explanatory diagram of an example of a screen display when a moving speed of the viewpoint is specified; FIG. [Diagram 36] FIG. 2 is an explanatory diagram of the significance of a target in the embodiment. [Figure 37] FIG. 11 is an explanatory diagram of the meaning of a target in the embodiment. [Figure 38] FIG. 11 is an explanatory diagram of target movement designation in the embodiment. [Figure 39] FIG. 11 is an explanatory diagram of target movement designation in the embodiment. [Diagram 40]FIG. 11 is an explanatory diagram of target movement designation in the embodiment. [Diagram 41] FIG. 11 is an explanatory diagram of target movement designation in the embodiment. [Diagram 42] FIG. 11 is an explanatory diagram of target movement designation in the embodiment. [Diagram 43] 13 is an explanatory diagram of an example of an operation for specifying a period during which the target is faced. FIG. [Diagram 44] FIG. 13 is an explanatory diagram of an example of a designation operation for a period during which the user faces a target. [Diagram 45] FIG. 13 is an explanatory diagram of an example of a designation operation for a period during which the user faces a target. [Figure 46] 13A and 13B are diagrams showing examples of how a preview image is displayed when a designation regarding movement of a target is made. [Figure 47] FIG. 13 is a diagram showing an example of a preview image displayed when a target movement is designated. [Figure 48] FIG. 13 is a diagram showing an example of a preview image displayed when a target movement is designated. [Figure 49] FIG. 13 is a diagram showing an example of a preview image displayed when a target movement is designated. [Figure 50] FIG. 13 is a diagram showing another setting example of the period during which the target is faced. [Figure 51] FIG. 13 is an explanatory diagram of an example in which the period during which the target is faced is individually specified for multiple added targets. [Figure 52] 13 is a flowchart of a process related to generation and display of a viewpoint movement trajectory in accordance with a designation of an In camera and an Out camera. [Diagram 53] 11 is a flowchart showing a process for generating and displaying a viewpoint movement trajectory in response to a designated waypoint. [Figure 54] FIG. 11 is a diagram illustrating an example of an initial screen of a camera work designation screen according to an embodiment. [Figure 55] 11 is an explanatory diagram of an example of an image import method used to generate a free viewpoint image. [Figure 56] FIG. 11 is an explanatory diagram of an example of an image import method used to generate a free viewpoint image. [Figure 57] FIG. 13 is a diagram showing an example of a screen display after importing an image. [Figure 58] FIG. 2 is a diagram showing an example of a display of an image viewed from an X-axis perspective. [Figure 59] FIG. 13 is a diagram showing a display example of an image viewed from the Y-axis perspective. [Figure 60] FIG. 13 is a diagram showing an example of a display of an image viewed from the Z axis. [Figure 61] FIG. 2 is a diagram showing an example of a display of an image from the Pe viewpoint. [Figure 62] FIG. 13 is a diagram showing an example of a display of an image from the Ca viewpoint. [Figure 63] 11A to 11C are diagrams for explaining an example of an operation procedure for filtering and displaying camera work. [Figure 64] FIG. 13 is a diagram showing an example of a camera work filtering display. [Figure 65] 11 is an explanatory diagram of a reset button of a filtering operation unit in the embodiment. FIG. [Figure 66] FIG. 13 is an explanatory diagram of a modified example of the filtering operation unit. [Figure 67] FIG. 13 is an explanatory diagram of another modified example of the filtering operation unit. [Figure 68] 13 is a diagram showing a display example of visualization information of the movement speed of the viewpoint. FIG. [Figure 69] 13A and 13B are diagrams showing other display examples of visualization information of the movement speed of the viewpoint. [Figure 70] 13 is an explanatory diagram of editing of target positions on the camerawork designation screen of the embodiment. FIG. [Figure 71] FIG. 13 is an explanatory diagram of editing of target positions on the camerawork designation screen of the embodiment. [Figure 72] 13A and 13B are diagrams illustrating examples of display of notification information of a camera in which a change has been detected. [Figure 73] 11 is a flowchart illustrating a process related to filtering of camera work based on tag information displayed on a screen. [Figure 74] 11 is a flowchart illustrating a process for filtering camera work according to an input keyword. [Figure 75] 11 is a flowchart illustrating a process related to a notification of a camera that requires calibration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The embodiments will be described below in the following order. <1. System configuration> <2. Configuration of Image Creation Controller and Free Viewpoint Image Server> <3. GUI Overview> <4. Clips containing free viewpoint images> <5. Clip creation process> <6. Camera movement detection> <7. GUI for creating camerawork> <8. GUI for creating free viewpoint images> <9. Variations> <10. Summary of the embodiment> <11. This Technology>

[0026] <1. System configuration> FIG. 1 shows an example of the configuration of an image processing system according to an embodiment of the present technology. The image processing system includes an image creation controller 1, a free viewpoint image server 2, a video server 3, multiple (e.g., four) video servers 4A, 4B, 4C, and 4D, a NAS (Network Attached Storage) 5, a switcher 6, an image conversion unit 7, a utility server 8, and multiple (e.g., 16) imaging devices 10. In the following description, the term "camera" refers to the imaging device 10. For example, the term "camera arrangement" refers to an arrangement of multiple imaging devices 10. Moreover, when the video servers 4A, 4B, 4C, and 4D are referred to collectively without distinction, they will be referred to as "video server 4." In this image processing system, based on captured images (e.g., image data V1 to V16) obtained from multiple imaging devices 10, free viewpoint images corresponding to images observed from any viewpoint in three-dimensional space can be generated, and an output clip including the free viewpoint images can be created.

[0027] In FIG. 1, the connection state of each part is indicated by a solid line, a dashed line, or a double line. The solid lines indicate connections for SDI (Serial Digital Interface), an interface standard for connecting broadcasting equipment such as cameras and switchers, and are compatible with 4K, for example. Image data is mainly sent and received between each device via SDI wiring.

[0028] The double lines indicate connections of communication standards for constructing a computer network, such as 10 Gigabit Ethernet. The image creation controller 1, free viewpoint image server 2, video server 3, 4A, 4B, 4C, 4D, NAS 5, and utility server 8 are connected by a computer network, enabling them to send and receive image data and various control signals to and from each other.

[0029] The dashed line between video servers 3 and 4 indicates that video servers 3 and 4 equipped with inter-server file sharing functionality are connected by, for example, a 10G network. This allows each video server between video server 3 and video servers 4A, 4B, 4C, and 4D to preview and transmit materials stored in other video servers. In other words, a system using multiple video servers is constructed, enabling efficient highlight editing and transmission.

[0030] Each imaging device 10 is configured as a digital camera device having an imaging element such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, and obtains captured images (image data V1 to V16) as digital data. In this example, each imaging device 10 obtains captured images as moving images.

[0031] In this example, each imaging device 10 captures an image of a game such as basketball or soccer being played, and each is disposed in a predetermined position and a predetermined orientation in the venue where the game is held. In this example, the number of imaging devices 10 is 16, but the number of imaging devices 10 may be at least two in order to enable the generation of a free viewpoint image. By increasing the number of imaging devices 10 and capturing images of the target subject from more angles, the accuracy of the three-dimensional reconstruction of the subject can be improved, and the image quality of the virtual viewpoint image can be improved.

[0032] Fig. 2 shows an example of the placement of imaging devices 10 around a basketball court. Let us assume that the circles represent imaging devices 10. This is an example of the camera placement when one wishes to focus on capturing images of the area near the goal on the left side of the drawing. Of course, the placement and number of cameras are just an example, and should be set according to the content and purpose of the filming or broadcasting.

[0033] The image creation controller 1 is configured by an information processing device. This image creation controller 1 can be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like. The image creation controller 1 controls and manages the operations of the video servers 3 and 4, and performs processing for creating clips. As an example, the image creation controller 1 is a device that can be operated by an operator OP1. The operator OP1, for example, selects clip contents and issues instructions for creation.

[0034] The free viewpoint image server 2 is configured as an information processing device that actually performs processing to create a free viewpoint image (an FV (Free View) clip to be described later) in response to an instruction from the image creation controller 1. This free viewpoint image server 2 can also be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like. As an example, the free viewpoint image server 2 is a device that can be operated by an operator OP2. The operator OP2 performs work related to creating an FV clip as a free viewpoint image, for example. Specifically, the operator OP2 performs a designation operation (selection operation) of camera work for generating a free viewpoint image, etc. In this example, the operator OP2 also performs a work of creating camera work.

[0035] The configurations and processes of the image creation controller 1 and the free viewpoint image server 2 will be described in detail later. In addition, although it is assumed that the operations are performed by operators OP1 and OP2, for example, the image creation controller 1 and the free viewpoint image server 2 may be arranged side by side and operated by a single operator.

[0036] Each of the video servers 3 and 4 is an image recording device, and includes a data recording unit such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), and a control unit that controls recording and playback of data in the data recording unit.

[0037] Each of the video servers 4A, 4B, 4C, and 4D can accept, for example, four input systems, and each of them simultaneously records images captured by the four imaging devices 10. For example, the video server 4A records image data V1, V2, V3, and V4. The video server 4B records image data V5, V6, V7, and V8. The video server 4C records image data V9, V10, V11, and V12. The video server 4D records image data V13, V14, V15, and V16. As a result, images captured by all 16 imaging devices 10 are recorded simultaneously. The video servers 4A, 4B, 4C, and 4D are assumed to perform continuous recording, for example, during a sports match to be broadcast.

[0038] The video server 3 is, for example, directly connected to the image creation controller 1, and is capable of, for example, two input systems and two output systems. Image data Vp and Vq are shown as the two input systems. As the image data Vp and Vq, it is possible to select images captured by any two of the image capture devices 10 (any two of the image data V1 to V16). Of course, images captured by other image capture devices may also be used.

[0039] The image data Vp, Vq can be displayed as a monitor image on a display by the image creation controller 1. The operator OP1 can check the status of a scene being shot and recorded for broadcast, for example, by the image data Vp, Vq input to the video server 3. In addition, since the video servers 3 and 4 are connected in a file sharing state, the image creation controller 1 can also display on the monitor the images captured by each imaging device 10 that are recorded in the video servers 4A, 4B, 4C, and 4D, allowing the operator OP1 to check them one by one.

[0040] In this example, a time code is added to the image captured by each imaging device 10, making it possible to achieve frame synchronization in the processing in the video servers 3, 4A, 4B, 4C, and 4D.

[0041] The NAS 5 is a storage device arranged on a network, and is composed of, for example, an SSD, an HDD, etc. In this example, the NAS 5 is a device that stores some frames of the image data V1, V2, ..., V16 recorded in the video servers 4A, 4B, 4C, and 4D for processing in the free viewpoint image server 2 when the frames are transferred for generating free viewpoint images, and stores the generated free viewpoint images.

[0042] The switcher 6 is a device that receives the images output via the video server 3 and selects the main line image PGMout to be finally selected and broadcast. For example, a broadcast director or the like performs the necessary operations.

[0043] The image conversion unit 7 performs, for example, resolution conversion and synthesis of image data from the imaging device 10, generates a monitoring image of the camera arrangement, and supplies it to the utility server 8. For example, 16-channel image data (V1 to V16) that is an 8K image is resolution-converted to a 4K image, and then the image is converted to four-channel images arranged in a tiled pattern, and supplied to the utility server 8.

[0044] The utility server 8 is a computer device capable of various related processes, but in this example, it is a device that performs a process of detecting camera movement for calibration in particular. For example, the utility server 8 detects camera movement by monitoring image data from the image conversion unit 7. Camera movement refers to movement of any of the positions of the imaging devices 10 arranged, for example, as shown in FIG. 2. Information on the position of the imaging devices 10 is an important element in generating a free viewpoint image, and if the position of the imaging devices 10 changes, it is necessary to reset the parameters. For this reason, the camera movement is monitored.

[0045] <2. Configuration of Image Creation Controller and Free Viewpoint Image Server> The image creation controller 1, the free viewpoint image server 2, the video servers 3 and 4, and the utility server 8 in the above configuration can be realized as an information processing device 70 having the configuration shown in FIG.

[0046] 3, a CPU 71 of an information processing device 70 executes various processes according to a program stored in a ROM 72 or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes as appropriate. The CPU 71, the ROM 72, and the RAM 73 are interconnected via a bus 74. To this bus 74, an input / output interface 75 is also connected.

[0047] The input / output interface 75 is connected to an input unit 76 including an operator and an operating device. For example, the input unit 76 may be various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71.

[0048] Further, the input / output interface 75 is connected, either integrally or separately, to a display unit 77 formed of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, or the like, and an audio output unit 78 formed of a speaker, or the like. The display unit 77 is a display unit that performs various displays, and is configured, for example, by a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, or the like. The display unit 77 displays images for various types of image processing, moving images to be processed, etc., on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface), based on instructions from the CPU 71.

[0049] The input / output interface 75 may be connected to a storage unit 79 configured with a hard disk or solid-state memory, or a communication unit 80 configured with a modem or the like. The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like.

[0050] A drive 82 is also connected to the input / output interface 75 as required, and a removable recording medium 81 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. The drive 82 allows data files such as image files MF and various computer programs to be read from the removable recording medium 81. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the sound output unit 78. Furthermore, the computer programs and the like read from the removable recording medium 81 are installed in the storage unit 79 as necessary.

[0051] In this information processing device 70, software can be installed via network communication by a communication unit 80 or via a removable recording medium 81. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0052] When the image creation controller 1 and the free viewpoint image server 2 are realized using such an information processing device 70, the processing functions as shown in FIGS. 4 and 5 are realized in the CPU 71 by software, for example.

[0053] FIG. 4 shows a section specification processing section 21, a target image transmission control section 22, and an output image generating section 23 as functions formed in a CPU 71 of an information processing device 70 serving as the image creation controller 1.

[0054] The section identification processing unit 21 performs a process of identifying a target image section for generating a free viewpoint image for a plurality of captured images (image data V1 to V16) captured simultaneously by a plurality of imaging devices 10. For example, in response to an operation by an operator OP1 to select a scene to be replayed within an image, the section identification processing unit 21 performs a process of identifying a time code for the scene, particularly for a section of the scene to be a free viewpoint image (target image section for generation), and notifying the free viewpoint image server 2 of the time code.

[0055] The target image section here refers to the frame section that will actually be used as a free viewpoint image. When generating a free viewpoint image for a single frame in a video, that single frame will be the target image section. In this case, the in and out points for the free viewpoint image will have the same time code. When generating a free viewpoint image for a section of multiple frames in a video, the multiple frames become the image section to be generated. In this case, the in point and out point for the free viewpoint image have different time codes. Although the structure of a clip will be described later, it is assumed that the in / out points of the image section to be generated will be different from the in / out points of the final output clip, because a previous clip and a next clip will be combined, as described later.

[0056] The target image transmission control unit 22 controls the transmission of image data of the image section to be generated in each of the multiple captured images 10, i.e., one or multiple frames of the image data V1 to V16, as image data used for generating a free viewpoint image in the free viewpoint image server 2. Specifically, the control unit 22 controls the transfer of the image data as the image section to be generated from the video servers 4A, 4B, 4C, and 4D to the NAS 5.

[0057] The output image generating unit 23 performs processing for generating an output image (output clip) including the free viewpoint image (FV clip) generated by the free viewpoint image server 2 and received. For example, the image creation controller 1 generates an output clip by combining, on the time axis, a previous clip, which is an actual moving image at a time point before the FV clip, which is a virtual image generated by the free viewpoint image server 2, and a subsequent clip, which is an actual moving image at a time point after the FV clip, through processing by the output image generating unit 23. In other words, the previous clip + FV clip + subsequent clip are treated as one output clip. Of course, the previous clip and the FV clip may be one output clip. Alternatively, the FV clip and the after clip may be one output clip. Furthermore, an output clip may be generated that is only an FV clip, without combining the preceding clip or the following clip. In either case, the image creation controller 1 generates an output clip including an FV clip and outputs it to the switcher 6 so that it can be used for broadcasting.

[0058] FIG. 5 shows a target image acquisition unit 31, an image generation processing unit 32, a transmission control unit 33, and a camerawork generation processing unit 34 as functions formed in a CPU 71 of an information processing device 70 serving as the free viewpoint image server 2.

[0059] The target image acquisition unit 31 performs a process of acquiring image data of a target image section for generating a free viewpoint image in each of a plurality of captured images (image data V1 to V16) captured simultaneously by a plurality of imaging devices 10. That is, the image creation controller 1 acquires image data of one frame or multiple frames specified by the in-point / out-point of the target image section identified by the function of the section identification processing unit 21 from the video servers 4A, 4B, 4C, 4D via the NAS 5, so that the image data can be used for generating a free viewpoint image.

[0060] For example, the target image acquisition unit 31 acquires image data of one frame or multiple frames of the image section to be generated for all of the image data V1 to V16. The image data of the image section to be generated for all of the image data V1 to V16 is acquired in order to generate a high-quality free viewpoint image. As described above, it is possible to generate a free viewpoint image by using images captured by at least two or more imaging devices 10, but by increasing the number of imaging devices 10 (i.e., the number of viewpoints), it is possible to generate a more detailed 3D model and generate a high-quality free viewpoint image. Therefore, for example, when 16 imaging devices 10 are arranged, image data of the image section to be generated is acquired for all of the image data (V1 to V16) of the 16 imaging devices 10.

[0061] The image generation processing unit 32 has a function of generating a free viewpoint image, that is, an FV clip in this example, using the image data acquired by the target image acquisition unit 31. For example, the image generation processing unit 32 performs processes such as modeling processing including 3D model generation and subject analysis, and rendering for generating a free viewpoint image, which is a two-dimensional image, from the 3D model.

[0062] 3D model generation is a process of generating 3D model data that represents a subject in a three-dimensional space (i.e., that restores the three-dimensional structure of the subject from a two-dimensional image) based on images captured by each imaging device 10 and camera parameters for each imaging device 10 input from, for example, the utility server 8. Specifically, the 3D model data includes data that represents the subject in a three-dimensional coordinate system of (X, Y, Z). Based on 3D model data, subject analysis involves analyzing the position, orientation, and posture of the subject (player). Specifically, it estimates the subject's position, generates a simple model of the subject, and estimates the subject's orientation. A free viewpoint image is then generated based on the 3D model data and the subject analysis information. For example, a free viewpoint image is generated in which the viewpoint is moved for a 3D model of a stationary subject, a player.

[0063] The viewpoint of a free viewpoint image will be described with reference to FIG. 6A shows an image of a free viewpoint image in which a subject is captured from a required viewpoint set in a three-dimensional space. In this free viewpoint image, the subject S1 is viewed substantially from the front, and the subject S2 is viewed substantially from the back. Fig. 6B shows an image of a virtual viewpoint image when the viewpoint position is changed in the direction of arrow C in Fig. 6A and a viewpoint that views subject S1 from approximately the back is set. In the free viewpoint image in Fig. 6B, subject S2 is viewed approximately from the front, and subject S3 and a basketball goal that were not shown in Fig. 6A are shown. For example, an image of about 1 to 2 seconds in which the viewpoint is gradually moved from the state of Fig. 6A in the direction of arrow C to the state of Fig. 6B is generated as a free viewpoint image (FV clip). Of course, the time length of the FV clip as a free viewpoint image and the trajectory of the viewpoint movement can be variously considered.

[0064] Here, the free viewpoint image server 2 (CPU 71) in this example has a function as a display processing unit 32a as part of the functions of the image generation processing unit 32. The display processing unit 32a performs display processing of a camerawork designation screen Gs that accepts designation operations of camerawork information used for generating a free viewpoint image. Note that details of the camerawork related to the free viewpoint image and the camerawork designation screen Gs will be described later.

[0065] Furthermore, the free viewpoint image server 2 in this example also has a function as a camerawork editing processor 32b as part of the functions of the image generation processor 32, and this function as the camerawork editing processor 32b will also be described later.

[0066] The transmission control unit 33 controls the transmission of the free viewpoint image (FV clip) generated by the image generation processing unit 32 to the image creation controller 1 via the NAS 5. In this case, the transmission control unit 33 controls so that accompanying information for generating an output image is also transmitted to the image creation controller 1. The accompanying information is assumed to be information specifying the images of the previous clip and the next clip. That is, the accompanying information is information specifying which image from the image data V1 to V16 is to be used to create (cut out) the previous clip and the next clip. Information specifying the duration of the previous clip and the next clip is also assumed to be the accompanying information.

[0067] The camerawork generation processing unit 34 performs processing related to the generation of camerawork information used to generate a free viewpoint image. When creating a free viewpoint image, a plurality of candidate cameraworks are created in advance to accommodate various scenes. In order to enable such cameraworks to be created in advance, a software program for creating cameraworks is installed in the free viewpoint image server 2 of this example. The camerawork generation processing unit 34 is a function realized by this software program, and performs a process of generating cameraworks based on a user's operation input. The camerawork generation processing unit 34 has a function as a display processing unit 34a. The display processing unit 34a performs display processing of a creation operation screen Gg so as to enable reception of various operation inputs for camerawork creation by a user (operator OP2 in this example).

[0068] <3. GUI Overview> 7 and 8, the camerawork designation screen Gs used for creating a free viewpoint image and the creation operation screen Gg used for creating a camerawork will be outlined. In this example, the camerawork designation screen Gs and the creation operation screen Gg are displayed on the display unit 77 in the free viewpoint image server 2, for example, and can be confirmed and operated by the operator OP2.

[0069] On the camerawork designation screen Gs shown in FIG. 7, a scene window 41, a scene list display section 42, a camerawork window 43, a camerawork list display section 44, a parameter display section 45, and a transmission window 46 are arranged. In the scene window 41, for example, an image of the image section to be generated is displayed on a monitor, so that the operator OP2 can check the contents of the scene for which a free viewpoint image is to be generated. For example, a list of scenes designated as the image interval to be generated is displayed in the scene list display section 42. The operator OP2 can use the scene list display section 42 to select a scene to be displayed in the scene window 41.

[0070] The camerawork window 43 displays the positions of the imaging devices 10 that are arranged, the selected camerawork, or a plurality of selectable cameraworks.

[0071] Here, the camerawork information is information that indicates at least the movement trajectory of the viewpoint in the free viewpoint image. For example, when creating an FV clip in which the position of the viewpoint, the line of sight, and the angle of view (focal length) are changed for a subject for which a 3D model has been generated, the parameters required to determine the movement trajectory of the viewpoint, the change in the line of sight, and the change in the angle of view are regarded as the camerawork information. In the camerawork window 43, at least information showing a visualized movement trajectory of the viewpoint is displayed as a display of the camerawork.

[0072] A list of various camera work information that has been created and stored in advance is displayed in the camera work list display section 44. The operator OP2 can select and specify the camera work to be used in generating an FV clip from among the camera works displayed in the camera work list display section 44. The parameter display section 45 displays various parameters related to the selected camera work.

[0073] The transmission window 46 displays information regarding the transmission of the created FV clip to the image creation controller 1.

[0074] Next, the creation operation screen Gg in FIG. 8 will be described. On the creation operation screen Gg, a preset list display section 51, a camerawork list display section 52, a camerawork window 53, an operation panel section 54, and a preview window 55 are arranged.

[0075] The preset list display section 51 can selectively display a camera preset list, a target preset list, and a 3D model preset list. The camera preset list is a list of position information (position information in three-dimensional space) preset by the user for the camera placement position on-site. As described later, when the camera preset list is selected, the preset list display unit 51 displays a list of information indicating the position for each camera identification information (e.g., camera1, camera2, . . ., camera16).

[0076] In addition, in the preset list of targets, a target means a target position that determines the line of sight from a viewpoint in a free viewpoint image. In generating a free viewpoint image, the line of sight from the viewpoint is determined to face the target. When a preset list of targets is selected, the preset list display section 51 displays a list of identification information about targets preset by the user and information indicating their positions. Hereinafter, the target that determines the line of sight from the viewpoint in the free viewpoint image as described above will be referred to as "target Tg."

[0077] The 3D model preset list is a preset list of 3D models to be displayed as the background of the camerawork window 43, and when a 3D model preset list is selected, the preset list display unit 51 displays a list of identification information of the preset 3D models.

[0078] The camerawork list display section 52 is capable of displaying a list of camerawork information created through the creation operation screen Gg and camerawork information (entries, described later) to be newly created through the creation operation screen Gg.

[0079] In the camerawork window 53, at least information showing a visualized movement trajectory of the viewpoint is displayed as a display of the camerawork. The operation panel section 54 is an area for receiving various operational inputs for creating camerawork. An observation image from the viewpoint is displayed in the preview window 55. When an operation for moving the viewpoint on the movement trajectory is performed, the observation images from each viewpoint position on the movement trajectory are sequentially displayed in the preview window 55. As will be described later, in a state where a camera preset list is displayed in the preset list display section 51, when an operation for specifying a camera from the camera preset list is performed, the preview window 55 in this example displays an observation image from the arrangement position of the camera.

[0080] Details of the camerawork specification screen Gs shown in FIG. 7 and a specific procedure for specifying camerawork, as well as details of the creation operation screen Gg shown in FIG. 8 and a specific procedure for creating camerawork, will be explained later.

[0081] <4. Clips containing free viewpoint images> Next, an output clip including an FV clip as a free viewpoint image will be described. FIG. 10 shows an example of an output clip, which is configured by linking a previous clip, an FV clip, and a next clip.

[0082] For example, the previous clip is an actual moving image in the section from time code TC1 to TC2 in certain image data Vx among image data V1 to V16. The subsequent clip is an actual moving image in the section from time code TC5 to TC6 in certain image data Vy among the image data V1 to V16. It is generally assumed that the image data Vx is image data of the image capturing device 10 before the viewpoint movement due to the FV clip begins, and the image data Vy is image data of the image capturing device 10 at the end of the viewpoint movement due to the FV clip.

[0083] In this example, the previous clip is a video with a duration of t1, the FV clip is a free viewpoint image with a duration of t2, and the next clip is a video with a duration of t3. The total playback time of the output clip is t1+t2+t3. For example, a 5-second output clip may consist of a 1.5-second video, a 2-second free viewpoint image, and a 1.5-second video.

[0084] Here, the FV clip is shown as the section from time code TC3 to TC4, but this may or may not correspond to the number of frames in the actual video. In other words, with an FV clip, there are cases where the viewpoint is moved while the video time is stopped (TC3 = TC4), and cases where the viewpoint is moved without stopping the video time (TC3 ≠ TC4). For the sake of explanation, an FV clip in which the viewpoint is moved while the video time is frozen will be called a "still image FV clip," and an FV clip in which the viewpoint is moved without freezing the video time will be called a "video FV clip."

[0085] If a still image FV clip is shown based on video frames, it will look like Figure 10. In this example, the time codes TC1 and TC2 of the previous clip are the time codes of frames F1 and F81, and the time code of the following frame F82 is the time code TC3=TC4 in Figure 9. And the time codes TC5 and TC6 of the next clip are the time codes of frames F83 and F166. That is, this is the case where a free viewpoint image in which the viewpoint moves is generated for a single still image of frame F82.

[0086] On the other hand, a video FV clip will look like that shown in Figure 11. In this example, the time codes TC1 and TC2 of the previous clip become the time codes of frames F1 and F101, and the time codes of frames F102 and F302 become the time codes TC3 and TC4 in Figure 9. And the time codes TC5 and TC6 of the next clip become the time codes of frames F303 and F503. That is, this is the case where a free viewpoint image in which the viewpoint moves is generated for a video in a section of a plurality of frames from frame F102 to frame 302.

[0087] Therefore, the image section to be generated determined by the image creation controller 1 is the one-frame section of frame F82 when creating the still image FV clip of Figure 10, and is the multiple-frame section from frame F102 to frame 302 when creating the video FV clip of Figure 11.

[0088] For the example still image FV clip in FIG. 10, an example of the image content of the output clip is shown in FIG. 12, the previous clip is the actual moving image from frame F1 to frame F81. The FV clip is a virtual image in which the viewpoint is moved in the scene of frame F81. The next clip is the actual moving image from frame F83 to frame F166. For example, an output clip including an FV clip is generated in this way and used as the image to be broadcast.

[0089] <5. Clip creation process> Below, a description will be given of an example of processing for creating an output clip performed in the image processing system of Fig. 1. The description will be focused mainly on the processing of the image creation controller 1 and the free viewpoint image server 2. First, the flow of processing including the operations of operators OP1 and OP2 will be described with reference to Fig. 13. Note that the processing of operator OP1 in Fig. 13 collectively shows the GUI processing of the image creation controller 1 and the operator operations. Also, the processing of operator OP2 collectively shows the GUI processing of the free viewpoint image server 2 and the operator operations.

[0090] Step S1: Select a scene When creating an output clip, the operator OP1 first selects a scene to be used as an FV clip. For example, the operator OP1 searches for a scene to be used as an FV clip while monitoring the captured images displayed on the display unit 77 on the image creation controller 1 side. Then, the operator OP1 selects an image section to be generated of one or more frames. This information on the image section to be generated is transmitted to the free viewpoint image server 2, and is made recognizable to the operator OP2 by the GUI on the display unit 77 on the free viewpoint image server 2 side. The information on the image section to be generated is specifically the information on the time codes TC3 and TC4 in Fig. 9. As described above, in the case of a still image FV clip, the time code TC3=TC4.

[0091] Step S2: Scene image transfer instruction The operator OP2 performs an operation to instruct the transfer of the image of the corresponding scene according to the designation of the image section to be generated. In response to this operation, the free viewpoint image server 2 transmits a transfer request for the image data of the section of time codes TC3 and TC4 to the image generation controller 1.

[0092] Step S3: Synchronous cutting In response to a request to transfer image data, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to extract the sections of time codes TC3 and TC4 from each of the 16 systems of image data from image data V1 to image data V16. Step S4: NAS transfer The image creation controller 1 then transfers to the NAS 5 all data in the sections of time codes TC3 and TC4 of image data V1 to V16.

[0093] Step S5: Thumbnail display The free viewpoint image server 2 displays thumbnails of the image data V1 to V16 in the section of the time codes TC3 and TC4 transferred to the NAS5. Step S6: Scene check An operator OP2 checks the scene contents of the section indicated by the time codes TC3 and TC4 on the camerawork designation screen Gs by the free viewpoint image server 2. Step S7: Camerawork selection The operator OP2 selects (specifies) camera work that he / she considers appropriate on the camera work specification screen Gs according to the contents of the scene. Step S8: Execute generation After selecting the camera work, the operator OP2 performs an operation to generate and execute an FV clip.

[0094] Step S9: Modeling The free viewpoint image server 2 generates a 3D model of the subject and performs subject analysis, etc., using frame data for the time codes TC3 and TC4 in each of the image data V1 to V16, and parameters such as the placement positions of each imaging device 10 that were previously input. Step S10: Rendering The free viewpoint image server 2 generates a free viewpoint image based on the 3D model data and the subject analysis information. At this time, the free viewpoint image is generated so that the viewpoint is moved based on the camera work selected in step S7.

[0095] Step S11: Transfer The free viewpoint image server 2 transfers the generated FV clip to the image creation controller 1. At this time, not only the FV clip but also information specifying the previous clip and the next clip and information specifying the time lengths of the previous clip and the next clip can be transmitted as accompanying information. Step S12: Quality check On the free viewpoint image server 2 side, the operator OP2 can check the quality before or after the transfer in step S11. That is, the free viewpoint image server 2 plays and displays the generated FV clip on the camerawork designation screen Gs so that the operator OP2 can check it. In some cases, it may be possible for the operator OP2 to re-create the FV clip without executing the transfer.

[0096] Step S13: Playlist generation The image creation controller 1 generates an output clip using the transmitted FV clip. In this case, the output clip is generated by combining the FV clip with either or both of the preceding and succeeding clips on the time axis. This output clip may be generated as stream data in which each frame as the previous clip, each virtually generated frame as the FV clip, and each frame as the next clip are actually linked in chronological order, but in this processing example, they are virtually linked as a playlist. That is, by generating a playlist so that a frame section as the previous clip is played, followed by the FV clip, and then the frame section as the next clip is played, it becomes possible to play the output clip without generating actual linked stream data as the output clip.

[0097] Step S14: Quality check Playback based on the playlist is performed by the GUI on the image creation controller 1 side, and an operator OP1 checks the contents of the output clip. Step S15: Playback instruction The operator OP1 issues a playback instruction by a predetermined operation in response to the quality check. The image creation controller 1 recognizes the input of the playback instruction. Step S16: Playback In response to the playback instruction, the image creation controller 1 supplies the output clip to the switcher 6. This makes it possible to broadcast the output clip.

[0098] <6. Camera movement detection> In order to generate a free viewpoint image, a 3D model is generated using image data V1, V2, . . . V16, and therefore parameters including position information of each imaging device 10 are important. For example, if the position of a certain imaging device 10 is moved during broadcasting, or the imaging direction is changed in the panning direction, tilting direction, or the like, calibration of the parameters is required accordingly. For this reason, in the image processing system of Fig. 1, the utility server 8 is configured to detect camera fluctuations. The camera fluctuation here means that at least one of the camera position and imaging direction changes.

[0099] The processing procedure of the image creation controller 1 and the utility server 8 when detecting camera fluctuations will be described with reference to Fig. 14. Note that Fig. 14 shows the processing procedure in the same format as Fig. 13, but the utility server 8 is also operated by an operator OP2 as an example.

[0100] · Step S30: HD output To detect camera movement, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to output image data to the image conversion unit 7. The images from the video servers 4A, 4B, 4C, and 4D, i.e., the images from the 16 imaging devices 10, are subjected to resolution conversion by the image conversion unit 7 and supplied to the utility server 8.

[0101] Step S31: Background generation The utility server 8 generates background images based on the supplied images. Since the background images are images that do not change unless there is a movement in the camera, for example, background images excluding subjects such as players are generated for 16 systems of image data (V1 to V16). Step S32: Check the difference The background image is displayed in the GUI, allowing operator OP2 to confirm changes in the image. Step S33: Automatic fluctuation detection By comparing background images at each point in time, camera fluctuations can also be automatically detected.

[0102] Step S34: Camera movement detection As a result of step S33 or step S32 above, a fluctuation in a certain imaging device 10 is detected. Step S35: Image acquisition Calibration becomes necessary in response to detection of a change in the imaging device 10. The utility server 8 then requests the image creation controller 1 to provide image data in a state after the change. Step S36: Extract clip In response to a request for image acquisition from the utility server 8, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to execute clip extraction for the image data V1 to V16. Step S37: NAS transfer The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data cut out as clips to the NAS 5.

[0103] Step S38: Feature point correction By transferring the image to the NAS 5, the utility server 8 can refer to and display the image in the state after the camera movement. The operator OP2 performs operations required for calibration, such as correcting feature points. Step S39: Recalibration The utility server 8 re-executes calibration for creating a 3D model using the image data (V1 to V16) in the state after the camera movement.

[0104] Step S40: Reacquire background After the calibration, in response to an operation by the operator OP2, the utility server 8 requests reacquisition of image data for the background image. Step S41: Extract clip In response to a request for image acquisition from the utility server 8, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to execute clip extraction for the image data V1 to V16. Step S42: NAS transfer The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data cut out as clips to the NAS 5. Step S43: Background generation The utility server 8 generates a background image using the image data transferred to the NAS 5. This background image is used as a reference for subsequent camera movement detection, for example.

[0105] For example, by performing camera fluctuation detection and calibration according to the above procedure, even if the position or imaging direction of the imaging device 10 is changed during broadcasting, the parameters are corrected accordingly, making it possible to continuously generate accurate FV clips.

[0106] <7. GUI for creating camerawork> Hereinafter, details of the creation operation screen Gg shown in FIG. 8, an example of a camerawork creation procedure, and various functions related to camerawork creation will be described with reference to FIGS. FIG. 15 is a diagram showing an example of the initial screen of the creation operation screen Gg. As described above, on the creation operation screen Gg, the preset list display section 51, the camerawork list display section 52, the camerawork window 53, the operation panel section 54, and the preview window 55 are arranged.

[0107] As shown in the figure, a camera button B1, a target button B2, and a 3D model button B3 are provided for the preset list display section 51. The camera button B1 is a button for issuing an instruction to display the preset list of the camera described above in the preset list display section 51, and the target button B2 and the 3D model button B3 are buttons for issuing an instruction to display the preset list of the target described above and the preset list of the 3D model as the background in the preset list display section 51, respectively. In the drawing, the camera button B1 is underlined, which means that the camera preset list display is selected.

[0108] In addition, a folder reference button B4 is provided in the preset list display section 51. By operating this folder reference button B4, the user can refer to a folder that stores data that the user wishes to display in a list in the preset list display section 51.

[0109] A new entry button B5 is provided for the camera work list display section 52. The user can issue an instruction to add a new camera work entry by operating this new entry button B5. The added camera work entry is displayed in the camera work list display section 52.

[0110] For the camerawork window 53, an X viewpoint button B6, a Y viewpoint button B7, a Z viewpoint button B8, a Ca viewpoint button B9, and a Pe viewpoint button B10 are provided. Each of these viewpoint buttons is a button for instructing an observation viewpoint for an object to be displayed in the camerawork window 53. Specifically, the X viewpoint button B6, the Y viewpoint button B7, and the Z viewpoint button B8 are buttons for instructing an X-axis viewpoint, a Y-axis viewpoint, and a Z-axis viewpoint, respectively, as viewpoints for observing the visualized information of the camerawork information to be displayed in the camerawork window 53, and the Pe viewpoint button B10 is a button for instructing a transition to a mode in which the observation viewpoint of the visualized information of the camerawork information is changed to an arbitrary position. In addition, the Ca viewpoint button B9 is a button for instructing the display of an image obtained by observing the three-dimensional space of the object from a viewpoint movement trajectory determined as the camerawork information. Please refer to FIG. 58 to FIG. 62 described later for images of the X-axis viewpoint, the Y-axis viewpoint, the Z-axis viewpoint, the Pe viewpoint, and the Ca viewpoint.

[0111] Here, in the creation operation screen Gg, the displayed images in the camera work window 53 and the preview window 55 can be enlarged or reduced in response to a predetermined operation such as a mouse wheel operation. Also, in the camera work window 53 and the preview window 55, the displayed images can be scrolled in response to a predetermined operation such as a drag operation. Note that the enlargement, reduction, and scrolling of the displayed image can also be performed in response to the operation of a button provided on the screen.

[0112] The operation panel section 54 is provided with a play button B11, a pause button B12, a stop button B13, a timeline operation section 54a, a speed adjustment operation section 56, and a path shape adjustment operation section 57. The play button B11, pause button B12, and stop button B13 are buttons for instructing playback, pausing, and stopping, respectively, of the visualized image of camerawork information displayed in the camerawork window 53, and the observed image from the viewpoint displayed in the preview window 55. The play button B11, pause button B12, and stop button B13 are enabled at a stage when at least information on the movement trajectory of the viewpoint has been determined as camerawork information.

[0113] The timeline operation section 54a is an area for accepting operations related to the creation of camerawork on a timeline that represents a period during which the viewpoint of a free viewpoint image moves. An example of an operation on the timeline operation section 54a is an operation of dragging and dropping one of the cameras displayed in a list on the preset list display section 51 to an arbitrary position on the timeline (i.e., an arbitrary time point within the period during which the viewpoint moves) (see Figs. 27 to 29). As will be described later, this operation functions as an operation for specifying the timing at which the viewpoint passes the position of the dragged and dropped camera within the period during which the viewpoint moves.

[0114] Various operation buttons for adjusting the movement speed of the viewpoint are arranged on speed adjustment operation section 56. Moreover, various operation buttons for adjusting the shape of the movement trajectory of the viewpoint are arranged on trajectory shape adjustment operation section 57. The speed adjustment operation section 56 and the trajectory shape adjustment operation section 57 will be described later.

[0115] This section explains the camerawork creation procedure and various functions related to camerawork creation. First, a user (operator OP2 in this example) performs an operation for acquiring a camera preset list as shown in Fig. 16. This operation is an operation for acquiring a camera preset list indicating the positions of the cameras actually installed at the site. The camera's preset list can be obtained by operating the folder reference button B4 and specifying the corresponding folder.

[0116] When a folder is specified, the display processing unit 34a performs processing to display on the preset list display unit 51 a preset list of the camera according to the data contents of the specified folder. At the same time, the display processing unit 34a performs a process of displaying information visually indicating the arrangement of each camera based on the acquired camera position information in the camerawork window 53. Specifically, the display processing unit 34a performs a process of displaying a camera position mark Mc indicating the position of each camera. Regarding the display of the camera position marks Mc, each camera may be displayed in a different color. For example, each camera may be displayed in a different color in the camera preset list, and each camera position mark Mc may be displayed in the same color in the camerawork window 43. Also, in the camerawork window 53, it is conceivable to display camera identification information (for example, camera1, camera2, . . ., cameraX, etc.) for the camera position mark Mc that is placed over with the mouse.

[0117] In this example, the 3D model displayed as the background in the camerawork window 53 can be changed. 17 and 18, the change of the background 3D model will be described. When a user wishes to change the background 3D model, the user operates the 3D model button B3 to change the display state of the preset list display unit 51 to the display state of the preset list of 3D models. In this display state, the default designation button B14, the grid designation button B15, and the N / A designation button B16 are displayed in the preset list display unit 51 as shown in the figure, and the user can switch the background 3D model by operating these buttons. Here, the default designation button B14 is a button that instructs switching to a background 3D model (e.g., a 3D model showing a stage, a ground, etc.) that is prepared in advance as an initial setting, and the grid designation button B15 is a button that instructs switching to a background 3D model (e.g., a grid line, a square, etc.) whose distance and angle are visible. In addition, the N / A designation button B16 is a button that instructs turning off the display of the background 3D model. Fig. 17 shows an example of a background 3D model of the camerawork window 53 when the Grid designation button B15 is operated, as an example, and Fig. 18 shows an example of a background 3D model of the camerawork window 53 when the Default designation button B14 is operated.

[0118] To start creating camerawork, the user operates the new creation button B5 as shown in FIG. In response to an operation of the New Creation button B5, the display processing unit 34a displays a new entry of camerawork in the camerawork list display unit 52. In this entry, an operation unit for specifying an In camera that is the start point of the viewpoint movement and an Out camera that is the end point of the viewpoint movement is displayed. In this example, in generating a free viewpoint image, a texture based on an image captured by a camera is applied to a 3D model of a subject, so it is desirable to create a viewpoint movement trajectory that passes through the camera position as much as possible. In particular, since the start and end points of the viewpoint movement in the free viewpoint image are the switching points between the images of the previous and next clips, it is desirable to make the start and end points of the viewpoint movement coincide with the camera position. For this reason, the start and end points of the viewpoint movement are specified as the camera positions of the In camera and Out camera, respectively. The start and end points of the viewpoint are not necessarily limited to the camera position, but may be any positions other than the camera position.

[0119] As shown in the figure, the operation unit for specifying the In camera and Out camera is, for example, an operation unit for allowing the user to specify the camera in a pull-down format. When the user operates to specify the pull-down, information indicating each camera listed in the preset list of specifiable cameras, i.e., the cameras specified by the user (in this example, camera number information) is displayed (see Figs. 22 and 24 described later).

[0120] In addition, in response to operation of the New Creation button B5, a new camerawork entry is displayed in the camerawork list display section 52 as described above, and a mark indicating the position of the target Tg set by the user (hereinafter referred to as the "target mark Mt") is displayed in the camerawork window 53. Here, in the creation of the camerawork, the position of the target Tg is set to a position considered appropriate for the target scene, such as a position near the goal in the target three-dimensional space (for example, a soccer field in the case of soccer) when an image of a shooting scene is to be generated as a free viewpoint image. Here, it is assumed that the position of the target Tg can be set in the free viewpoint image server 2 in advance by the user.

[0121] As mentioned above, the free viewpoint image can be generated so that the line of sight from the viewpoint faces the target Tg. Specifically, the free viewpoint image of this example can be generated so that the target Tg continues to be located at a predetermined position (e.g., the center position) in the image frame during at least a part of the movement period of the viewpoint. In generating a free viewpoint image, the above-mentioned process of positioning the target Tg at a predetermined position within the image frame is expressed as "following the target Tg." This "following the target Tg" is synonymous with the line of sight from the viewpoint continuing to face the target Tg while the viewpoint is moving.

[0122] Here, when the camera preset list is displayed in the camerawork window 53, the field of view when the viewpoint is set to the position of the specified camera and the observed image from the viewpoint (an image of the three-dimensional space observed from the viewpoint) are displayed in the camerawork window 53 and the preview window 55, respectively, in accordance with the camera specification operation from the camera preset list.

[0123] Specifically, FIGS. 20 and 21 show examples of the display contents of the creation operation screen Gg when a camera named camera1 and a camera named camera2 are respectively designated from the camera preset list. In this case, for a camera specified from the preset list of cameras, visual field range information Fv showing a visualization of the field of view from that camera is displayed in the camerawork window 53. As shown in the figure, in this example, information showing the field of view range in a graphic form is displayed as the field of view range information Fv. In this case, the camera position mark Mc for the specified camera is highlighted in the camera work window 53 more than the camera position marks Mc for the other cameras (the figure shows an example of a larger size), allowing the user to easily understand the position of the specified camera.

[0124] In the preview window 55, an image of the three-dimensional space observed from a specified camera is displayed. Here, in this example, it is assumed that the camerawork creation work is performed prior to the generation of the free viewpoint image. That is, it is assumed that the camerawork creation work is performed in a state where the captured image used for generating the free viewpoint image has not yet been acquired. For this reason, the image of the three-dimensional space observed from the viewpoint referred to here is not an image obtained by rendering as a two-dimensional image a 3D model (hereinafter, for the sake of explanation, referred to as a "real three-dimensional model") generated by performing subject detection, etc., from the captured images of each camera capturing the target real space, but an image obtained by rendering as a two-dimensional image a virtual 3D model (hereinafter, referred to as a "virtual three-dimensional model") that imitates the target real space. In the process of generating the observation image from the viewpoint in this case, since the captured image has not yet been acquired, the process of pasting the texture generated from the captured image onto the 3D model is not performed.

[0125] 22 to 25 are explanatory diagrams of a method for specifying the In camera and the Out camera. As shown in FIG. 22, when specifying an In camera, a camera specification operation is performed from the In camera pull-down list in the entry added to the camera work list display section 52.

[0126] 23 shows the state of the creation operation screen Gg when camera1 is specified as the In camera. When camera1 is specified as the In camera, "1" is displayed in the In camera item in the entry added to the camera work list display section 52 as shown in the figure. In addition, in the camerawork window 53, the camera position mark Mc for camera1 is highlighted, and the field of view range information Fv for camera1 is displayed. In contrast to Figure 20 above, the display manner of the camera position mark Mc and the field of view range information Fv in the camera work window 53 may be different when the camera is specified from the camera preset list and when the camera is specified as the In camera. In the preview window 55, an image of the three-dimensional space observed through the camera 1 is displayed.

[0127] When the Out camera is to be specified, as shown in FIG. 24, a camera specification operation is performed from the Out camera pull-down list in the entry added to the camera work list display section 52. FIG. 25 shows the appearance of the creation operation screen Gg when camera 9 is specified as the Out camera. In this case, “9” is displayed in the Out camera item in the entry added to the camera work list display section 52. Here, the movement trajectory of the viewpoint is determined by specifying the In camera and the Out camera. Therefore, in this case, information indicating the movement trajectory of the viewpoint connecting the positions of the In camera and the Out camera, which is shown as movement trajectory information Mm in the figure, is displayed in the camerawork window 53. Here, the movement trajectory information Mm is information that visualizes the movement trajectory of the viewpoint. Specifically, in this case, in the camerawork window 53, the camera position mark Mc for camera 9 designated as the Out camera is highlighted, and linear movement trajectory information Mm connecting the positions of the In camera and the Out camera is additionally displayed from the case of Figure 23.

[0128] Although an illustrated explanation is omitted, in the creation operation screen Gg, after at least the In camera and Out camera have been specified and the viewpoint movement trajectory has been formed as described above, it is possible to display a preview of the camera work. Specifically, the preview display can be started by operating the playback button B11 on the operation panel unit 54. In the camerawork window 53, an image in which the field of view range information Fv changes from moment to moment as the viewpoint moves is displayed as a camerawork preview display. In conjunction with this camerawork preview display, an observation image of the three-dimensional space (an observation image from the viewpoint) that changes from moment to moment as the viewpoint moves is displayed in the preview window 55.

[0129] In this example, such a preview display of camera work or a preview display of an observed image in three-dimensional space can be achieved not only by operating the play button B11, but also by dragging the seek bar B17 in the timeline operation section 54a. FIG. 26 shows a state in which the seek bar B17 has been positioned at a desired position on the timeline by a drag operation of the seek bar B17 in the timeline operation section 54a. While the seek bar B17 is being dragged, the position of the seek bar B17 on the timeline, that is, on the time axis from the start timing to the end timing of the free viewpoint image, changes from moment to moment. Therefore, while the dragging is being performed, the field of view range information Fv corresponding to the viewpoint position at the timing indicated by the seek bar B17 is displayed in sequence in the camerawork window 53, and the user visually recognizes the field of view range information Fv as an image that changes from moment to moment as the viewpoint moves. Similarly, in the preview window 55, an observation image of the three-dimensional space from the viewpoint that changes from moment to moment in response to the movement of the seek bar B17 is displayed.

[0130] Next, the viewpoint waypoints will be described. In the creation operation screen Gg, it is possible to specify the waypoints of the viewpoint and the timing at which the viewpoint passes through the waypoints. 27 to 30 are explanatory diagrams for designating the waypoints and the waypoint timings of the viewpoints. In this example, the waypoint and the timing at which the viewpoint passes through the waypoint can be specified by dragging and dropping a camera to be specified as a waypoint onto the timeline in the timeline operation section 54a.

[0131] 27 to 29 are explanatory diagrams of an example of operation when camera6 is specified as a waypoint. 27, a camera to be specified as a waypoint is selected from the preset list of cameras in the preset list display section 51. In this example, the operation for this selection is performed by pressing the left button of the mouse. The camera selected in this way is dragged across the screen as shown in Fig. 28, and dropped at a desired position on the timeline of the timeline operation unit 54a as shown in Fig. 29 (in this example, the left click is released). This completes the specification of the camera to be used as a waypoint and the timing at which the viewpoint will pass through the waypoint. Upon completion of this designation, a waypoint mark Mv is displayed on the timeline of the timeline operation unit 54a as shown in Fig. 29. This waypoint mark Mv is displayed at a position on the timeline designated by the above-mentioned drop operation. That is, it is displayed at a position indicating the designated timing within the period from the start timing to the end timing of the free viewpoint image (within the movement period of the viewpoint). In this example, the waypoint mark Mv is initially displayed as a square mark as shown in the figure.

[0132] Furthermore, when the designation of the waypoints and the timing of the waypoints is completed, the camera position mark Mc for the camera (here, camera 6) designated as the waypoint is highlighted in the camerawork window 53, and the field of view range information Fv indicating the field of view range from the camera is displayed. Moreover, the preview window 55 displays an image of the three-dimensional space observed from the viewpoint of the camera position designated as the waypoint.

[0133] By selecting a camera from the preset list as described above and performing a drag-and-drop operation on the selected camera, it is possible to specify a waypoint of the viewpoint and the timing at which the viewpoint passes through the waypoint. FIG. 30 shows an example of the creation operation screen Gg when waypoints and time periods for two more cameras are specified in the same manner as above. Although the example in which the camera position is specified as the waypoint of the viewpoint has been given here, any position other than the camera position can also be specified as the waypoint.

[0134] Moreover, in the creation operation screen Gg, it is possible to specify the type of shape of the movement trajectory of the viewpoint. Specific operation procedures and screen transitions according to the operations will be described with reference to Figs. 31 to 33. First, as shown in Fig. 31, a target range for which a type of movement trajectory is to be specified is specified on the timeline in the timeline operation unit 54a. The figure shows an example in which a range from a first waypoint to a third waypoint is specified when three waypoints of the viewpoint are set as in Fig. 30.

[0135] To specify the type of shape of the movement trajectory, an operation button provided on the trajectory shape adjustment operation unit 57 is operated. For example, as shown in FIG.

[0136] In response to an operation of this curved line interpolation button B18, the camerawork generation processing unit 34 performs curved line interpolation of the movement trajectory for a specified partial range of the viewpoint movement trajectory specified in Fig. 31. Then, the display processing unit 34a performs processing to display the movement trajectory information Mm generated by this curved line interpolation in the camerawork window 53, as exemplified in Fig. 33.

[0137] By making the movement trajectory of the viewpoint curved in this way, it is possible to prevent a large change in the distance from the target subject to the viewpoint even when the viewpoint moves, i.e., to prevent a large change in the size of the target subject in the free viewpoint image.

[0138] In this example, when the curved line interpolation is performed as described above, a process is performed to change the shape of the waypoint mark Mv displayed on the timeline in the timeline operation unit 54a to a shape corresponding to the curved line interpolation. Specifically, in this example, as illustrated in the figure, the shape of the waypoint mark Mv is changed from a square mark to a round mark. This makes it possible to notify the user on the timeline that curved line interpolation is being performed on the movement trajectory of the viewpoint connecting the waypoints. In this example, an operation button for instructing the movement trajectory shape to be linear is provided in the trajectory shape adjustment operation unit 57, and when the operation button is operated, the shape of the corresponding waypoint mark Mv is changed to a square mark.

[0139] Here, the shape of the movement trajectory can be set to a shape other than a curve or a straight line. For example, as shown in the movement trajectory information Mm in Fig. 33, a shape in which curves and straight lines are mixed can be used. In addition, the curvature of the movement trajectory based on the curve is not limited to be constant, and it is also possible to set the curvature to be different in some sections. Furthermore, when providing variation in the movement trajectory shape in this manner, the waypoint mark Mv may be displayed in a different display form corresponding to each variation, rather than being limited to the two display forms of straight lines and curved lines as exemplified above.

[0140] Moreover, in the creation operation screen Gg of this example, it is possible to specify the movement speed of the viewpoint. In specifying the movement speed, first, as shown in FIG. 31, a target range is specified on the timeline in the timeline operation unit 54a. Here, as shown in FIG. 34, it is assumed that a range spanning only the second waypoint is specified. Then, an operation button provided in the speed adjustment operation unit 56 is operated. For example, as shown in FIG. 34, the speed adjustment button B19 provided in the speed adjustment operation unit 56 is operated.

[0141] In response to the operation of this speed adjustment button B19, the camerawork generation processing unit 34 adjusts the speed of the viewpoint in accordance with the operated button for a portion of the specified viewpoint movement trajectory. In this example, in response to such speed adjustment, the display processing unit 34a performs a process of changing the shape of the corresponding waypoint mark Mv on the timeline to a shape corresponding to the aspect of the executed speed adjustment, as illustrated in Fig. 35. Note that the display aspect illustrated is merely an example. By performing such a shape change, it is possible to notify the user on the timeline that the speed adjustment has been performed in the relevant range of the viewpoint movement trajectory.

[0142] Next, the adjustment of the target position will be described. First, the significance of the target Tg will be explained with reference to FIGS. As described above, the target Tg is used to determine the line of sight from the viewpoint in the free viewpoint image. Fig. 36 shows the field of view Rf (Rf1, Rf3, Rf6, Rf9 for each camera position) and the line of sight Dg (Dg1, Dg3, Dg6, Dg9 for each camera position) from each camera position (camera1, 3, 6, 9 in this case) on the movement trajectory of the viewpoint.

[0143] As described above, in the generation of the free viewpoint image in this example, it is possible to specify the period during which the viewpoint faces the target Tg within the movement period. In other words, it is possible to specify the period during which the target Tg is kept positioned at a predetermined position within the image frame as the period during which the target Tg is tracked. In this example, the target Tg is tracked by keeping the target Tg at the center position within the image frame, as shown in FIG. Figure 36 illustrates an example of the gaze direction Dg and field of view Rf when tracking a target Tg during the period when the viewpoint moves from camera 1 to camera 9. As shown in the figure, in this example, each gaze direction Dg and field of view Rf is set to capture the target Tg at the center position within the image frame.

[0144] In this example, the position of the target Tg can be adjusted on the creation operation screen Gg. The adjustment operation of the target Tg can be, for example, an operation of adjusting the position of the target mark Mt displayed in the camerawork window 53. When the position of the target Tg is changed by adjustment, the camerawork generation processing unit 34 sets the line of sight direction Dg and the field of view range Rf at each viewpoint position so that the changed target Tg is kept positioned at a predetermined position within the image frame.

[0145] Here, in the creation operation screen Gg of this example, it is possible to specify that the position of the target Tg is moved with the passage of time during the movement of the viewpoint. A specific operation procedure for specifying the movement of the target Tg will be described with reference to FIGS. First, as shown in Fig. 38, a designation operation of a new point Ptn of the target Tg is performed. Specifically, an operation is performed to set a target position designation mark Mtn for designating the new point Ptn at a desired position in the camerawork window 53. Although not shown in the figure, the target position designation mark Mtn is displayed superimposed on the target mark Mt in the initial state, and the user drags and drops this target position designation mark Mtn from the position of the target mark Mt to the desired position. By such an operation, a new point Ptn of the target Tg is specified.

[0146] Next, the user operates the target button B2 provided on the preset list display section 51 to bring the preset list display section 51 into a state in which a list of targets Tg is displayed. In this display state, as shown in Fig. 39, an add button B20 for the target Tg is displayed on the preset list display section 51, and the user can operate this add button B20 to give an instruction to add a new target Tg to the position specified by the target position specification mark Mtn. In response to an operation of the Add button B20, as shown in the figure, identification information ("Target 0" in the figure) and position information (position information indicating the new point Ptn) for the added target Tg are displayed in the preset list display section 51. Also, in the camerawork window 53, an added target mark Mtt is displayed as a mark representing the added target Tg at the position of the target position designation mark Mtn.

[0147] Next, the user performs an operation of adding a new target onto the timeline as shown by the transition from Fig. 40 to Fig. 42. Specifically, the user performs an operation of dragging and dropping the target (here, "Target 0") newly displayed in the preset list display section 51 to a desired position on the timeline of the timeline operation section 54a. At the position on the timeline specified by the drop operation, a target arrival timing mark Mem as shown in Fig. 42 is displayed. This target arrival timing mark Mem is a mark indicating the target arrival timing of the target Tg to the new point Ptn when the position of the viewpoint target Tg is moved from the position indicated by the target mark Mt (i.e., the initial position of the target Tg) to the position indicated by the added target mark Mtt (i.e., the new point Ptn). In other words, the operation of adding a new target on the timeline as described above is an operation of specifying the target timing for the position of the target Tg to reach the new point Ptn with respect to the movement of the target Tg.

[0148] The user performs an operation of adding a new target on the timeline as described above, and then performs an operation of specifying a period for facing the target Tg as shown in FIGS. When designating the period for facing the target Tg, first, as shown in FIG. 43, the LookAt button B21 provided on the timeline operation unit 54a is operated. Then, a period designation bar B22 for designating the period is displayed on the timeline as shown in the figure. When the LookAt button B21 is operated, this period designation bar B22 is displayed in a manner that designates the period from the start of the movement of the viewpoint to the time indicated by the arrival target timing mark Mem as shown in the figure. When the user wants to change the period for facing the target Tg, the user performs an operation to extend or shorten the period designation bar B22. Here, as shown in FIG. 44 and FIG. 45, the period designation bar B22 is extended, and the period for facing the target Tg is designated as the period until the end of the movement of the viewpoint (the period from the start to the end of the movement of the viewpoint).

[0149] Figures 46 to 49 are diagrams for explaining a preview playback image of the camerawork and a preview playback image of the observed image from the viewpoint when various designation operations are performed regarding the movement of the target Tg described with reference to Figures 38 to 45. Specifically, they illustrate transitions of images displayed in the camerawork window 53 and the preview window 55 in response to the operation of the playback button B11 after various designation operations are performed regarding the movement of the target Tg described with reference to Figures 38 to 45.

[0150] In this case, the target Tg is moved so as to reach the new point Ptn at the timing indicated by the arrival target timing mark Mem on the timeline within the period from the start to the end of the viewpoint movement. Therefore, from the start of the viewpoint movement to the timing indicated by the arrival target timing mark Mem, the target mark Mt gradually approaches the target position designation mark Mtn (additional target mark Mtt in the camerawork window 53) as shown in Figures 46 and 47. Here, in Figures 46 and 47, the target initial position mark Mst is shown in each of the camerawork window 53 and the preview window 55, and this target initial position mark Mst is a mark indicating the position of the target Tg at the start of the viewpoint movement.

[0151] In this example, the period during which the eye faces the target Tg is specified as the entire period from the start to the end of the movement of the viewpoint, i.e., the period during which the eye faces the target Tg is specified as a period that exceeds the period until the timing indicated by the attainment target timing mark Mem. In this way, when the period for facing the target Tg is specified as a period exceeding the period until the target timing mark Mem, in this example, during the period exceeding the period until the target timing mark Mem, the movement of the target Tg is performed by gradually returning the position of the target Tg from the new point Ptn to the movement start position. Therefore, during the period facing the target Tg that exceeds the period until the target timing mark Mem is reached, the target mark Mt will gradually approach the target initial position mark Mst, which indicates the start position of the movement, over time, as shown in Figures 48 and 49.

[0152] By allowing the target Tg to move as described above, it is possible to improve the degree of freedom in creating a free viewpoint image compared to a case in which the position of the target Tg is fixed.

[0153] In the above, when a designation is made to move the position of the target Tg, a case has been exemplified in which a period longer than the period up to the target timing mark Mem is designated as the period for facing the target Tg, but as shown in Fig. 50, the period from the viewpoint movement start point to the target timing mark Mem can also be designated as the period for facing the target Tg. In this case, a free viewpoint image that does not follow the designated target Tg is generated during the viewpoint movement period after the period up to the target timing mark Mem.

[0154] Further, in the above, an example of adding one target Tg has been given as an example of adding a new target Tg using the target position designation mark Mtn, but it is also possible to add a plurality of targets Tg. In this case, the period for facing the target Tg can be specified individually for each added target Tg, for example, as illustrated in Fig. 51. Fig. 51 shows a state in which two targets Tg (hereinafter referred to as target Tg-1 and target Tg-2) are added as new targets Tg by an operation using the target position specification mark Mtn (see Figs. 38 and 39), and the targets Tg-1 and Tg-2 are added to different positions on the timeline, respectively, so that the reaching target timing marks Mem-1 and Mem-2 are displayed individually on the timeline. In this case, for the target Tg-1, as shown by the period specification bar B22-1 in the figure, the period from the start of the movement of the viewpoint to the reaching target timing mark Mem-1 is specified as the period for facing the target Tg. Furthermore, with regard to target Tg-2, as indicated by the period designation bar B22-2 in the figure, the period for facing target Tg is specified as the period from a predetermined time after the time indicated by the target timing mark Mem-1 to the time indicated by the target timing mark Mem-2.

[0155] In this case, during the period indicated by the period designation bar B22-1 within the viewpoint movement period, a free viewpoint image is generated in which the position of the target Tg gradually moves from the initial position (the position of the target Tg at the start of the viewpoint movement) to the position of the target Tg-1, and during the period indicated by the period designation bar B22-2, a free viewpoint image is generated in which, for example, the position of the target Tg gradually moves from the initial position to the position of the target Tg-2.

[0156] As can be understood from the explanation of FIG. 51, the creation operation screen Gg is capable of accepting an operation for specifying the positions of a plurality of targets Tg. This makes it possible to generate a free viewpoint image that tracks target A during a certain period within a viewpoint movement period and tracks target B during other periods, thereby increasing the freedom to set the target to be tracked. Therefore, the degree of freedom in creating a free viewpoint image can be improved.

[0157] In the above, an example of specifying the position of the target Tg is given in which the position of the target Tg is specified as a destination point when the position of the target Tg is moved during a period in which the viewpoint is moving. However, the position of the target Tg can also be specified as a position of a target Tg that does not move during a period in which the viewpoint is moving.

[0158] The process of generating and displaying a trajectory in response to an operational input on the creation operation screen Gg will be described with reference to the flowcharts of FIGS. 52 and 53 are executed by the CPU 71 of the free viewpoint image server 2. This processing is processing for realizing a part of the functions of the camerawork generation processing unit 34 described above.

[0159] FIG. 52 shows the process of generating and displaying a viewpoint movement trajectory in accordance with the designation of the In camera and Out camera. First, in step S101, the CPU 71 waits for an operation to specify an In camera. In this example, this specification operation is performed as an operation to specify a camera number listed in a pull-down list of the In camera in the entry of the camerawork displayed in the camerawork list display unit 52 as exemplified in FIG.

[0160] When an operation to specify the In camera has been performed, the CPU 71 performs various display processes related to the In camera as described in Fig. 23, as the In camera display process of step S102. For example, the CPU 71 performs processes such as highlighting the camera position mark Mc for the camera specified as the In camera in the camerawork window 53 and displaying the field of view range information Fv.

[0161] In step S103 following step S102, the CPU 71 waits for an operation to specify the Out camera (see the description of FIG. 24), and if an operation to specify the Out camera has been performed, the process proceeds to step S104.

[0162] In step S104, the CPU 71 performs processing for generating a viewpoint movement trajectory connecting the In camera and the Out camera. In the next step S105, the CPU 71 executes a display process for the Out camera and the viewpoint movement trajectory, that is, various display processes related to the Out camera and a display process for the viewpoint movement trajectory information Mm as described with reference to FIG. The CPU 71 ends the series of processes shown in FIG. 52 in response to the execution of the display process in step S105.

[0163] FIG. 53 shows the process of generating and displaying a viewpoint movement trajectory in response to a designated waypoint. In step S110, the CPU 71 waits for a waypoint designation operation. In this example, this designation operation is a series of operations including operations on the timeline as described with reference to Figs.

[0164] When a waypoint is specified, the CPU 71 generates a viewpoint movement trajectory that passes through the specified point in step S111. That is, a viewpoint movement trajectory that connects the In camera, the specified point, and the Out camera is generated. Then, in the next step S112, the CPU 71 performs a process of displaying the waypoints and the viewpoint movement trajectory, i.e., for the specified waypoint, the CPU 71 performs a process for highlighting the camera position mark in the camerawork window 53 as described in Fig. 29, displaying the field of view range information Fv, and displaying the waypoint mark Mv on the timeline. The CPU 71 ends the series of processes shown in FIG. 53 in response to the execution of the display process in step S112.

[0165] <8. GUI for creating free viewpoint images> Next, details of the camera work designation screen Gs shown in FIG. 7, an example of a procedure for designating camera work used to generate a free viewpoint image, and various functions related to camera work designation will be described with reference to FIGS. 54 is a diagram illustrating an example of the initial screen of the camerawork designation screen Gs. Here, the processing related to the display of various information on the camerawork designation screen Gs described below is performed by the above-mentioned display processing unit 32a (see FIG. 5).

[0166] As described above, the camerawork designation screen Gs is provided with a scene window 41, a scene list display section 42, a camerawork window 43, a camerawork list display section 44, a parameter display section 45, and a transmission window 46. Furthermore, on the camerawork designation screen Gs, a camera designation operation section 47, a still image import button B31, and a video import button B32 are provided for the scene window 41, and a play button B33, a pause button B34, and a stop button B35 are provided at the bottom of the screen. Furthermore, on the camerawork designation screen Gs, an X-axis viewpoint button B36, a Y-axis viewpoint button B37, a Z-axis viewpoint button B38, a Ca viewpoint button B39, a Pe viewpoint button B40, a display path restriction button B41, and a restriction release button B42 are provided for the camerawork window 43, and a filtering operation section 48 is provided for the camerawork list display section 44. The filtering operation section 48 is provided with a pull-down button B43 and a reset button B44.

[0167] In the camerawork designation screen Gs, when generating a free viewpoint image, first, the user performs an operation to import images of a target section for generating a free viewpoint image as the above-mentioned image data V1 to V16, in other words, images of a scene to be generated as a target for generating a free viewpoint image. When performing this import, the user operates either a still image import button B31 or a video import button B32 in the figure. The still image import button B31 is a button for issuing an instruction to import image data V1 to V16 of still images for generating the above-mentioned still image FV clip as a free viewpoint image, and the video import button B32 is a button for issuing an instruction to import image data V1 to V16 of video for generating the above-mentioned video FV clip as a free viewpoint image.

[0168] In response to the operation of either the still image import button B31 or the video import button B32, a pop-up window W1 as shown in Fig. 55 is displayed on the camerawork designation screen Gs, and the user can operate the "GET IN / OUT TC" button provided in the pop-up window W1 to display information indicating the image data V1 to V16 to be imported in the pop-up window W1 as shown in Fig. 56. When importing the displayed data, the user operates the OK button provided in the pop-up window W1.

[0169] When the OK button is operated, as shown in Fig. 57, on the camerawork designation screen Gs, information of the imported scene is added to the scene list display section 42, and an image of the imported scene is displayed in the scene window 41. The information of the scene added to the list by import includes a thumbnail image of the scene, time information indicated by the time code (in this example, both the start time and end time of the scene are displayed), information indicating the period of the scene, etc. Note that the illustrated example shows a case where image data V1 to V16 as still images have been imported, and the value indicating the period of the scene is set to "0".

[0170] The camera designation operation section 47 is provided with a camera selection button for selecting the images of the imported scene taken by each camera, that is, the images taken by each camera from the image data V1 to V16, to be displayed.

[0171] On the camerawork designation screen Gs, a camerawork window 43 allows a preview of the camerawork. In the camerawork window 43, it is possible to switch the observation viewpoint of the camerawork in three-dimensional space using the X-axis viewpoint button B36, the Y-axis viewpoint button B37, the Z-axis viewpoint button B38, the Ca viewpoint button B39, and the Pe viewpoint button B40.

[0172] The X-axis viewpoint button B36, the Y-axis viewpoint button B37, and the Z-axis viewpoint button B38 are buttons for switching the observation viewpoints in the three-dimensional space to the viewpoint on the X-axis, the viewpoint on the Y-axis, and the viewpoint on the Z-axis, respectively. Here, the X-axis, the Y-axis, and the Z-axis are the three axes that define the three-dimensional space, and in this example, the X-axis is the horizontal direction, the Y-axis is the vertical direction, and the Z-axis is the axis that defines a direction perpendicular to both the X-axis and the Y-axis. The Pe viewpoint button B40 is a button for switching the observation viewpoint in the three-dimensional space to an arbitrary viewpoint designated by the user. The Ca viewpoint button B39 is a button for switching the observation viewpoint in three-dimensional space to a viewpoint in camerawork (a point on the viewpoint movement trajectory).

[0173] Figures 58, 59, 60, 61, and 62 respectively show examples of images displayed in the camerawork window 43 when the X-axis viewpoint button B36, the Y-axis viewpoint button B37, the Z-axis viewpoint button B38, the Pe viewpoint button B40, and the Ca viewpoint button B39 are operated.

[0174] Here, the camerawork window 43 displays information indicating the camerawork displayed in the camerawork list display section 44. The camerawork displayed in the camerawork list display section 44 is the camerawork created through the above-mentioned creation operation screen Gg, and is a candidate for the camerawork to be used in generating a free viewpoint image. In other words, it is the camerawork that can be designated as the camerawork to be used in generating a free viewpoint image.

[0175] In the camerawork window 43, the above-mentioned movement trajectory information Mm, camera position mark Mc, and field of view range information Fv (also in this case, the field of view range is assumed to be information showing by a graphic) are displayed as information showing the camerawork. In particular, as shown in Fig. 61, the camera position mark Mc showing the position of the In camera and the Out camera is displayed larger than the other camera position marks Mc, and the positions of the In camera and the Out camera are shown. Here, the camera position marks Mc of the In camera and the Out camera, that is, the information showing the positions of the camera which is the start point of the movement of the viewpoint and the camera which is the end point of the movement (start point arrangement position information, end point arrangement position information) may be displayed in a manner different from the camera position marks Mc of the cameras other than the In camera and the Out camera, and is not limited to being displayed in a different size. Furthermore, in the camerawork window 43, a target mark Mt indicating the position of the above-mentioned target Tg is displayed as information indicating the camerawork. In the case of Figure 62, i.e., when the Ca viewpoint button B39 is operated, the image is observed from the viewpoint movement trajectory, so the movement trajectory information Mm, camera position mark Mc, and field of view range information Fv are not displayed, and the target mark Mt is displayed.

[0176] In the figures, two target marks Mt are displayed as an example (see Figures 60 and 61 in particular), but this means that cameraworks in which different targets Tg are set are mixed as multiple cameraworks as candidates displayed in the camerawork list display unit 44. That is, the candidate cameraworks in this case include a mixture of cameraworks in which a target Tg is set, the position of which is indicated by the target mark Mt on the left side in Figure 61, and a mixture of cameraworks in which a target Tg is set, the position of which is indicated by the target mark Mt on the right side in the same figure.

[0177] Dynamic preview playback of camerawork can be performed in the camerawork window 43. Such dynamic preview playback can be instructed by operating the playback button B33. When the playback button B33 is operated, in the cases of the X-axis viewpoint, Y-axis viewpoint, Z-axis viewpoint, and arbitrary viewpoint shown in Fig. 58 to Fig. 61, an image is displayed in which the position and shape of the field of view range information Fv changes from moment to moment as the viewpoint moves on the movement trajectory. Also, in the case of Fig. 62, images of the target three-dimensional space observed from each point on the viewpoint movement trajectory from the viewpoint movement start point to the end point are successively switched and displayed. That is, an observation image of the three-dimensional space that changes from moment to moment as the viewpoint moves is displayed. The pause button B12 and the stop button B13 are buttons for instructing to pause and stop the dynamic preview playback as described above, respectively.

[0178] Here, when previewing the observed image from the viewpoint as in the case of Fig. 62, the free viewpoint image as the FV clip generated by using the selected camera work can be displayed as the display image. That is, the above-mentioned real 3D model is generated based on the imported image data V1 to V16, and the 2D image rendered by applying texture to the real 3D model is displayed as the preview image.

[0179] However, generating a free viewpoint image as an FV clip requires a considerable processing load and processing time, which may result in the user having to wait a long time until preview playback begins, which may hinder the rapid creation of free viewpoint images. Therefore, in this example, when previewing an observation image from a viewpoint as in the case of Figure 62, the displayed image is not a real three-dimensional model generated based on image data V1 to V16 (i.e., captured images in real space), but an image rendered from the virtual three-dimensional model described above (virtual 3D model in real space). This makes it possible to shorten the processing time required for preview display of an image observed from a viewpoint, and to rapidly perform the task of creating a free viewpoint image.

[0180] As mentioned above, the camerawork window 43 can display the selected camerawork or a plurality of selectable cameraworks. Figures 58 to 62 show examples of cases where camerawork information for only the selected camerawork is displayed, but it is also possible for the camerawork window 43 to display camerawork information for all of the multiple cameraworks displayed in the camerawork list display section 44. In this way, the number of cameraworks displayed in the camerawork window 43 can be switched by the display path limit button B41 and the limit release button B42. The display path limit button B41 is a button for instructing display of only the selected camerawork, and the limit release button B42 is a button for releasing the state limited to displaying only the selected camerawork, and functions as a button for instructing display of camerawork information for all of the multiple cameraworks displayed in the camerawork list display area 44.

[0181] Next, the camera work list display section 44 will be described. The camerawork list display unit 44 displays cameraworks as candidates that can be used to generate a free viewpoint image (see, for example, FIG. 63, etc.). The camerawork information displayed on the camerawork list display unit 44 includes the camerawork ID, the identification information of the In camera and the Out camera, and tag information. In this example, a thumbnail image of the movement trajectory information Mm is displayed for each camerawork in the camerawork list display unit 44. By displaying such thumbnail images, the user can confirm what kind of viewpoint movement trajectory each camerawork has even on the camerawork list.

[0182] Here, the above tag information is information that can be added to each created camerawork when creating the camerawork through the above-mentioned creation operation screen Gg, and in this example, is text information. Setting of tag information for a camerawork can be performed, for example, by inputting information in the "Tag" field (for example, see FIG. 19, etc.) provided in the entry of the camerawork in the camerawork list display section 52 of the creation operation screen Gg. In the following, this tag information will be referred to as "tag information I1."

[0183] The camerawork list display section 44 is provided with a filtering operation section 48 for filtering the camerawork to be displayed in a list, that is, the camerawork to be displayed in the camerawork list display section 44 . The camerawork filtering function using the filtering operation unit 48 will be described with reference to FIGS. 63 to 65. FIG.

[0184] First, when performing filtering, as shown in FIG. 63, the user operates the pull-down button B43 in the filtering operation unit 48. Then, as shown in the figure, a pull-down list 48a is displayed. A list of tag information I1 is displayed in this pull-down list 48a. The tag information I1 displayed in the pull-down list 48a is the tag information I1 set for each camera work as a candidate. That is, as exemplified in the figure, if tag information I1 such as "CW, Cam9" or "CW, Right" is set as a camera work as a candidate, the tag information I1 such as "CW, Cam9" or "CW, Right" is displayed in the pull-down list 48a.

[0185] The user can instruct the camera work list display section 44 to display only the camera work for which the tag information I1 is set by performing an operation (e.g., a click operation) to specify the tag information I1 displayed in the pull-down list 48a. As can be understood from this point, the display portion of each tag information I1 in the pull-down list 48a corresponds to filtering condition information that indicates the filtering conditions for filtering and displaying the camera work information.

[0186] 64 shows an example of the camerawork specification screen Gs when "CW, Right" is specified as the tag information I1. In this case, only camerawork for which "CW, Right" is set as the tag information I1 is displayed in the camerawork list display section 44. In this case, since there is only one camerawork set to "CW, Right", information on this camerawork set to "CW, Right" is displayed in the camerawork window 43. The camerawork set to "CW, Right" is the camerawork set for the target Tg whose position is indicated by the right-hand target mark Mt, of the targets Tg whose positions are indicated by the two target marks Mt shown in the camerawork window 43 of Fig. 61 above. Therefore, in this case, only one target mark Mt is displayed as camerawork information in the camerawork window 43. In addition, if there are multiple camera works to which the specified tag information I1 is set, it is possible to display the information of the camera work displayed at a specified position, such as the first position on the list, in the camera work window 43.

[0187] By filtering camera work based on tag information I1 as described above, filtering based on any criteria can be realized depending on the information content set as tag information I1. For example, if team information (e.g., team A, team B, etc.) is set as tag information I1, filtering based on whether the camera work targets a shooting scene of team A or a shooting scene of team B can be realized. Alternatively, filtering based on the moving direction of the viewpoint can be realized by setting information indicating the moving direction of the viewpoint (e.g., clockwise, counterclockwise, etc.) as tag information I1. In addition, by setting the camera closest to the field of view of interest, such as the field of view closest to the subject that is the target Tg, as tag information I1, it is possible to realize filtering of camerawork based on the field of view of interest.

[0188] In the filtering operation unit 48, the reset button B44 is a button for instructing to reset filtering. When the reset button B44 is operated, as exemplified by the screen transition from Fig. 64 to Fig. 65, the filtered display state of the camerawork in the camerawork list display unit 44 is released, and the cameraworks as candidates to be used for generating a free viewpoint image are displayed in a list.

[0189] Here, in the above, an example of filtering of camerawork based on tag information I1 has been given, but filtering of camerawork can also be performed based on information of the In camera and Out camera included in the camerawork information.

[0190] In addition, in the above example, information indicating filtering conditions such as tag information I1 is displayed in the pull-down list 48a, but information indicating filtering conditions (filtering condition information) can also be displayed as buttons as shown in FIG. In this case, the information to be displayed as the button can be determined based on the history information of the camerawork used in the past to generate the free viewpoint image. For example, the tag information I1 of the camerawork that has been used most frequently in the past and is ranked at the top can be displayed as the button. FIG. 66 shows an example of a button arrangement that corresponds to the case where the most frequently used camera work in a specific position is camera work with tag information I1 such as "goal mouth," "Left," or "Right."

[0191] Furthermore, the button displaying the filtering condition information may be made customizable by the user. FIG. 67 shows an example of button display in this case. In this case, the user can set any information for the display information of each button. When the user sets the display information of the button, the image generation processing unit 32 manages the set information as information indicating the filtering conditions of the camerawork. For example, the information of "Team A", "Left", and "Right" illustrated in the figure is managed as information indicating the filtering conditions of the camerawork. When a button is operated, the image generation processing unit 32 (specifically, the display processing unit 32a) performs processing to display on the camerawork list display unit 44 the camerawork for which tag information I1 matching the information managed in response to that button is set.

[0192] Furthermore, the information indicating the filtering condition may be accepted as keyword information input by the user. In that case, for example, a keyword input section 48b as exemplified in Fig. 66 or 67 is provided in the filtering operation section 48. In this case, the display processing section 32a performs processing to display, in response to a keyword input to the keyword input section 48b, cameraworks for which tag information I1 matching the input keyword information is set, in the camerawork list display section 44.

[0193] In the above example, the camerawork used to generate the free viewpoint image is specified by specifying the camerawork displayed in the camerawork list display unit 44, but the camerawork can also be specified by specifying the camerawork displayed in the camerawork window 43.

[0194] In this embodiment, the camerawork information displayed in the camerawork window 43 is displayed as information that visualizes the movement speed of the viewpoint. Figures 68 and 69 show examples of visualization information of the movement speed of the viewpoint. Note that Figures 68 and 69 show examples of displaying an observation image of camerawork information from the viewpoint on the Y axis described above in the camerawork window 43. In this example, information indicating a period during which the viewpoint movement speed slows down is displayed as information visualizing the viewpoint movement speed. Fig. 68 shows an example in which a camera position mark Mc indicating a camera positioned in a section in which the viewpoint movement speed slows down is displayed in a different display mode from the other camera position marks Mc. For example, it is possible to display the relevant camera position mark Mc in a different color or size from the other camera position marks Mc.

[0195] 69 shows an example in which the display mode of the relevant section in the movement trajectory information Mm is different from that of the other sections during the period when the movement speed of the viewpoint decreases. For example, as illustrated in the figure, it is possible to display the movement trajectory of the relevant section as a dotted line and the movement trajectory of the other sections as a solid line. Alternatively, it is possible to display the movement trajectory of the relevant section and the other sections with different colors, thickness, line shapes (for example, straight lines and wavy lines, etc.).

[0196] Although not illustrated, the moving speed of the viewpoint can be expressed by the density of dots when the moving trajectory is shown by a dotted line. For example, the faster the moving speed, the higher the density of dots can be displayed.

[0197] Although the images observed from the viewpoint on the Y axis are shown as examples in Figs. 68 and 69, the same display can be obtained from the viewpoint on the X axis, the viewpoint on the Z axis, or the Pe viewpoint (arbitrary viewpoint).

[0198] Furthermore, in this embodiment, with regard to the camerawork information displayed in the camerawork window 43, a process is performed to update the information on the position of the target Tg in the camerawork information in response to an operation to change the position of the target mark Mt. This processing is performed by the camerawork editing processing unit 32b shown in FIG.

[0199] Such target position editing processing by the camerawork edit processing unit 32b will be described with reference to FIGS. FIG. 70 shows an example of the camerawork information displayed in the camerawork window 43. It is assumed that an operation to change the position of the target mark Mt as shown in the figure has been performed in the camerawork window 43. The operation to change the position of the target mark Mt may be, for example, a drag-and-drop operation of the target mark Mt. Here, the position of the target Tg after such a change operation is represented as "position Pta", and the position of the target Tg before the change is represented as "Ptb".

[0200] In response to the above-mentioned operation of changing the position of the target mark Mt, the camerawork editing processing unit 32b performs a process of updating the position information of the target Tg from position Ptb to position Pta for the camerawork information displayed in the camerawork window 43.

[0201] By updating the position information of the target Tg in the camera work information in this way, in the generation process of a free viewpoint image using the camera work information, a free viewpoint image is generated so that the line of sight direction Dg from each position on the viewpoint movement trajectory is directed to the updated position of the target Tg. Figure 71 shows an image of the change in the field of view Rf when the line of sight direction Dg from each position on the viewpoint movement trajectory is directed to the updated position Pta.

[0202] As described above, by making it possible to edit camerawork information in response to operations on the camerawork designation screen Gs, when one wishes to edit camerawork information at the stage of specifying the camerawork to be used in generating a free viewpoint image, there is no need to launch software for generating camerawork information. Therefore, even if editing of camera work information is required, the work of creating a free viewpoint image can be quickly performed.

[0203] Furthermore, in this embodiment, the camerawork designation screen Gs performs the above-mentioned display process of information notifying the user of the cameras that require calibration. Specifically, the display processing unit 32a determines whether or not there is a camera in which a change has been detected among the cameras displaying the camera position mark Mc in the camerawork window 43, based on the result of the change detection of each camera by the utility server 8 described above in Fig. 14 (for example, the automatic change detection in step S33). If there is a camera in which a change has been detected, the display processing unit 32a performs a process of displaying information in the camerawork window 43 notifying the corresponding camera (i.e., the camera in which a change has been detected). It should be noted that the camera in which the variation is detected can be said to be a camera in which a change in the field of view is detected.

[0204] FIG. 72 shows an example of the display of notification information for a camera in which a change has been detected. Regarding the display of the notification information, as shown in the figure, the camera position mark Mc of the corresponding camera is displayed in a display mode different from other camera position marks Mc (in this case, it is also possible to make the color, size, shape, etc. different). Also, it is also possible to display information to call attention near the corresponding camera position mark Mc, such as an exclamation mark as exemplified in the figure.

[0205] In generating a free viewpoint image, in order to accurately generate three-dimensional information from images captured by multiple cameras, it is necessary for each camera to maintain a pre-assumed position and orientation, and if a change in position or orientation occurs in any of the cameras, calibration of the parameters used to generate the three-dimensional information is required. By notifying the user of the camera in which a change in the field of view has been detected as described above, it is possible to notify the user of the camera that requires calibration. Therefore, it is possible to generate a free viewpoint image based on accurate three-dimensional information, and it is possible to improve the image quality of the free viewpoint image.

[0206] The above-mentioned processing related to the filtering of the camerawork will be described with reference to the flowcharts of Fig. 73 and Fig. 74. Note that the processing shown in Fig. 73 and Fig. 74 is executed by the CPU 71 of the free viewpoint image server 2 as the processing of the display processing unit 32a.

[0207] FIG. 73 shows a process corresponding to the case where camera work filtering is performed based on tag information I1 displayed on the camera work designation screen Gs as in FIG. First, in step S201, the CPU 71 performs a process of acquiring tag information I1 in each piece of candidate camera work information. That is, each piece of candidate camera work information that can be used to generate a free viewpoint image is acquired. Here, the candidate camera work information is stored in a readable storage device inside or outside the free viewpoint image server 2. In the process of step S201, the candidate camera work information stored in this way is acquired.

[0208] In step S202 following step S201, the CPU 71 performs display processing of the tag information I1. That is, when displaying in the pull-down list 48a as shown in Fig. 63, the tag information I1 included in the camera work information acquired in step S201 is displayed in response to the operation of the pull-down button B43.

[0209] In step S203 following step S202, the CPU 71 waits for a designation operation of tag information I1, and if there is a designation operation of tag information I1, the process proceeds to step S204 and performs processing to filter and display camerawork to which the designated tag information I1 is attached. That is, processing to display camerawork information including the designated tag information I1 (to which the designated tag information I1 is set) among the candidate camerawork information in the camerawork list display unit 44. As illustrated in Fig. 64, in this example, the camerawork information to be displayed is, for example, camerawork identification information, In camera, Out camera information, tag information I1, etc. The CPU 71 ends the series of processes shown in FIG. 73 in response to execution of the process of step S204.

[0210] FIG. 74 shows the process of filtering camera work according to an input keyword. 74, the CPU 71 waits for a keyword input from the user in step S210, and when a keyword is input, selects camerawork including the input keyword in step S211. That is, from among the camerawork information candidates, camerawork information including the input keyword in tag information I1 is selected. Then, in the next step S212, the CPU 71 executes a process of displaying the selected camerawork. That is, the CPU 71 executes a process of displaying the selected camerawork information in the camerawork list display section 44. In response to executing the process of step S212, the CPU 71 ends the series of processes shown in FIG.

[0211] Fig. 75 is a flowchart of a process related to a notification of a camera that requires calibration, as exemplified in Fig. 72. Note that the process shown in Fig. 75 is also executed by the CPU 71 of the free viewpoint image server 2 as a process of the display processing unit 32a, similar to the processes in Figs. 73 and 74.

[0212] In step S301, the CPU 71 waits for a camera fluctuation notification. That is, the CPU 71 waits for a fluctuation notification to be transmitted when the utility server 8 detects a camera fluctuation by the above-mentioned automatic fluctuation detection (step S33 in FIG. 14). This fluctuation notification includes information for identifying the camera from which the fluctuation was detected.

[0213] When a camera change notification is received, the CPU 71 determines in step S302 whether or not the camera is being displayed. That is, it determines whether or not the camera notified by the change notification is a camera that is displaying the camera position mark Mc in the camerawork window 43. If the camera is not being displayed, the CPU 71 ends the series of processes shown in FIG.

[0214] On the other hand, if it is the camera being displayed, the CPU 71 proceeds to step S303 to execute a change notification process. That is, the CPU 71 executes a process of displaying information notifying a change in the corresponding camera position mark Mc being displayed in the camerawork window 43 in a display mode such as that exemplified in FIG. In response to executing the process of step S303, the CPU 71 ends the series of processes shown in FIG.

[0215] <9. Variations> It should be noted that the embodiment is not limited to the specific example described above, and various modified configurations may be adopted. For example, in the above example, a device that processes the display of the creation operation screen Gg and accepts operation inputs for generating camerawork, and a device that processes the display of the camerawork designation screen Gs and accepts operation inputs for generating free viewpoint images are treated as a common device that serves as the free viewpoint image server 2, but it is also possible to adopt a configuration in which these devices are separate devices.

[0216] In the above, an example was given of filtering the display of camerawork on the camerawork specification screen Gs according to an operated unit such as a button indicating a filtering condition, but the display of camerawork can also be filtered according to the specification of a target Tg, such as the specification of a target mark Mt displayed in the camerawork window 43. Specifically, only camerawork for which a specified target Tg is set is displayed in a filtered manner among candidate cameraworks.

[0217] Furthermore, on the creation operation screen Gg and the camera work specification screen Gs, if there is a range on the viewpoint movement trajectory where image quality cannot be guaranteed because, for example, the actual camera is too far from the subject (for example, a range where the resolution is below a predetermined value), information notifying of that range can also be displayed.

[0218] <10. Summary of the embodiment> As described above, the first information processing device of the embodiment is equipped with a display processing unit (34a) that performs display processing of a screen including a designation operation reception area (camerawork list display unit 44, operation panel unit 54, etc.) that accepts operation input to designate at least a portion of the camerawork information, which is information indicating at least the movement trajectory of the viewpoint in a free viewpoint image, and a camerawork display area (camerawork window 53) that visualizes and displays the movement trajectory of the viewpoint based on the camerawork information reflecting the content specified by the operation input, as a creation operation screen (Gg) for camerawork information. This allows the user to perform camerawork creation operations while visually checking the visualized movement trajectory of the viewpoint on the camerawork creation operation screen. Therefore, the efficiency of camerawork creation work can be improved.

[0219] In addition, in the first information processing device of the embodiment, the designation operation receiving area is capable of receiving an operation for designating a start point and an end point of a movement trajectory (see FIGS. 22 to 25 and 52). This makes it possible to set any starting point and ending point for the movement trajectory of the viewpoint, rather than setting a fixed one. Therefore, the degree of freedom in creating a free viewpoint image can be improved.

[0220] Furthermore, in the first information processing device of the embodiment, the designation operation receiving area is capable of receiving an operation for designating a waypoint of the viewpoint (see FIGS. 27 to 30 and 53). This makes it possible to set a trajectory of the viewpoint that passes through specified points, rather than a straight line connecting two points, the starting point and the end point. Therefore, the degree of freedom in creating a free viewpoint image can be improved.

[0221] Furthermore, in the first information processing device of the embodiment, the designating operation receiving area is capable of receiving a designating operation for the timing when the viewpoint passes through a waypoint (see FIGS. 27 to 30). This makes it possible to set not only the waypoints of the viewpoint but also the timing at which the viewpoint passes through the waypoints. Therefore, it is possible to improve the degree of freedom in setting the position through which the viewpoint will pass, as well as the degree of freedom in setting the timing at which the viewpoint will pass through the waypoint, thereby improving the degree of freedom in creating a free viewpoint image.

[0222] In addition, in the first information processing device of the embodiment, the designation operation receiving area is capable of receiving a designation operation for the type of shape of the movement trajectory (see FIGS. 32 and 33). This makes it possible for the shape and type of the movement trajectory of the viewpoint to be variable rather than fixed. Therefore, the degree of freedom in creating a free viewpoint image can be improved. For example, if the shape type of the movement trajectory is a curved shape, the distance from the target subject to the viewpoint does not change significantly even if the viewpoint moves. In other words, the size of the target subject in the free viewpoint image does not change significantly.

[0223] Furthermore, in the first information processing device of the embodiment, the designation operation receiving area is capable of receiving an operation for designating a movement speed of the viewpoint (see FIGS. 34 and 35). This makes it possible to make the movement speed of the viewpoint variable rather than fixed. Therefore, the degree of freedom in creating a free viewpoint image can be improved.

[0224] Furthermore, in the first information processing device of the embodiment, the designating operation receiving area is capable of receiving a designating operation of a section for changing the moving speed in the moving trajectory (see FIG. 34 and FIG. 35). This makes it possible to dynamically change the movement speed of the viewpoint along the movement trajectory. Therefore, the degree of freedom in creating a free viewpoint image can be improved.

[0225] In addition, in the first information processing device of the embodiment, the designated operation reception area is capable of receiving operation input for a timeline indicating the period from the start point of the movement of the viewpoint to the end point of the movement (see timeline operation unit 54a). By accepting input operations on the timeline, it becomes possible to simultaneously specify a waypoint and the timing of that waypoint by, for example, dragging and dropping a camera icon onto the timeline, or to specify a range by dragging on the timeline to specify a section where a specified effect should be applied, such as a section where curved interpolation of the movement trajectory should be performed. This makes it possible to facilitate the specification of various information related to camerawork. Therefore, the efficiency of camerawork creation work can be improved.

[0226] Furthermore, in the first information processing device of the embodiment, the display processing unit performs processing to display information (field of view range information Fv) that visualizes the field of view range from the viewpoint in the camerawork display area (see Figure 20, etc.). By visually indicating the field of view, it becomes easier for the user to understand the camera work. Therefore, it becomes possible for the user to easily understand how the camera work will change in response to operational input, and the efficiency of the camera work creation work can be improved.

[0227] Furthermore, in the first information processing apparatus of the embodiment, the display processing section performs processing for displaying information that graphically represents the field of view range from the viewpoint in the camerawork display area. By showing the field of view in graphical form, the user can easily understand the camera work. Therefore, it becomes possible for the user to easily understand how the camera work will change in response to operational input, and the efficiency of the camera work creation work can be improved.

[0228] In addition, in the first information processing apparatus of the embodiment, the display processing section performs processing for displaying an image of a three-dimensional space observed from a viewpoint on the creation operation screen (see preview window 55). This makes it possible to provide a preview of an image similar to a free viewpoint image generated based on camera work information to the user, making it easier for the user to understand the camera work. Therefore, the efficiency of camerawork creation work can be improved.

[0229] Furthermore, in the first information processing device of the embodiment, the designation operation receiving area is capable of receiving a designation operation for the position of a target that determines the line of sight from the viewpoint (see FIGS. 38 and 39, etc.). This makes it possible to generate an image that tracks a target as a free viewpoint image. An image that tracks a target means an image in which the target is kept positioned at a predetermined position (e.g., the center position) within the image frame. Therefore, the degree of freedom in creating a free viewpoint image can be improved.

[0230] Furthermore, in the first information processing device of the embodiment, the designating operation receiving area is capable of receiving a designating operation during a period in which the user faces the target (see FIGS. 41 to 52). The target facing period means a period during which the target is kept positioned at a predetermined position within the image frame of the free viewpoint image. By making it possible to specify the target facing period as described above, it becomes possible to generate an image that tracks the target position during a certain period within the viewpoint movement period and does not track the target position during other periods as the free viewpoint image, thereby improving the freedom of setting the period during which the target position is tracked. Therefore, the degree of freedom in creating a free viewpoint image can be improved.

[0231] Furthermore, in the first information processing device of the embodiment, the designation operation receiving area is capable of receiving a designation operation for the positions of a plurality of targets as a designation operation for the position of a target (see FIG. 51). This makes it possible to generate a free viewpoint image that tracks target A during a certain period within a viewpoint movement period and tracks target B during other periods, thereby increasing the freedom to set the target to be tracked. Therefore, the degree of freedom in creating a free viewpoint image can be improved.

[0232] Furthermore, a first information processing method of the embodiment is an information processing method in which an information processing device performs display processing of a screen as a creation operation screen for camerawork information, which is information indicating at least the movement trajectory of the viewpoint in a free viewpoint image, including a designation operation reception area that accepts operation input to designate at least a portion of the camerawork information, and a camerawork display area that visualizes and displays the movement trajectory of the viewpoint based on the camerawork information that reflects the specified content by the operation input. According to the first information processing method, it is possible to obtain the same functions and effects as those of the first information processing apparatus described above.

[0233] In addition, the second information processing device of the embodiment is equipped with a display processing unit (32a) that performs display processing of a screen showing camerawork information that corresponds to user input information among multiple camerawork information, as a camerawork designation screen (Gs) that accepts an operation to designate camerawork information, which is information that indicates at least the movement trajectory of the viewpoint in a free viewpoint image. By filtering and displaying camera work information according to information input by the user, it becomes possible to make it easier for the user to find the camera work information they desire, thereby shortening the time required to specify camera work information. Therefore, the task of creating a free viewpoint image can be carried out quickly.

[0234] Furthermore, in the second information processing device of the embodiment, the display processing unit performs processing on the camerawork designation screen to filter and display camerawork information corresponding to keywords as the input information (see Figures 66, 67, 74, etc.). This makes it possible to perform appropriate filtering of camerawork information that reflects the user's intentions. Therefore, it becomes possible for the user to more easily find the camera work information he or she desires, and the time required to specify camera work information can be further shortened.

[0235] Furthermore, in the second information processing device of the embodiment, an operated part indicating filtering conditions for camerawork information is arranged on the camerawork designation screen, and the display processing part performs processing to filter and display the camerawork information according to the filtering conditions indicated by the operated operated part in response to the operation of the operated part (see Figures 63, 64, 66, 67, and 73). This makes it possible to reduce the operation required for filtering and displaying camera work information to just selecting filtering condition information. Therefore, the operational burden on the user required for filtering and displaying camera work information can be reduced.

[0236] Furthermore, in the second information processing device of the embodiment, the display processing unit performs processing for displaying information that visualizes the movement trajectory of the viewpoint on the camerawork designation screen (see FIG. 61 and the like). By displaying information that visualizes the movement trajectory of the viewpoint, it becomes easier for users to imagine what kind of camera work is being performed. Therefore, when specifying camera work information to be used in creating a free viewpoint image, it becomes possible for the user to easily find the desired camera work information, and the time required to specify camera work information can be shortened.

[0237] Furthermore, in the second information processing device of the embodiment, the display processing unit performs processing to display camera placement position information indicating the placement positions of multiple cameras that capture images for generating free viewpoint images on the camerawork designation screen (see Figure 61, etc.). By displaying information indicating the placement position of each camera, the user can easily imagine what kind of image should be generated as a free viewpoint image. Therefore, the task of creating a free viewpoint image can be carried out quickly.

[0238] Furthermore, in the second information processing device of the embodiment, the display processing unit performs processing to display start point placement position information and end point placement position information indicating the respective positions of the camera that is the start point of the viewpoint movement and the camera that is the end point of the movement among the multiple cameras on the camera work designation screen (see Figure 61, etc.). This allows the user to understand from which camera position the movement of the viewpoint begins and ends. Therefore, when specifying the camerawork information to be used for creating a free viewpoint image, it is possible to make it easier for the user to find the desired camerawork information. In particular, when generating an image by connecting a front clip and a rear clip to a free viewpoint image as described above, it is desirable to match the camera that is the start point of the movement of the viewpoint with the imaging camera of the front clip and the camera that is the end point of the movement of the viewpoint with the imaging camera of the rear clip so that the connection between the clips becomes natural, but by displaying the positions of the camera that is the start point of the movement and the camera that is the end point of the movement as described above, it is possible to easily specify appropriate camerawork according to the imaging camera of the front clip and the imaging camera of the rear clip.

[0239] In addition, in the second information processing device of the embodiment, the display processing unit performs processing to display the start point placement position information and the end point placement position information, and the placement position information of the cameras other than the camera that is the start point of movement and the camera that is the end point of movement among the multiple cameras, in different manners. This allows the user to intuitively grasp at which camera position the movement of the viewpoint starts and ends. Therefore, when specifying camera work information to be used in creating a free viewpoint image, the user can more easily find the desired camera work information.

[0240] Furthermore, in the second information processing device of the embodiment, the display processing unit performs processing for displaying information that visualizes the movement speed of the viewpoint on the camerawork designation screen (see Figs. 68 and 69). During the period in which the viewpoint is moved, the period in which the movement speed of the viewpoint is changed is considered to be an important element in creating a free viewpoint image. Therefore, by displaying visualization information of the viewpoint movement speed as described above, it is possible to make it easier for the user to find the camera work information he or she desires, and the time required to specify the camera work information can be shortened.

[0241] Furthermore, in the second information processing apparatus of the embodiment, the display processing unit performs processing for displaying information indicating a period during which the moving speed decreases, as information that visualizes the moving speed of the viewpoint. During the period in which the viewpoint is moved, the period in which the movement speed of the viewpoint is reduced is considered to be an important factor in creating a free viewpoint image. Therefore, by displaying information indicating the period during which the viewpoint movement speed slows down as described above, it is possible to make it easier for the user to find the camera work information they desire, and the time required to specify camera work information can be shortened.

[0242] Furthermore, in the second information processing device of the embodiment, the display processing unit performs processing to display information that visualizes the field of view range from the viewpoint on the camerawork designation screen (see FIG. 61 and the like). By visually indicating the field of view, it becomes easier for the user to understand the camera work. Therefore, it becomes possible for the user to easily find the camera work information he or she desires, and the time required to specify the camera work information can be shortened.

[0243] Furthermore, in the second information processing device of the embodiment, the display processing unit performs processing for displaying a target for determining the line of sight from the viewpoint on the camerawork designation screen (see FIG. 61 and the like). This allows the user to easily grasp at what position in three-dimensional space the subject is targeted by the camera work. Therefore, it becomes possible for the user to easily find the camera work information he or she desires, and the time required to specify the camera work information can be shortened.

[0244] Furthermore, the second information processing device of the embodiment is equipped with a camerawork editing processing unit (32b) that updates the information on the position of the target in the camerawork information in response to a change in the position of the target on the camerawork designation screen (see Figures 70 and 71). This makes it possible to eliminate the need to launch software for generating camerawork information when editing the camerawork information at the stage of specifying the camerawork information to be used in generating a free viewpoint image. Therefore, even if editing of camera work information is required, the work of creating a free viewpoint image can be quickly performed.

[0245] Furthermore, in the second information processing device of the embodiment, the display processing unit performs processing to display an image obtained by observing a three-dimensional space from a viewpoint on a camerawork designation screen (see FIG. 62). This makes it possible to provide a preview of an image similar to a free viewpoint image generated based on camera work information to the user, making it easier for the user to understand the camera work. Therefore, it becomes possible for the user to easily find the camera work information he or she desires, and the time required to specify the camera work information can be shortened.

[0246] In addition, in the second information processing device of the embodiment, the display processing unit performs processing to display an image rendered from a virtual three-dimensional model of the real space as an image of the three-dimensional space observed from a viewpoint (see Figure 62). This makes it possible to realize a preview display of an observed image from a viewpoint without the need to perform rendering processing using a three-dimensional model generated from a captured image of the target real space. Therefore, the processing time required for preview display of an image observed from a viewpoint can be shortened, and the task of creating a free viewpoint image can be quickly performed.

[0247] Furthermore, in the second information processing apparatus of the embodiment, the display processing unit performs display processing of information notifying a camera in which a change in the visual field range has been detected, among a plurality of cameras (see FIG. 72). In generating a free viewpoint image, in order to accurately generate three-dimensional information from images captured by multiple cameras, it is necessary for each camera to maintain a pre-assumed position and orientation, and if a change in position or orientation occurs in any of the cameras, calibration of the parameters used to generate the three-dimensional information is required. By notifying the user of the camera in which a change in the field of view has been detected as described above, it is possible to notify the user of the camera that requires calibration. Therefore, it is possible to generate a free viewpoint image based on accurate three-dimensional information, and it is possible to improve the image quality of the free viewpoint image.

[0248] In addition, a second information processing method of the embodiment is an information processing method in which an information processing device performs display processing of a screen that shows camerawork information that corresponds to user input information among multiple camerawork information, as a camerawork designation screen that accepts an operation to designate camerawork information, which is information that indicates at least the movement trajectory of the viewpoint in a free viewpoint image. According to the second information processing method, it is possible to obtain the same functions and effects as those of the second information processing apparatus described above.

[0249] Here, as an embodiment, a program that causes a CPU, a DSP (Digital Signal Processor), or a device including these to execute the processing by the display processing unit 34a described in FIG. 52, FIG. 53, etc. can be considered. In other words, the first program of the embodiment is a program readable by a computer device, and causes the computer device to realize the function of performing display processing of a screen including a designation operation reception area that accepts operation input to designate at least a portion of the camera work information, which is information indicating at least the movement trajectory of the viewpoint in a free viewpoint image, and a camera work display area that visualizes and displays the movement trajectory of the viewpoint based on the camera work information that reflects the content specified by the operation input, as a creation operation screen for camera work information. With such a program, the above-mentioned display processing unit 34a can be realized in the device serving as the information processing device 70.

[0250] As an embodiment, a program for causing a CPU, a DSP, or a device including these to execute the processes by the display processing unit 32a described with reference to FIG. 73, FIG. 74, etc. can be considered. In other words, the second program of the embodiment is a program that can be read by a computer device and causes the computer device to execute a function of performing display processing of a screen that shows camera work information that corresponds to user input information among multiple camera work information, as a camera work designation screen that accepts an operation to designate camera work information, which is information that indicates at least the movement trajectory of the viewpoint in a free viewpoint image. By using such a program, the above-mentioned display processing unit 32a can be realized in the device serving as the information processing device .

[0251] These programs can be recorded in advance on a HDD as a recording medium built into a device such as a computer device, or on a ROM in a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0252] Moreover, such a program is suitable for a wide range of applications of the display processing unit 34a and the display processing unit 32a of the embodiment. For example, by downloading the program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, the personal computer or the like can function as a device that realizes the processing of the display processing unit 34a or the display processing unit 32a of the present disclosure.

[0253] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0254] <11. This Technology> The present technology can also be configured as follows. (1) The display processing unit performs display processing of a screen showing camera work information corresponding to user input information among a plurality of camera work information as a camera work specification screen for accepting a specification operation of camera work information, which is information showing at least a movement trajectory of a viewpoint in a free viewpoint image. Information processing device. (2) The display processing unit is On the camerawork specification screen, a process is performed to filter and display camerawork information corresponding to keywords as the input information. The information processing device according to (1). (3) The camerawork specification screen displays filtering condition information indicating filtering conditions for camerawork information, The display processing unit is The camera work information is filtered and displayed according to the filtering conditions indicated by the selected filtering condition information as the input information. The information processing device according to (1) or (2). (4) The display processing unit is A process of displaying information visualizing the movement trajectory of the viewpoint on the camera work designation screen is performed. The information processing device according to any one of (1) to (3). (5) The display processing unit is A process of displaying camera arrangement position information indicating the arrangement positions of a plurality of cameras that capture images for generating a free viewpoint image on the camera work designation screen is performed. The information processing device according to any one of (1) to (4). (6) The display processing unit is A process is performed to display start point arrangement position information and end point arrangement position information indicating the positions of a camera that is a movement start point and a movement end point of a camera among the plurality of cameras on the camera work designation screen. The information processing device according to (5). (7) The display processing unit is A process is performed to display the start point arrangement position information and the end point arrangement position information in a different manner from arrangement position information of the cameras other than the camera serving as the movement start point and the camera serving as the movement end point among the plurality of cameras. The information processing device according to (6). (8) The display processing unit is A process of displaying information visualizing the movement speed of the viewpoint on the camera work designation screen is performed. The information processing device according to any one of (4) to (7). (9) The display processing unit is A process of displaying information indicating a period during which the moving speed of the viewpoint is decreased as information visualizing the moving speed of the viewpoint is performed. The information processing device according to (8). (10) The display processing unit is A process of displaying information visualizing the field of view from the viewpoint on the camera work designation screen is performed. The information processing device according to any one of (4) to (9). (11) The display processing unit is A process of displaying a target for determining a line of sight from the viewpoint on the camera work designation screen is performed. The information processing device according to any one of (4) to (10). (12) a camerawork editing processing unit that updates information on the position of the target in the camerawork information in response to a change operation of the position of the target on the camerawork designation screen; The information processing device according to (11). (13) The display processing unit is A process of displaying an image obtained by observing a three-dimensional space from the viewpoint on the camera work specification screen is performed. The information processing device according to any one of (1) to (12). (14) The display processing unit is As an image of the three-dimensional space observed from the viewpoint, a process is performed to display an image obtained by rendering a virtual three-dimensional model of the real space, rather than a three-dimensional model generated from a captured image of the real space. The information processing device according to (13). (15) The display processing unit is A process for displaying information notifying a camera in which a change in the visual field range has been detected among the plurality of cameras is performed. The information processing device according to (5). (16) An information processing device, As a camerawork specification screen for accepting a specification operation of camerawork information, which is information showing at least a movement trajectory of a viewpoint in a free viewpoint image, a display process is performed on a screen showing camerawork information corresponding to input information of a user among a plurality of pieces of camerawork information that can be specified, by filtering the camerawork information. Information processing methods. (17) A computer readable program, The computer device is made to realize a function of performing display processing of a screen showing, as a camerawork designation screen for accepting designation operation of camerawork information, which is information showing at least a movement trajectory of a viewpoint in a free viewpoint image, a filtered screen showing camerawork information corresponding to input information from a user among a plurality of pieces of camerawork information that can be designated. program. [Explanation of symbols]

[0255] 2. Free viewpoint image server 8 Utility Server 10. Imaging device 21 Section specific processing unit 22 Target image transmission control section 23 Output image generation unit 31 Target image acquisition unit 32 Image generation processing section 32a Display processing unit 32b Camera work editing processing section 33 Transmission control section Gs Camera work specification screen 41 Scene Window 42 Scene list display section 43 Camera Work Window 44 Camera work list display 70 Information processing equipment 71 CPU 72 ROM 73 RAM 74 Bus 75 Input / Output Interface 76 Input section 77 Display section 78 Audio output section 79 Memory section 80 Communications Department 81 Removable recording media 82 Drive Tg target Mc camera position mark Fv Field of view information Mt Target Mark Mm Movement trajectory information Mv waypoint mark Mtn target position mark Mtt additional target mark Mem target timing mark Mst target initial position mark Rf Field of View Dg View direction 48 Filtering operation section 48a Drop-down list 48b Keyword input section B33 Play button B34 Pause button B35 Stop button B36 X-axis viewpoint button B37 Y-axis viewpoint button B38 Z-axis viewpoint button B39 Ca Viewpoint Button B40 Pe Viewpoint Button B43 Pull-down button B44 Reset button

Claims

1. a display processing unit that performs display processing of a screen showing camera work information corresponding to user input information among a plurality of camera work information, as a camera work designation screen that accepts a designation operation of camera work information, which is information showing at least a movement trajectory of a viewpoint in a free viewpoint image, and The display processing unit is A process is performed to display information visualizing the movement trajectory of the viewpoint and information visualizing the movement speed of the viewpoint on the camera work designation screen. Information processing device.

2. The display processing unit is On the camerawork specification screen, a process is performed to filter and display camerawork information corresponding to keywords as the input information. The information processing device according to claim 1 .

3. The camerawork specification screen displays filtering condition information indicating filtering conditions for camerawork information, The display processing unit is The camera work information is filtered and displayed according to the filtering conditions indicated by the selected filtering condition information as the input information. The information processing device according to claim 1 .

4. The display processing unit is A process of displaying camera arrangement position information indicating the arrangement positions of a plurality of cameras that capture images for generating a free viewpoint image on the camera work designation screen is performed. The information processing device according to claim 1 .

5. The display processing unit is A process is performed to display start point arrangement position information and end point arrangement position information indicating the positions of a camera that is a movement start point and a movement end point of a camera among the plurality of cameras on the camera work designation screen. The information processing device according to claim 4.

6. The display processing unit is A process is performed to display the start point arrangement position information and the end point arrangement position information in a different manner from arrangement position information of the cameras other than the camera serving as the movement start point and the camera serving as the movement end point among the plurality of cameras. The information processing device according to claim 5 .

7. The display processing unit is A process of displaying information indicating a period during which the moving speed of the viewpoint is decreased as information visualizing the moving speed of the viewpoint is performed. The information processing device according to claim 1 .

8. The display processing unit is A process of displaying information visualizing the field of view from the viewpoint on the camera work designation screen is performed. The information processing device according to claim 1 .

9. The display processing unit is A process of displaying a target for determining a line of sight from the viewpoint on the camera work designation screen is performed. The information processing device according to claim 1 .

10. a camerawork editing processing unit that updates information on the position of the target in the camerawork information in response to a change in the position of the target on the camerawork designation screen; The information processing device according to claim 9.

11. The display processing unit is A process of displaying an image obtained by observing a three-dimensional space from the viewpoint on the camera work specification screen is performed. The information processing device according to claim 1 .

12. The display processing unit is A process is performed to display an image obtained by rendering a virtual three-dimensional model of the real space as an image obtained by observing the three-dimensional space from the viewpoint. The information processing device according to claim 11.

13. The display processing unit is A process for displaying information notifying a camera in which a change in the visual field range has been detected among the plurality of cameras is performed. The information processing device according to claim 4.

14. An information processing device, A display process is performed on a screen showing camera work information corresponding to user input information among a plurality of camera work information filtered as a camera work designation screen for accepting a designation operation of camera work information, which is information showing at least a movement trajectory of a viewpoint in a free viewpoint image, and a process is performed on the camera work designation screen for displaying information visualizing the movement trajectory of the viewpoint and information visualizing the movement speed of the viewpoint. Information processing methods.

15. A computer readable program, The computer device is provided with a function of performing a display process of a screen showing camera work information corresponding to user input information among a plurality of camera work information filtered as a camera work specification screen for accepting a specification operation of camera work information which is information showing at least a movement trajectory of a viewpoint in a free viewpoint image, and performing a process of displaying information visualizing the movement trajectory of the viewpoint and information visualizing the movement speed of the viewpoint on the camera work specification screen. program.

Images