Information processing device, information processing method, and program

Abstract

An information processing device according to the present technology comprises a selection processing unit that performs the selection of data to be used in free-viewpoint image generation in accordance with the importance of at least one of an event and a viewpoint, where a plurality of captured image data, which is obtained by capturing images of an event from a plurality of viewpoints, and processing data, which is obtained by executing on the captured image data a process relating to three-dimensional information generation of at least a subject, serve as selection target data.

Description

[Technical Field]

[0001] The present technology relates to an information processing device, a method thereof, and a program, and in particular to the technical field of processing related to generation of a free viewpoint image that allows a captured subject to be viewed from any viewpoint in three-dimensional space. [Background technology]

[0002] A technology is known that generates a free viewpoint image (also called a free viewpoint video, virtual viewpoint image (video), etc.), which corresponds to an image that can be viewed from any viewpoint in three-dimensional space, based on three-dimensional information that represents an image of a captured subject in three-dimensional space.

[0003] As a related prior art, the following Patent Document 1 can be cited: Patent Document 1 discloses a technology for distributing multi-viewpoint video images obtained by a plurality of cameras to a client PC via the Internet. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-183209 Summary of the Invention [Problem to be solved by the invention]

[0005] Here, images captured by a large number of cameras are used to generate a free viewpoint image, but storing the images captured by all the cameras for generating the free viewpoint image requires a huge memory capacity.

[0006] The present technology has been developed in view of the above circumstances, and aims to reduce the amount of data stored for generating free viewpoint images. [Means for solving the problem]

[0007] The information processing device according to the present technology includes a selection processing unit that selects data to be used for generating a free viewpoint image based on the importance of at least one of the events or the viewpoints, using multiple captured image data obtained by capturing an event from multiple viewpoints and processed data obtained by performing processing on the captured image data related to generating at least three-dimensional information of a subject as selection target data. This makes it possible to, for example, store captured image data for only important scenes among multiple scenes constituting an event for free viewpoint image generation, or store captured image data for only important viewpoints among multiple viewpoints for free viewpoint image generation, or to store captured image data for important scenes for free viewpoint image generation, while storing processed data rather than captured image data for unimportant scenes for free viewpoint image generation.

[0008] In addition, an information processing method related to the present technology is an information processing method in which an information processing device selects data to be used for generating a free viewpoint image based on the importance of at least one of the event or the viewpoints, using multiple captured image data obtained by capturing an event from multiple viewpoints and processed data obtained by performing processing on the captured image data related to generating at least three-dimensional information of a subject as selection target data. Furthermore, the program related to the present technology is a program readable by a computer device, and causes the computer device to realize a function of selecting data to be used for generating a free viewpoint image, using multiple captured image data obtained by capturing an event from multiple viewpoints and processed data obtained by performing processing on the captured image data related to generating at least three-dimensional information of a subject, as selection target data, in accordance with the importance of at least one of the event or the viewpoints. These information processing methods and programs make it possible to realize the information processing device according to the present technology. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 1 is a block diagram of a system configuration according to an embodiment of the present technology. [Figure 2] FIG. 2 is an explanatory diagram illustrating an example of a camera arrangement for generating a free viewpoint image according to an embodiment. [Figure 3] FIG. 1 is a block diagram illustrating a hardware configuration of an information processing apparatus according to an embodiment. [Figure 4] FIG. 2 is an explanatory diagram of functions of an image creation controller according to an embodiment. [Figure 5] FIG. 2 is an explanatory diagram of functions of a free viewpoint image server according to an embodiment. [Figure 6] FIG. 2 is an explanatory diagram of a viewpoint in a free viewpoint image according to an embodiment. [Figure 7] FIG. 10 is an explanatory diagram of an overview of a generation operation screen in the embodiment. [Figure 8] FIG. 10 is an explanatory diagram illustrating an overview of a path creation screen according to an embodiment. [Figure 9] FIG. 4 is an explanatory diagram of an output clip according to the embodiment. [Figure 10] 10 is an explanatory diagram of an output clip including a still image FV clip according to an embodiment. FIG. [Figure 11] FIG. 10 is an explanatory diagram of an output clip including a moving image FV clip according to an embodiment. [Figure 12] 10A and 10B are explanatory diagrams illustrating an example of an image of an output clip according to the embodiment. [Figure 13] 10A to 10C are explanatory diagrams of a procedure for creating a clip according to an embodiment. [Figure 14] 10A to 10C are explanatory diagrams of a procedure for detecting camera fluctuations according to an embodiment. [Figure 15] FIG. 2 is an explanatory diagram of a data flow related to free viewpoint image generation in the embodiment. [Figure 16] FIG. 10 is an explanatory diagram of silhouette image data. [Figure 17] 17 is a diagram illustrating an example of an image of 3D data corresponding to the subject illustrated in FIG. 16. FIG. [Figure 18] FIG. 10 is a diagram illustrating an example of polygon mesh data. [Figure 19] 10 is a flowchart illustrating an example of a processing procedure corresponding to a first example and a second example of a method for selecting data to be saved according to an embodiment. [Figure 20] 10 is a flowchart illustrating an example of a processing procedure corresponding to a third example of a method for selecting data to be saved according to an embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0010] The embodiments will be described below in the following order. <1. System configuration> <2. Configuration of Image Creation Controller and Free Viewpoint Image Server> <3. GUI Overview> <4. Clips containing free viewpoint images> <5. Clip creation process> 6. Camera movement detection <7. Data flow for generating free viewpoint images> 8. Storage data selection method as an embodiment <9. Processing Procedure> <10. Variations> <11. Summary of embodiments> <12. This Technology>

[0011] <1. System configuration> FIG. 1 shows an example of the configuration of an image processing system according to an embodiment of the present technology. The image processing system includes an image creation controller 1, a free viewpoint image server 2, a video server 3, multiple (e.g., four) video servers 4A, 4B, 4C, and 4D, a NAS (Network Attached Storage) 5, a switcher 6, an image conversion unit 7, a utility server 8, and multiple (e.g., 16) imaging devices 10. Hereinafter, the term "camera" refers to the imaging device 10. For example, "camera arrangement" refers to an arrangement of multiple imaging devices 10. Furthermore, when the video servers 4A, 4B, 4C, and 4D are referred to collectively without any particular distinction, they will be referred to as "video servers 4." This image processing system generates free viewpoint images corresponding to images seen from any viewpoint in three-dimensional space based on captured images (e.g., image data V1 to V16) obtained from multiple imaging devices 10, and can create an output clip containing the free viewpoint images.

[0012] In FIG. 1, the connection state of each part is indicated by a solid line, a broken line, or a double line. The solid lines indicate SDI (Serial Digital Interface) connections, an interface standard for connecting broadcasting equipment such as cameras and switchers, and are compatible with 4K, for example. Image data is mainly sent and received between each device via SDI wiring.

[0013] The double lines indicate connections based on communication standards for building a computer network, such as 10 Gigabit Ethernet. Image creation controller 1, free viewpoint image server 2, video servers 3, 4A, 4B, 4C, and 4D, NAS 5, and utility server 8 are connected via a computer network, enabling them to send and receive image data and various control signals to and from each other.

[0014] The dashed line between video servers 3 and 4 indicates that video servers 3 and 4, which are equipped with inter-server file sharing functionality, are connected via, for example, a 10G network. This allows each video server to preview and transmit material stored on other video servers between video server 3 and video servers 4A, 4B, 4C, and 4D. In other words, a system using multiple video servers is constructed, enabling efficient highlight editing and transmission.

[0015] Each imaging device 10 is configured as a digital camera device having an imaging element such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, and obtains captured images (image data V1 to V16) as digital data. In this example, each imaging device 10 obtains captured images as moving images.

[0016] In this example, each imaging device 10 captures an image of a sport such as basketball, soccer, or golf being played, and is disposed in a predetermined position and orientation at the venue where the sport is being held. In this example, there are 16 imaging devices 10, but the number of imaging devices 10 only needs to be at least two in order to enable the generation of a free viewpoint image. By increasing the number of imaging devices 10 and capturing images of the target subject from more angles, the accuracy of three-dimensional reconstruction of the subject can be improved, and the image quality of the virtual viewpoint image can be improved.

[0017] Fig. 2 shows an example of the placement of imaging devices 10 around a basketball court. The circles represent imaging devices 10. This is an example of camera placement when you want to focus on capturing images of the area near the goal on the left side of the drawing. Of course, the placement and number of cameras are just examples, and should be determined according to the content and purpose of the filming or broadcasting. Furthermore, events for which free viewpoint images are generated are not limited to sporting events such as basketball games, but are diverse.

[0018] The image creation controller 1 is configured by an information processing device and can be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like. The image creation controller 1 controls and manages the operations of the video servers 3 and 4, and performs processing for creating clips. As an example, the image creation controller 1 is a device that can be operated by an operator OP1. The operator OP1 selects clip contents, gives instructions for creation, and so on.

[0019] The free viewpoint image server 2 is configured as an information processing device that performs processing to actually create free viewpoint images (FV (Free View) clips, which will be described later) in response to instructions from the image creation controller 1. This free viewpoint image server 2 can also be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like. As an example, the free viewpoint image server 2 is a device that can be operated by an operator OP2. The operator OP2 performs work related to creating an FV clip as a free viewpoint image, for example. Specifically, the operator OP2 performs an operation of specifying (selecting) a camera path for generating a free viewpoint image. In this example, the operator OP2 also performs work of creating a camera path.

[0020] Here, the camera path information is information including at least information indicating the movement trajectory of the viewpoint in the free viewpoint image. For example, when creating a free viewpoint image in which the viewpoint position, line of sight direction, and angle of view (focal length) are changed for a subject for which 3D data (described later) is generated, the parameters required to determine the movement trajectory of the viewpoint, the change in line of sight direction, and the change in angle of view are regarded as the camera path information.

[0021] The configurations and processes of the image creation controller 1 and the free viewpoint image server 2 will be described in detail later. Furthermore, although it is assumed that the operations are performed by operators OP1 and OP2, for example, the image creation controller 1 and the free viewpoint image server 2 may be arranged side by side and operated by a single operator.

[0022] The video servers 3 and 4 are each an image recording device, and include a data recording unit such as an SSD (Solid State Drive) or HDD (Hard Disk Drive), and a control unit that controls recording and playback of data in the data recording unit.

[0023] Each of the video servers 4A, 4B, 4C, and 4D can accept, for example, four input systems, and simultaneously record images captured by the four imaging devices 10. For example, video server 4A records image data V1, V2, V3, and V4. Video server 4B records image data V5, V6, V7, and V8. Video server 4C records image data V9, V10, V11, and V12. Video server 4D records image data V13, V14, V15, and V16. As a result, images captured by all 16 imaging devices 10 are recorded simultaneously. The video servers 4A, 4B, 4C, and 4D are assumed to be constantly recording, for example, during a sports match to be broadcast.

[0024] The video server 3 is, for example, directly connected to the image creation controller 1, and is capable of, for example, two input systems and two output systems. Image data Vp and Vq are shown as the two input systems. As the image data Vp and Vq, images captured by any two of the image capture devices 10 (any two of the image data V1 to V16) can be selected. Of course, images captured by other image capture devices may also be used.

[0025] The image data Vp and Vq can be displayed as monitor images on a display by the image creation controller 1. The operator OP1 can check the status of a scene being shot and recorded for broadcast, for example, using the image data Vp and Vq input to the video server 3. Furthermore, since the video servers 3 and 4 are connected in a file sharing state, the image creation controller 1 can also display on the monitor the images captured by each imaging device 10 that are recorded on the video servers 4A, 4B, 4C, and 4D, allowing the operator OP1 to check them one by one.

[0026] In this example, a time code is added to the image captured by each imaging device 10, making it possible to achieve frame synchronization in the processing in the video servers 3, 4A, 4B, 4C, and 4D.

[0027] The NAS 5 is a storage device arranged on a network, and is configured, for example, with an SSD, HDD, etc. In this example, the NAS 5 is a device that stores some frames of image data V1, V2, ..., V16 recorded in the video servers 4A, 4B, 4C, and 4D when they are transferred for generating free-viewpoint images, for processing in the free-viewpoint image server 2, and stores the generated free-viewpoint images.

[0028] The switcher 6 is a device that receives the images output via the video server 3 and selects the main line image PGMout to be finally selected and broadcast. For example, a broadcast director or the like performs the necessary operations.

[0029] The image conversion unit 7 performs resolution conversion and synthesis of image data from the imaging device 10, for example, to generate a monitoring image of the camera arrangement and supply it to the utility server 8. For example, 16-channel image data (V1 to V16) that are 4K images are converted in resolution to HD images, and then the images are converted into 4-channel images arranged in a tiled pattern and supplied to the utility server 8.

[0030] The utility server 8 is a computer device capable of various related processes, but in this example, it is a device that performs a process of detecting camera movement for calibration in particular. For example, the utility server 8 monitors image data from the image conversion unit 7 to detect camera movement. Camera movement refers to movement of any of the placement positions of the image capture devices 10, which are placed, for example, as shown in FIG. 2. Information about the placement positions of the image capture devices 10 is an important element in generating free viewpoint images, and if the placement positions change, parameter settings must be redone. For this reason, camera movement is monitored.

[0031] <2. Configuration of Image Creation Controller and Free Viewpoint Image Server> The image creation controller 1, the free viewpoint image server 2, the video servers 3 and 4, and the utility server 8 in the above configuration can be realized as an information processing device 70 having the configuration shown in FIG. 3, for example.

[0032] 3, a CPU 71 of an information processing device 70 executes various processes in accordance with a program stored in a ROM 72 or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes as appropriate. The CPU 71, ROM 72, and RAM 73 are interconnected via a bus 74. An input / output interface 75 is also connected to this bus 74.

[0033] The input / output interface 75 is connected to an input unit 76 that includes an operator and an operation device. For example, the input unit 76 may be various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71.

[0034] Furthermore, the input / output interface 75 is connected integrally or separately to a display unit 77 made up of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, etc., and an audio output unit 78 made up of a speaker, etc. The display unit 77 is a display unit that displays various types of information, and is configured by, for example, a display device provided in the housing of the information processing device 70, or a separate display device connected to the information processing device 70, or the like. The display unit 77 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface), based on instructions from the CPU 71.

[0035] The input / output interface 75 may be connected to a storage unit 79 configured with a hard disk or solid-state memory, or a communication unit 80 configured with a modem or the like. The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like.

[0036] A drive 82 is also connected to the input / output interface 75 as required, and a removable recording medium 81 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. The drive 82 allows data files such as image files MF, various computer programs, etc. to be read from the removable recording medium 81. The read data files are stored in the storage unit 79, and the images and sounds contained in the data files are output on the display unit 77 and the sound output unit 78. Furthermore, the computer programs, etc. read from the removable recording medium 81 are installed in the storage unit 79 as necessary.

[0037] In this information processing device 70, software can be installed via network communication by a communication unit 80 or via a removable recording medium 81. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0038] When the image creation controller 1 and the free viewpoint image server 2 are realized using such an information processing device 70, the processing functions shown in FIGS. 4 and 5 are realized in the CPU 71 by software, for example.

[0039] FIG. 4 shows a section identification processing unit 21, a target image transmission control unit 22, an output image generation unit 23, and a selection processing unit 24 as functions formed in a CPU 71 of an information processing device 70 serving as the image creation controller 1.

[0040] The section identification processing unit 21 performs processing to identify a target image section for generating a free viewpoint image for a plurality of captured images (image data V1 to V16) captured simultaneously by a plurality of imaging devices 10. For example, in response to an operation by an operator OP1 to select a scene to be replayed within an image, the processing unit 21 performs processing to identify a time code for that scene, particularly for a section of the scene to be used as a free viewpoint image (target image section for generation), and notify the free viewpoint image server 2 of the time code.

[0041] To clarify, free viewpoint images are assumed to be generated during broadcasting (subject to time constraints) in order to obtain images distributed during broadcasting, such as replay images, and to be generated anew after broadcasting (with ample time) based on recorded data. Free viewpoint images generated after broadcasting can be used, for example, in news programs or saved as archive content. In the following description, unless otherwise specified, it is assumed that free viewpoint images are generated during broadcasting.

[0042] Here, the above-mentioned target image section refers to a frame section that will actually be used as a free viewpoint image. When a free viewpoint image is generated for a certain frame in a video, that frame becomes the target image section. In this case, the in point (start point) and out point (end point) for the free viewpoint image will have the same time code. When generating a free viewpoint image for a section of multiple frames in a video, the multiple frames are the image section to be generated. In this case, the in point and out point for the free viewpoint image have different time codes. The structure of a clip will be described later, but it is assumed that the in / out points of the image section to be generated will differ from the in / out points of the output clip that will ultimately be generated, because a previous clip and a next clip will be combined, as will be described later.

[0043] The target image transmission control unit 22 controls the transmission of image data of the image section to be generated in each of the multiple imaging devices 10, i.e., one or more frames of image data V1 to V16, as image data to be used for generating a free viewpoint image in the free viewpoint image server 2. Specifically, the target image transmission control unit 22 controls the transfer of image data as the image section to be generated from the video servers 4A, 4B, 4C, and 4D to the NAS 5.

[0044] The output image generating unit 23 performs processing to generate an output image (output clip) including the free viewpoint image (FV clip) generated by the free viewpoint image server 2 and received. For example, the image creation controller 1 generates an output clip by combining, on the time axis, a previous clip, which is an actual video at an earlier point in time, and a subsequent clip, which is an actual video at a later point in time, with an FV clip, which is a virtual image generated by the free viewpoint image server 2, through processing by the output image generation unit 23. That is, the previous clip + FV clip + subsequent clip are combined into one output clip. Of course, the previous clip and the FV clip may be used as one output clip. Alternatively, the FV clip and the after clip may be used as one output clip. Furthermore, an output clip may be generated that is only an FV clip without combining the preceding clip or the following clip. In either case, the image creation controller 1 generates an output clip including the FV clip and outputs it to the switcher 6 so that it can be used for broadcasting.

[0045] The selection processing unit 24 selects data to be used for generating a free viewpoint image. The details of the processing performed by the CPU 71 of the image creation controller 1 as the selection processing unit 24 will be explained later.

[0046] FIG. 5 shows a target image acquisition unit 31, an image generation processing unit 32, and a transmission control unit 33 as functions formed in a CPU 71 of an information processing device 70 serving as the free viewpoint image server 2.

[0047] The target image acquisition unit 31 performs processing to acquire image data of a target image section for generating a free viewpoint image in each of a plurality of captured images (image data V1 to V16) simultaneously captured by a plurality of imaging devices 10. That is, the image creation controller 1 acquires image data of one frame or multiple frames specified by the in-point / out-point of the target image section for generating a free viewpoint image from the video servers 4A, 4B, 4C, 4D via the NAS 5, so that the image data can be used to generate a free viewpoint image.

[0048] For example, the target image acquisition unit 31 acquires image data of one frame or multiple frames in the image section to be generated for all of the image data V1 to V16. The reason for acquiring image data of the image section to be generated for all of the image data V1 to V16 is to generate a high-quality free-viewpoint image. As described above, a free-viewpoint image can be generated by using images captured by at least two or more imaging devices 10, but by increasing the number of imaging devices 10 (i.e., the number of viewpoints), more detailed three-dimensional information about the subject can be generated, enabling the generation of a high-quality free-viewpoint image.

[0049] The image generation processing unit 32 has a function of generating a free viewpoint image, that is, an FV clip in this example, using the image data acquired by the target image acquisition unit 31. In this example, the image generation processing unit 32 is capable of generating free viewpoint images by a VDP (View Dependent Player) method and a VIDP (View Independent Player) method. The VDP method is a technique for generating free viewpoint images by applying texture images according to the viewpoint to 3D data generated from image data captured from multiple viewpoints using the visual hull method. The VDP method requires that an image for each viewpoint be prepared as a texture image. The VIDP method is a technique that generates a 3D model of a subject as polygon mesh data from image data captured from multiple viewpoints, generates a texture image as a UV map texture, and generates a free viewpoint image using CG (Computer Graphics) based on the polygon mesh data and UV map texture. Here, the UV map texture is two-dimensional data obtained by UV-expanding a 3D model made of a polygon mesh, and refers to data that indicates color information for each polygon (e.g., triangle). 3D data created using the volume intersection method and 3D models created using polygon meshes will be explained later.

[0050] The image generation processing unit 32 has a processing data generation unit 32a, a first FV generation unit 32b, and a second FV generation unit 32c as functional units for generating free viewpoint images using the VDP method or VIDP method as described above. The processing data generation unit 32a performs processing related to generating three-dimensional information of the subject based on the image data V1 to V16. Specifically, it performs processing to generate 3D data using the volume intersection method described above and a 3D model as a polygon mesh, and performs the generation of silhouette image data (described later), generation of 3D data based on the silhouette image data, generation of a 3D model based on the 3D data, generation of a UV map texture, etc. Specific examples of methods for generating silhouette image data, 3D data, and 3D models will be explained later.

[0051] The first FV generation unit 32b represents a function for generating a free viewpoint image by the VDP method, and the second FV generation unit 32c represents a function for generating a free viewpoint image by the VIDP method. Specifically, the first FV generating unit 32b generates a free viewpoint image by the VDP method based on the 3D data generated by the processing data generating unit 32a and a texture image for each viewpoint. The second FV generation unit 32c generates a free viewpoint image by the VIDP method based on the 3D model (polygon mesh data) and UV map texture generated by the processing data generation unit 32a. In the following, the generation of a two-dimensional image, a free viewpoint image, from three-dimensional information such as 3D data or a 3D model will sometimes be referred to as "rendering."

[0052] The viewpoint of a free viewpoint image will be described with reference to FIG. 6A shows an image of a free viewpoint image in which a subject is captured from a desired viewpoint set in three-dimensional space. In this free viewpoint image, subject M1 is viewed from approximately the front, and subject M2 is viewed from approximately the back. Fig. 6B shows an image of a virtual viewpoint image when the viewpoint position is changed in the direction of arrow C in Fig. 6A and a viewpoint that views subject M1 from approximately the back is set. In the free viewpoint image in Fig. 6B, subject M2 is viewed approximately from the front, and subject M3 and a basketball goal that were not shown in Fig. 6A are also shown. For example, if the viewpoint is gradually moved from the state of Figure 6A in the direction of arrow C to the state of Figure 6B, an image of about 1 to 2 seconds will be generated as a free viewpoint image (FV clip). Of course, the time length of the FV clip as a free viewpoint image and the trajectory of the viewpoint movement can be variously considered.

[0053] 5, the transmission control unit 33 controls the transmission of free viewpoint images (FV clips) generated by the image generation processing unit 32 to the image creation controller 1 via the NAS 5. In this case, the transmission control unit 33 also controls the transmission of accompanying information for generating output images to the image creation controller 1. The accompanying information is assumed to be information specifying images of the previous clip and the next clip. In other words, it is information specifying which image from the image data V1 to V16 is to be used to create the previous clip and the next clip. The accompanying information also may include information specifying the duration of the previous clip and the next clip.

[0054] Here, the CPU 71 of the free viewpoint image server 2 also performs processing related to the generation of camera path information used to generate free viewpoint images. When creating free viewpoint images, multiple candidate camera paths are created (preset) in advance to accommodate various scenes. To enable the pre-creation of such camera paths, a software program for creating camera paths is installed in the free viewpoint image server 2 of this example.

[0055] <3. GUI Overview> 7 and 8, an overview of the generation operation screen Gs used to generate a free viewpoint image and the path creation screen Gg used to create a camera path will be described. In this example, the generation operation screen Gs and the path creation screen Gg are displayed on the display unit 77 of the free viewpoint image server 2, for example, and can be confirmed and operated by the operator OP2.

[0056] On the generation operation screen Gs shown in FIG. 7, a scene window 41, a scene list display section 42, a camera path window 43, a camera path list display section 44, a parameter display section 45, and a transmission window 46 are arranged. In the scene window 41, for example, an image of the image section to be generated is displayed on a monitor, so that the operator OP2 can check the contents of the scene for which a free viewpoint image is to be generated. For example, a list of scenes designated as the image interval to be generated is displayed in the scene list display section 42. The operator OP2 can use the scene list display section 42 to select a scene to be displayed in the scene window 41.

[0057] The camera path window 43 displays the position of the image capture device 10, the selected camera path, or a plurality of selectable camera paths. As mentioned above, camera path information is information that includes at least information indicating the movement trajectory of the viewpoint in a free viewpoint image. For example, when creating a free viewpoint image in which the viewpoint position, line of sight, and angle of view are changed relative to a subject, the parameters necessary to determine the movement trajectory of the viewpoint, the changing manner of the line of sight, and the changing manner of the angle of view are considered to be camera path information. The camera path window 43 displays at least information visualizing the movement trajectory of the viewpoint as a camera path display.

[0058] Information about various camera paths that have been created and stored in advance is displayed in a list in the camera path list display section 44. The operator OP2 can select and specify a camera path to be used for generating an FV clip from among the camera paths displayed in the camera path list display section 44. The parameter display section 45 displays various parameters related to the selected camera path.

[0059] The transmission window 46 displays information about transmitting the created FV clip to the image creation controller 1.

[0060] Next, the path creation screen Gg of FIG. 8 will be described. The path creation screen Gg has a preset list display section 51, a camera path list display section 52, a camera path window 53, an operation panel section 54, and a preview window 55 arranged thereon.

[0061] The preset list display section 51 can selectively display a camera preset list, a target preset list, or a 3D model preset list. The camera preset list is a list of position information (position information in three-dimensional space) for each camera preset by the user regarding the camera placement position on-site. When the camera preset list is selected, the preset list display unit 51 displays a list of information indicating the position for each camera identification information (e.g., camera1, camera2, . . . , camera16).

[0062] In addition, in the target preset list, a target means a target position that determines the line of sight from the viewpoint in a free viewpoint image. In generating a free viewpoint image, the line of sight from the viewpoint is determined to face the target. When a preset list of targets is selected, the preset list display section 51 displays a list of identification information about the targets preset by the user and information indicating their positions. Hereinafter, the target that determines the line of sight direction from the viewpoint in the free viewpoint image as described above will be referred to as a "target Tg."

[0063] The 3D model preset list is a preset list of 3D models that is displayed as the background of the camera path window 43, and when a 3D model preset list is selected, the preset list display section 51 displays a list of identification information for the preset 3D models.

[0064] The camera path list display section 52 can display a list of information on camera paths created through the path creation screen Gg and information (information as entries) on camera paths to be newly created through the path creation screen Gg.

[0065] The camera path window 53 displays at least information that visualizes the movement trajectory of the viewpoint as a display of the camera path. The operation panel section 54 is an area for receiving various operation inputs for creating a camera path. An image seen from the viewpoint is displayed in the preview window 55. When an operation to move the viewpoint on the movement trajectory is performed, images seen from each viewpoint position on the movement trajectory are successively displayed in the preview window 55. Furthermore, in this example, when a camera preset list is displayed in the preset list display section 51 and an operation to specify a camera from the preset list of the camera is performed, the preview window 55 displays an image seen from the placement position of the camera.

[0066] For example, a user such as operator OP2 can use this path creation screen Gg to create and edit a camera path while successively previewing the contents of the camera path (changes in image content accompanying movement of the viewpoint).

[0067] <4. Clips containing free viewpoint images> Next, an output clip including an FV clip as a free viewpoint image will be described. FIG. 9 shows an example of an output clip that is configured by linking a preceding clip, an FV clip, and a following clip.

[0068] For example, the previous clip is an actual moving image in the section from time code TC1 to TC2 in certain image data Vx among image data V1 to V16. The subsequent clip is the actual video in the section from time code TC5 to TC6 in certain image data Vy among the image data V1 to V16. It is generally assumed that the image data Vx is image data of the imaging device 10 at the start of viewpoint movement by FV clip, and the image data Vy is image data of the imaging device 10 at the end of viewpoint movement by FV clip.

[0069] In this example, the previous clip is a video with a duration of t1, the FV clip is a free-viewpoint image with a duration of t2, and the next clip is a video with a duration of t3. The total playback time of the output clip is t1 + t2 + t3. For example, a 5-second output clip could consist of 1.5 seconds of video, 2 seconds of free-viewpoint image, and 1.5 seconds of video.

[0070] Here, the FV clip is shown as the section from time code TC3 to TC4, but this may or may not correspond to the number of frames in the actual video. In other words, for FV clips, there are cases where the viewpoint is moved while the video time is stopped (TC3 = TC4), and cases where the viewpoint is moved without stopping the video time (TC3 ≠ TC4). For the purpose of explanation, an FV clip in which the viewpoint is moved while the video time is stopped (called "time freeze") will be called a "still image FV clip," and an FV clip in which the viewpoint is moved without stopping the video time (called "free run") will be called a "video FV clip."

[0071] Figure 10 shows a still image FV clip relative to video frames. In this example, the time codes TC1 and TC2 of the previous clip are the time codes of frames F1 and F81, and the time code of the following frame F82 is the time code TC3=TC4 in Figure 9. The time codes TC5 and TC6 of the next clip are the time codes of frames F83 and F166. That is, a free viewpoint image in which the viewpoint moves is generated for a single still image of frame F82.

[0072] On the other hand, a video FV clip will look like Figure 11. In this example, the time codes TC1 and TC2 of the previous clip become the time codes of frames F1 and F101, and the time codes of frames F102 and F302 become the time codes TC3 and TC4 in Figure 9. The time codes TC5 and TC6 of the next clip become the time codes of frames F303 and F503. That is, this is the case where a free viewpoint image in which the viewpoint moves is generated for a video in the section of a plurality of frames from frame F102 to F302.

[0073] Therefore, the image section to be generated determined by the image creation controller 1 is the one-frame section of frame F82 when creating the still image FV clip of Figure 10, and is the multiple-frame section from frame F102 to frame 302 when creating the video FV clip of Figure 11.

[0074] For the example of the still image FV clip in FIG. 10, an example of the image content of the output clip is shown in FIG. 12, the previous clip is actual video from frame F1 to frame F81. The FV clip is a virtual image in which the viewpoint is moved in the scene of frame F82. The next clip is actual video from frame F83 to frame F166. For example, an output clip including an FV clip is generated in this way and used as the image to be broadcast.

[0075] <5. Clip creation process> Below, an example of the process of creating an output clip performed in the image processing system of Fig. 1 will be described, focusing mainly on the processes of the image creation controller 1 and the free viewpoint image server 2. First, the processing flow including the operations of operators OP1 and OP2 will be described with reference to Fig. 13. Note that the processing of operator OP1 in Fig. 13 collectively represents the GUI processing of the image creation controller 1 and the operator operations. Also, the processing of operator OP2 collectively represents the GUI processing of the free viewpoint image server 2 and the operator operations.

[0076] Step S1: Select a scene When creating an output clip, operator OP1 first selects a scene to be used as an FV clip. For example, operator OP1 searches for a scene that he or she wants to use as an FV clip while monitoring the captured images displayed on the display unit 77 on the image creation controller 1 side. Then, he or she selects an image section of one or more frames to be generated. This information on the image section to be generated is transmitted to the free viewpoint image server 2, and is made recognizable to the operator OP2 by the GUI on the display unit 77 on the free viewpoint image server 2 side. Specifically, the information on the image section to be generated is the information on the time codes TC3 and TC4 in Fig. 9. As described above, in the case of a still image FV clip, the time code TC3=TC4.

[0077] Step S2: Scene image transfer instruction The operator OP2 performs an operation to instruct the transfer of the image of the corresponding scene according to the designation of the image section to be generated. In response to this operation, the free viewpoint image server 2 transmits a transfer request for the image data of the section of time codes TC3 and TC4 to the image generation controller 1.

[0078] Step S3: Synchronous extraction In response to a request to transfer image data, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to extract the sections of time codes TC3 and TC4 for each of the 16 systems of image data from image data V1 to image data V16. Step S4: Transfer to NAS The image creation controller 1 then transfers to the NAS 5 all data in the sections of time codes TC3 and TC4 of image data V1 to V16.

[0079] Step S5: Thumbnail display The free viewpoint image server 2 displays thumbnails of the image data V1 to V16 in the section of time codes TC3 and TC4 transferred to the NAS 5. Step S6: Scene check The operator OP2 checks the scene contents of the section indicated by the time codes TC3 and TC4 on the generation operation screen Gs by the free viewpoint image server 2. Step S7: Camera path selection The operator OP2 selects (specifies) a camera path that he / she considers appropriate on the generation operation screen Gs according to the contents of the scene. Step S8: Execute generation After selecting the camera path, operator OP2 performs an operation to generate and execute an FV clip.

[0080] Step S9: 3D information generation The free viewpoint image server 2 generates three-dimensional information such as 3D data and polygon mesh data of the subject described above using frame data for the time code sections TC3 and TC4 in each of the image data V1 to V16, and parameter data such as the placement position of each imaging device 10 that has been input in advance. The parameter data of each image capture device 10 referred to here is data that includes at least the external parameters and internal parameters of each image capture device 10, and information on the focal length. Step S10: Rendering The free viewpoint image server 2 generates a free viewpoint image based on the three-dimensional information and parameter data of each image capture device 10. At this time, the free viewpoint image is generated so that the viewpoint is moved based on the camera path selected in step S7.

[0081] Step S11: Transfer The free viewpoint image server 2 transfers the generated FV clip to the image creation controller 1. At this time, not only the FV clip but also accompanying information such as information specifying the previous clip and the next clip and information specifying the time lengths of the previous clip and the next clip can be transmitted. Step S12: Quality check On the free viewpoint image server 2 side, the operator OP2 can check the quality before or after the transfer in step S11. That is, the free viewpoint image server 2 plays back and displays the generated FV clip on the generation operation screen Gs so that the operator OP2 can check it. In some cases, the operator OP2 can also redo the generation of the FV clip without executing the transfer.

[0082] Step S13: Playlist generation The image creation controller 1 generates an output clip using the transmitted FV clip, by combining the FV clip with either or both of a preceding clip and a succeeding clip on the time axis. This output clip may be generated as stream data in which each frame of the previous clip, each virtually generated frame of the FV clip, and each frame of the next clip are actually linked in chronological order, but in this processing example, they are virtually linked as a playlist. That is, by generating a playlist so that the frame section as the previous clip is played, followed by the FV clip, and then the frame section as the next clip is played, it becomes possible to play the output clip without generating the actual linked stream data as the output clip.

[0083] Step S14: Quality check The GUI on the image creation controller 1 side performs playback based on the playlist, and the operator OP1 checks the contents of the output clip. Step S15: Playback instruction The operator OP1 issues a playback instruction by a predetermined operation in response to the quality check, and the image creation controller 1 recognizes the input of the playback instruction. Step S16: Playback In response to the playback instruction, the image creation controller 1 supplies the output clip to the switcher 6. This makes it possible to broadcast the output clip.

[0084] 6. Camera movement detection To generate a free viewpoint image, three-dimensional information of the subject is generated using image data V1, V2, . . . V16, and therefore parameters including position information of each image capture device 10 are important. For example, if the position of a certain imaging device 10 is moved during a broadcast or the imaging direction is changed in the panning direction, tilting direction, or the like, calibration of the parameters accordingly becomes necessary. For this reason, in the image processing system of Fig. 1, camera fluctuation detection is performed by the utility server 8. Here, camera fluctuation means a change in at least one of the camera position and imaging direction.

[0085] The processing procedures of the image creation controller 1 and the utility server 8 when detecting camera fluctuations will be explained using Fig. 14. Note that Fig. 14 shows the processing procedures in the same format as Fig. 13, but the utility server 8 is also operated by an operator OP2 as an example.

[0086] Step S30: HD output To detect camera fluctuations, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to output image data to the image conversion unit 7. The images from the video servers 4A, 4B, 4C, and 4D, i.e., the images from the 16 imaging devices 10, are resolution-converted by the image conversion unit 7 and supplied to the utility server 8.

[0087] Step S31: Background generation The utility server 8 generates background images based on the supplied images. Since background images are images that do not change unless there is a movement in the camera, background images excluding subjects such as players are generated for the 16 systems of image data (V1 to V16). Step S32: Check the difference The background image is displayed in the GUI, allowing operator OP2 to confirm changes in the image. Step S33: Automatic fluctuation detection By comparing the background images at each point in time, camera fluctuations can also be automatically detected.

[0088] Step S34: Camera movement detection As a result of step S33 or step S32, fluctuations in a certain imaging device 10 are detected. Step S35: Image acquisition Calibration becomes necessary in response to the detection of a change in the imaging device 10. Therefore, the utility server 8 requests the image creation controller 1 to provide image data in a state after the change. Step S36: Extract Clip In response to a request for image acquisition from the utility server 8, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to execute clip extraction for the image data V1 to V16. Step S37: Transfer to NAS The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data extracted as clips to the NAS 5.

[0089] Step S38: Feature point correction By transferring the image to the NAS 5, the utility server 8 can refer to and display the image in the state after the camera movement. The operator OP2 performs operations required for calibration, such as correcting feature points. Step S39: Recalibration The utility server 8 re-executes calibration for creating a 3D model using the image data (V1 to V16) in the state after the camera movement.

[0090] Step S40: Reacquire background After the calibration, in response to an operation by the operator OP2, the utility server 8 requests reacquisition of image data for the background image. Step S41: Extract Clip In response to a request for image acquisition from the utility server 8, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to execute clip extraction for the image data V1 to V16. Step S42: Transfer to NAS The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data extracted as clips to the NAS 5. Step S43: Background generation The utility server 8 generates a background image using the image data transferred to the NAS 5. This background image is used as a reference for subsequent camera movement detection, for example.

[0091] For example, by performing camera fluctuation detection and calibration according to the above procedure, even if the position or imaging direction of the imaging device 10 changes during broadcasting, the parameters are corrected accordingly, making it possible to continuously generate accurate FV clips.

[0092] <7. Data flow for generating free viewpoint images> With reference to FIG. 15, a data flow relating to free viewpoint image generation in this embodiment will be described. First, captured image data (image data V1 to V16 in this example) is obtained for each of the image capturing devices 10 arranged at each viewpoint. Here, the imaging device 10 used to generate a free viewpoint image may include an imaging device 10 for obtaining an image to be used to generate 3D data (hereinafter referred to as an "object sensing camera"), and an imaging device 10 for obtaining a texture image to be pasted onto the 3D data when generating a free viewpoint image (hereinafter referred to as a "texture camera"). For example, it is conceivable that some of the imaging devices 10 used to generate a free viewpoint image are object sensing cameras and the others are texture cameras. Alternatively, the object sensing cameras and the texture cameras do not necessarily need to be separate imaging devices 10, and one imaging device 10 can be used as both an object sensing camera and a texture camera. Furthermore, all imaging devices 10 can be used as such dual-purpose cameras.

[0093] When generating 3D data, a foreground extraction process P1 is performed using captured image data (hereinafter referred to as "sensing captured image data") obtained by each imaging device 10 as a camera for sensing an object, and silhouette image data is generated.

[0094] FIG. 16 is an explanatory diagram of silhouette image data. In the foreground extraction process P1, a background image such as the one shown in the middle of the figure is generated for each object sensing camera based on the sensing captured image data. In generating a free viewpoint image, the target object is a moving object such as a player, so the background image can be generated by, for example, extracting the difference between frames. By taking the difference between this background image and the sensing captured image data, a foreground image in which the image portion of the target object has been extracted can be obtained for each object sensing camera. Then, by generating image data for these foreground images, for example, with the image area of ​​the subject set to "1" and other areas set to "0", silhouette image data showing the silhouette of the subject can be obtained for each viewpoint of the subject sensing camera, as shown in the example at the bottom of the figure.

[0095] 15, in 3D data generation process P2, 3D data of the subject is generated by the volume intersection method using the silhouette image data for each viewpoint and the parameter data of each camera. As mentioned above, the parameter data is data that includes information on the external parameters, internal parameters, and focal length of the camera (subject sensing camera). Fig. 17 shows an example of an image of 3D data corresponding to the subject shown in Fig. 16. In other words, 3D data is data that indicates the area of ​​the subject in three-dimensional space.

[0096] Here, the 3D data is not generated individually for each target subject, such as each individual player. If multiple target subjects are captured within the field of view of the camera and the silhouette image data is data showing the silhouettes of those multiple subjects, a single 3D data showing a three-dimensional image of those multiple subjects is generated according to the silhouette image data.

[0097] In FIG. 15, the 3D data is used to generate a free viewpoint image by the first FV generating unit 32b using the VDP method. Specifically, the first FV generating unit 32b generates a free viewpoint image by the VDP method based on the 3D data, the image data captured by the texture camera, and the parameter data of the texture camera.

[0098] The 3D data is also used to generate 3D models that enable free viewpoint image generation using the VIDP method described above. Specifically, polygon mesh data as a 3D model of the subject is generated from the 3D data by 3D model generation processing P3 in the drawing. In this example, polygon mesh data is generated for each subject. For reference, FIG. 18 shows an example of polygon mesh data for a certain subject.

[0099] The VIDP method uses the UV map texture described above to generate a free viewpoint image. This UV map texture is generated based on image data captured by a texture camera in a texture generation process P4 shown in FIG. In the texture generation process P4 in this example, a UV map texture is generated for each object, corresponding to the 3D model generation process P3 generating polygon mesh data for each object.

[0100] The second FV generation unit 32c generates a free viewpoint image by the VIDP method based on the 3D model (polygon mesh data) of the subject obtained in the 3D model generation process P3 and the UV map texture obtained in the texture generation process P4.

[0101] In the above-mentioned data flow, the foreground extraction process P1 for obtaining silhouette image data, the 3D data generation process P2 for generating 3D data from the silhouette image data, the 3D model generation process P3 for generating polygon mesh data as a 3D model from the 3D data, and the texture generation process P4 for generating a UV map texture are executed by the CPU 71 of the free viewpoint image server 2 as the above-mentioned processed data generation unit 32a.

[0102] Here, the VDP method is a technique for pasting texture images prepared for each viewpoint, which has the advantage of being able to suppress degradation in image quality of free viewpoint images even when the 3D data to be pasted is rough. In contrast, the VIDP method has the advantage of not requiring texture images to be prepared for each viewpoint, but if the polygon mesh is coarse, that coarseness will be directly reflected in the image quality of the free viewpoint image.

[0103] 8. Storage data selection method as an embodiment As can be understood from the above description, generating a free viewpoint image requires image data captured by a large number of imaging devices 10. However, storing the image data captured by all of the imaging devices 10 for generating a free viewpoint image would require a huge memory capacity.

[0104] As mentioned above, free viewpoint images can be generated in two ways: for example, to obtain images distributed during broadcast, such as replay images, or after broadcast, based on recorded data. In particular, when free viewpoint images are generated after broadcast, as in the latter case, it is necessary to store captured image data for all scenes during the event in order to be able to generate free viewpoint images for any scene during the event, which results in an enormous amount of stored data.

[0105] Therefore, in this embodiment, a method is adopted in which data to be used for generating a free viewpoint image is selected according to the importance of at least one of the event or the viewpoint. Specifically, multiple captured image data sets obtained by capturing an event from multiple viewpoints and processed data obtained by performing processing related to generating at least three-dimensional information of a subject on the captured image data are used as selection target data, and data to be used for generating a free viewpoint image is selected according to the importance of at least one of the event or the viewpoint. In this example, the selection of data to be used for generating such a free viewpoint image is performed by the selection processing unit 24 shown in FIG.

[0106] The importance here may be the importance of the scenes that make up the event. As a specific example, predetermined specific scenes such as shooting scenes, goal scenes, foul scenes, home run scenes, and scenes in progress of play (periods excluding when play is paused) may be detected as scenes of high importance. In this case, it is considered that the detection of a specific scene as a scene of high importance is performed by image analysis of image data captured by the imaging device 10. For example, a method using AI (artificial intelligence) trained to determine whether or not a scene is a specific scene, image analysis by template matching, etc. can be used. Alternatively, when audio data synchronized with the captured image data is recorded by a microphone, the detection of a specific scene may be performed by audio analysis of the audio data. For example, a scene in which a specific audio associated with the specific scene is detected may be detected as the specific scene. The specific scene may be detected using both image analysis and audio analysis.

[0107] Furthermore, when the target event is a professional sports match such as soccer, baseball, or American football, the detection of a specific scene can be performed based on information distributed from a site that distributes stats information for the match. This stats information includes information specifying the type of play, such as a shot, goal, or home run, as well as the time when the play occurred, making it possible to detect a specific scene.

[0108] Furthermore, the detection of a specific scene may be performed based on information posted to a social networking service (SNS) where information about the target event is posted. For example, if the specific scene is a home run scene, a time period during which there are many posts containing keywords associated with the specific scene, such as "XXX player hits a home run!", may be detected as the time period of the specific scene.

[0109] Here, the specific scene as a high-importance scene is not limited to a predetermined scene. For example, a scene in which the audience is excited can be considered to be a high-importance scene. This exciting scene can be detected, for example, based on image analysis or audio analysis of image data captured by the imaging device 10, or information posted to SNS. In the case of image analysis, for example, an exciting scene can be detected based on the movements of the audience by image analysis of the audience seats. In the case of audio analysis, an exciting scene can be detected based on the volume of cheers from the audience. In addition, as a detection based on information posted to SNS, for example, a time period with a large number of posts (for example, a time period with a predetermined number of posts per unit time or more) can be detected as a time period of an exciting scene.

[0110] The selection processing unit 24 in this example selects data to be used for generating a free viewpoint image based on the importance of the scene as described above. In this case, the data is selected by selecting the captured image data (image data V in this example) of the imaging device 10 and the processed data obtained by performing processing related to generating at least three-dimensional information of the subject on the captured image data. Specifically, the selection processing unit 24 selects, depending on the importance, whether to use the captured image data as data to be used for generating a free viewpoint image or the processed data as data to be used for generating a free viewpoint image. The processing data here refers to processing data generated by the processing data generation unit 32a described above, and specifically includes, in this example, at least one of silhouette image data, 3D data (using volume intersection method), 3D model (polygon mesh data), and UV map texture. The processing data may also include a texture image to be applied to the 3D data. In this example, the texture image is generated by the first FV generation unit 32b, but the texture image may also be generated by the processing data generation unit 32a.

[0111] As described above, by selecting whether to use captured image data or processed data for generating free viewpoint images depending on the importance, it is possible to select captured image data when the importance is high and processed data when the importance is low. When processed data is selected as the data used to generate a free-viewpoint image, there is a risk that the degree of freedom in generating the free-viewpoint image and the image quality may be limited compared to when captured image data is selected. For example, when silhouette image data of only some viewpoints is selected as the processed data, there is a risk that the degree of freedom in the camera path may be limited when generating the free-viewpoint image after broadcasting. Furthermore, when a 3D model and a UV map texture are selected as the processed data, it is possible to generate a free-viewpoint image in CG using the VIDP method, but there is a risk that the image quality may be inferior to that of the VDP method. Therefore, by being able to select captured image data when the importance is high and processed data when the importance is low as described above, it is possible to prevent limitations on the degree of freedom in generating the free-viewpoint image when the importance is high, and to select processed data with a smaller amount of stored data when the importance is low, thereby reducing the amount of stored data. In other words, it is possible to reduce the amount of stored data while preventing the degree of freedom in generating the free-viewpoint image from being impaired when the importance is high.

[0112] Regarding the data to be selected for use in generating a free viewpoint image, in this example, the image data captured by the imaging device 10 is recorded in the video server 4. On the other hand, the processed data is generated by the processed data generation unit 32a included in the free viewpoint image server 2 in this example, and is therefore recorded in the NAS 5. In this example, the selection processing unit 24 performs a process of generating management information for data selected from the selection target data according to importance as data to be used for generating a free viewpoint image so that the data is retained in the recorded state on the recording medium on which the data is recorded. This allows for an appropriate reduction in the amount of stored data according to importance, in accordance with specifications for generating free viewpoint images using captured image data and processed data that are kept in a recorded state on one or more recording media (e.g., the recording media of the video server 4 or the NAS 5).

[0113] In the above, it is assumed that the processed data is recorded in the NAS 5, but it is also possible to consider a specification in which the processed data is recorded in the video server 3 or 4 instead of the NAS 5. In that case, the data to be selected in free viewpoint image generation (captured image data and processed data) will be recorded in the video server 3 or 4. On the other hand, the data used in free viewpoint image generation should be recorded in the NAS 5 used by the free viewpoint image server 2. In this case, the selection processing unit 24 performs a process in which the data selected from the selection target data according to its importance as data to be used for generating a free viewpoint image is output from one or more recording media (here, either or both of the video servers 3 and 4) on which the data is recorded to another recording medium (NAS5 in this example). This allows the data used for generating free viewpoint images, among the captured image data and processed data recorded on one or more recording media, to be stored on a separate recording medium, thereby enabling an appropriate reduction in the amount of stored data according to the level of importance in accordance with the specifications for generating free viewpoint images. Specifically, in this case, it is possible to reduce the amount of stored data on the separate recording media.

[0114] Here, the importance of a scene can be divided into three or more values. For example, in the case of shooting scenes, shooting scenes in which no goal was scored could be rated as medium importance, shooting scenes in which a goal was scored could be rated as high importance, and scenes other than shooting scenes could be rated as low importance. Alternatively, it is possible to separately detect the level of excitement among the audience for specific scenes such as shooting scenes, goal scenes, foul scenes, and home run scenes, and classify the importance as low, medium, or high depending on the level of excitement among the audience. For example, for goal scenes and foul scenes, if the audience's excitement level is low, the importance level may be medium; if the audience's excitement level is high, the importance level may be high; and scenes other than goal scenes and foul scenes may be low.

[0115] The importance level can also be determined based on a user operation input, because the importance level here is basically determined based on whether or not the user wants to re-create a free viewpoint image for the scene after broadcasting, etc. In this case, the selection processing unit 24 accepts an input specifying the importance from the user during broadcasting (while the captured image data is being recorded). Then, in response to the input specifying the importance, the selection processing unit 24 determines an image section corresponding to the timing at which the input specifying the importance was made (for example, an image section of a predetermined length including that timing) as an image section of the target scene, determines the captured image data and processed data in that image section as selection target data for free viewpoint image generation, and selects data to be used for free viewpoint image generation from the selection target data according to the importance.

[0116] Note that the determination of the importance of a scene is not limited to real time. For example, a scene showing a player selected for an award such as the Most Valuable Player after the broadcast can be determined to be an important scene after the fact.

[0117] Here, the importance is not limited to the importance related to an event, such as the importance of a scene, but may also include the importance related to a viewpoint. Specifically, the importance of each viewpoint may be based on the purpose of the camera (image capture device 10) arranged at each viewpoint, or on the subject being imaged from the viewpoint.

[0118] Examples of camera uses include the aforementioned object sensing camera and texture camera. In this case, the use of each camera can be determined based on use identification information (information indicating whether the camera is for object sensing or texture) that is assigned to each camera in advance. Alternatively, cameras with different characteristics may be used as the object sensing camera and the texture camera, in which case it may be possible to determine the purpose of each camera based on the characteristics of those cameras. As an example, a camera with a wide angle of view equipped with an IR (infrared) image sensor may be used as the object sensing camera, and a camera with a close angle of view equipped with an RGB image sensor may be used as the texture camera, in which case the purpose of each camera can be determined from information on the type of image sensor and the angle of view.

[0119] An example of the importance based on the image capture target from the viewpoint is the importance in terms of whether or not a target event is included in the field of view. Examples of noteworthy events here include, for example, when the event is a ball game such as basketball, soccer, or American football, scenes of a player holding the ball, shooting scenes, goal scenes, foul scenes, etc. As with the determination of specific scenes, the determination of whether a particular event is included in the field of view can be performed based on image analysis of the captured image data. A camera that is determined to include a particular event in its field of view is determined to be an important camera. Alternatively, when an event of interest occurs on the home court side, the camera capturing the home court side may be determined to be an important camera, and the camera capturing the away court side may be determined to be a non-important camera.

[0120] Here, the importance of a camera determined based on whether or not the event of interest is included in the field of view can be determined not only based on whether or not the event of interest is included in the field of view, but also based on the size of the event of interest that is captured in the field of view. Specifically, if the image area size of the part of the field of view where the event of interest is occurring (for example, the size of the image area where the target subject is captured) is equal to or smaller than a predetermined size, the camera may not be determined to be important. Also, it may be possible to determine that the larger the image area size of the part of the field of view where the event of interest is occurring, the more important the camera.

[0121] The importance of a camera may also be determined based on whether or not the camera is desired to be included in a camera path later. For example, in a scene where Team A is on the offensive, a camera capturing the route taken by Team A's ball-carrying player from the front within Team B's court may be determined to be an important camera.

[0122] Furthermore, the importance of a camera may be determined based on whether or not it includes a specific subject. For example, a camera capturing a specific subject such as a star overseas player may be determined to be an important camera.

[0123] The following examples are possible examples of data selection according to the importance of the camera. Specifically, when a texture camera is set to be more important than an object sensing camera in terms of importance based on camera use, for some cameras determined to be texture cameras, the captured image data is selected from the selection target data as data to be used for generating a free viewpoint image. When generating a free-viewpoint image using three-dimensional data, it is desirable to have textures from as many viewpoints as possible to apply to the three-dimensional data in order to improve the quality of the free-viewpoint image. That is, for the viewpoint of the texture camera, it is desirable to select captured image data as the data to be used for generating the free-viewpoint image. On the other hand, for the object-sensing camera (camera for generating three-dimensional data), if three-dimensional data has already been generated, there is no need to use the captured image data for generating the free-viewpoint image. For example, in terms of the relationship between the texture camera and the object-sensing camera, it is conceivable to treat the texture camera as having a higher importance and the three-dimensional data generation camera as having a lower importance. As described above, it is possible to select the captured image data as data to be used for generating free viewpoint images only for texture cameras, which are of high importance, and not to select the captured image data as data to be used for generating free viewpoint images for object sensing cameras, which are of low importance.

[0124] At this time, for an object sensing camera that is considered to be of low importance, it is conceivable to select processing data from among the selection target data. This makes it possible to select captured image data for some viewpoints that are considered important among the multiple viewpoints, and processed data for other viewpoints that are considered unimportant, as data to be used in generating a free viewpoint image. Therefore, it is possible to appropriately reduce the amount of stored data according to the importance of the viewpoint.

[0125] In addition, when selecting data according to importance, a selection is made based on the importance of the event, such as the importance of the scene, as to whether the captured image data will be used as data to generate a free viewpoint image, or whether the processed data will be used as data to generate a free viewpoint image.Furthermore, if a selection is made to use the captured image data as data to generate a free viewpoint image, a selection can also be made based on the importance of the viewpoint as to which viewpoint the captured image data will be used as data to generate a free viewpoint image. Specifically, for important scenes during the event being imaged, only the captured image data of a camera determined to be important based on the importance of the viewpoint (e.g., a texture camera) is selected as data to be used for generating a free viewpoint image, and the captured image data of other cameras (subject sensing cameras) is not selected as data to be used for generating a free viewpoint image.

[0126] This makes it possible to prevent data that is deemed unimportant from being saved for generating free viewpoint images, depending on the importance of the event or viewpoint, thereby reducing the amount of data that needs to be saved for generating free viewpoint images.

[0127] <9. Processing Procedure> An example of a processing procedure for realizing a method for selecting data to be saved according to an embodiment will be described with reference to the flowcharts of FIGS. In this example, the processes shown in FIGS. 19 and 20 are executed by the CPU 71 in the image creation controller 1 based on a program stored in the storage unit 79, for example.

[0128] Here, three examples, a first example to a third example, will be given as examples of processing procedures for realizing the save data selection method according to the embodiment. The first example corresponds to a case where a viewpoint is constantly specified during broadcasting, and free-viewpoint images are generated according to the specified viewpoint. That is, during broadcasting, free-viewpoint images are generated for the entire period of an event, such as a match, and data for generating free-viewpoint images is saved for only a part of the event for new free-viewpoint image generation after the broadcast (post-event free-viewpoint image generation). In this case, for the purpose of generating a free viewpoint image after the fact, during broadcasting, for example, an operator OP1 specifies a range of data to be saved. Then, in the processing as a first example described below, data within this range to be saved (captured image data and processed data) is used as selection target data, and data selection is performed according to the importance of the scene.

[0129] An example of a processing procedure corresponding to the first example will be described with reference to FIG. The process shown in this figure is executed during the broadcast of the event. First, in step S101, the CPU 71 waits for a storage range to be designated, that is, a data storage range for subsequent generation of a free viewpoint image to be designated by an operator OP1 or the like.

[0130] If it is determined in step S101 that a storage range has been specified, the CPU 71 proceeds to step S102 and calculates scene importance for the storage range. Here, the scene importance is determined to be one of three values: low, medium, or high. Note that the method for determining whether a scene is low, medium, or high in importance has already been explained, so a duplicate explanation will be avoided.

[0131] In step S103 following step S102, the CPU 71 determines whether the importance level is low. If the importance level is low in step S103, the CPU 71 proceeds to step S105, performs processing for saving the 3D model and UV map texture, and ends the series of processing shown in FIG. Here, the "processing for saving" referred to here can be either a process to keep the data in its recorded state, as exemplified above, or a process to output the data to another recording medium.

[0132] As described above, by saving 3D models and UV map textures for scenes of low importance, it is possible to reduce the amount of data stored for generating free viewpoint images while enabling subsequent generation of free viewpoint images.

[0133] If it is determined in step S103 that the importance is not low, the CPU 71 proceeds to step S104 to determine whether the importance is medium. If it is determined that the importance is medium, the CPU 71 proceeds to step S106 and performs processing to save the 3D data and texture. The texture here refers to image data obtained by extracting an image portion of a target subject, such as a player, from image data captured by a texture camera. By performing the saving process in step S106, when the importance is medium, it is possible to generate a free viewpoint image with higher image quality as a post-event free viewpoint image than when the importance is low, while reducing the amount of saved data compared to when image data captured by each camera is saved.

[0134] After executing the save process in step S106, the CPU 71 ends the series of processes shown in FIG.

[0135] Furthermore, if it is determined in step S104 that the importance is not medium (i.e., the importance is high), the CPU 71 proceeds to step S107 and calculates the camera importance. Here, for example, the importance is calculated based on the use of the camera, and the importance of the texture camera is calculated as high and the importance of the object sensing camera is calculated as low. In other words, the texture camera is determined to be an important camera (high importance camera), and the object sensing camera is determined to be an unimportant camera (low importance camera).

[0136] In step S108 following step S107, the CPU 71 performs processing for saving the captured image data of the important camera and the silhouette image data of the non-important camera. As a result, when the importance is high, it is possible to generate a free viewpoint image with a higher degree of freedom than when the importance is medium, as a post-event free viewpoint image generation.

[0137] After executing the saving process in step S108, the CPU 71 ends the series of processes shown in FIG.

[0138] Regarding the saving process in step S106, in generating a free viewpoint image using 3D data and texture, the above-mentioned parameter data (at least the parameter data of each texture camera) is necessary to apply an appropriate texture for each viewpoint. Therefore, the saving process in step S106 actually involves saving the parameter data of each texture camera together with the 3D data and texture.

[0139] The same applies to the saving process in step S108, where processing is performed to save the parameter data of each texture camera along with the captured image data of the important camera (texture camera) and the silhouette image data of the non-important camera (object sensing camera).

[0140] Next, a second example will be described. The second example does not assume that free viewpoint images are generated constantly during broadcasting as in the first example, but rather that free viewpoint images are generated only during broadcasting in the target section specified by an operator OP1 or the like. In this case, new free viewpoint image generation after broadcasting is performed for the section designated by the operator OP1 or the like during broadcasting as the target section for free viewpoint image generation. That is, in this case, the range to be saved in step S101 is the range designated as the target section for generating free viewpoint images during broadcasting, which is different from the first example described above. The processing procedure example corresponding to the second example is similar to that of the first example shown in FIG. 19, so a duplicated explanation will be avoided.

[0141] The third example differs from the first and second examples in the data selection method for scenes with high importance. An example of a processing procedure corresponding to the third example will be described with reference to FIG. First, in this case, if it is assumed that free viewpoint images are constantly generated during broadcasting as in the first example, the storage range in step S101 is designated as the data storage range for subsequent free viewpoint image generation, and if it is assumed that free viewpoint images are generated during broadcasting only for the target section specified by operator OP1 or the like during broadcasting as in the second example, the storage range in step S101 is designated as the target section for free viewpoint image generation during broadcasting.

[0142] The difference from the process shown in FIG. 19 is that the process of calculating the camera importance in step S107 is omitted, and the process of step S201 is executed instead of the process of step S108. In step S201, the CPU 71 performs processing to save the captured image data of all the cameras. That is, in this case, for a scene that is determined to have high importance, the camera importance is not calculated, and the captured image data of all the cameras is saved. Even with this type of processing, if the importance is high, it is possible to generate a free viewpoint image with a higher degree of freedom than when the importance is medium, as a post-event free viewpoint image generation.

[0143] <10. Variations> The embodiment is not limited to the specific example described above, and various modified configurations can be adopted. For example, in the above example, the image creation controller 1 executes the selection of data to be saved according to the embodiment, but it is also possible to configure the selection to be executed by another information processing device, such as the free viewpoint image server 2. Furthermore, with regard to the configuration of an image processing system as an embodiment of the present technology (in the example of Figure 1, the image creation controller 1, the free viewpoint image server 2, the video servers 3 and 4, the NAS 5, the switcher 6, the image conversion unit 7, the utility server 8, and the imaging device 10), it is also possible to provide some or all of the components, excluding the imaging device 10 and the operation units (corresponding to the remote controller units) of each device, on a cloud that is available via a network.

[0144] Furthermore, although the above example shows a case where the event for which free viewpoint images are generated is a sports match, the present technology can also be suitably applied to generating free viewpoint images for other events, such as live music shows, musicals, variety shows, and other television programs. For example, in the case of a live music concert, scenes in which the subject is singing, the chorus of the song, and scenes in which back-up dancers are dancing may be detected as important scenes.

[0145] Furthermore, the amount of stored data can be reduced according to the level of importance by converting the resolution or frame rate of at least one of the captured image data and the processed data according to the level of importance. Specifically, the lower the level of importance, the lower the resolution or frame rate can be. This makes it possible to reduce the amount of data stored for generating free viewpoint images.

[0146] <11. Summary of embodiments> As described above, the information processing device (image creation controller 1) of the embodiment is equipped with a selection processing unit (24) that selects data to be used for generating a free viewpoint image based on the importance of at least one of the event or the viewpoint, using multiple captured image data obtained by capturing an event from multiple viewpoints and processed data obtained by performing processing on the captured image data related to generating at least three-dimensional information of the subject as selection target data. This makes it possible to, for example, store captured image data for only important scenes among multiple scenes constituting an event for free viewpoint image generation, or store captured image data for only important viewpoints among multiple viewpoints for free viewpoint image generation, or to store captured image data for important scenes for free viewpoint image generation, while storing processed data rather than captured image data for unimportant scenes for free viewpoint image generation. Therefore, depending on the importance of the event or viewpoint, it is possible to prevent data that is deemed unimportant from being saved for generating free viewpoint images, thereby reducing the amount of data that needs to be saved for generating free viewpoint images.

[0147] In the information processing apparatus according to the embodiment, the importance includes the importance of the scenes that make up the event. This makes it possible to appropriately select the data to be used for generating free viewpoint images according to the importance of the scene, for example, by selecting captured image data for important scenes among the multiple scenes that make up an event, and processed data for unimportant scenes. Therefore, it is possible to appropriately reduce the amount of data to be saved according to the importance of the scene.

[0148] Furthermore, in the information processing apparatus according to the embodiment, the importance includes the importance for each viewpoint. This makes it possible to appropriately select the data to be used for generating a free viewpoint image according to the importance of the viewpoint, such as selecting captured image data for important viewpoints among multiple viewpoints and processed data for unimportant viewpoints as the data to be used for generating a free viewpoint image. Therefore, it is possible to appropriately reduce the amount of stored data according to the importance of the viewpoint.

[0149] Furthermore, in the information processing apparatus of the embodiment, the importance of each viewpoint is based on the purpose of the camera arranged at each viewpoint. As cameras for generating free viewpoint images, it is expected that cameras with different purposes will be used, such as a camera for generating three-dimensional data to obtain images for generating three-dimensional data of a subject, and a texture camera for obtaining texture images to be pasted onto the three-dimensional data. When generating a free-viewpoint image using three-dimensional data, it is desirable to have textures from more viewpoints as textures to be applied to the three-dimensional data in order to improve the quality of the free-viewpoint image. That is, for the viewpoint of the texture camera, it is desirable to select captured image data as data to be used for generating the free-viewpoint image. On the other hand, for the camera for generating three-dimensional data, if three-dimensional data has already been generated, it can be said that there is no need to use the captured image data for generating the free-viewpoint image. For example, in terms of the relationship between the texture camera and the camera for generating three-dimensional data, it is possible to treat the camera for generating texture as having a higher importance and the camera for generating three-dimensional data as having a lower importance. For example, the captured image data of only the camera for generating texture with a higher importance is selected as data to be used for generating the free-viewpoint image, and the captured image data of the camera for generating three-dimensional data with a lower importance is not selected as data to be used for generating the free-viewpoint image. In this way, the data to be used for generating the free-viewpoint image can be appropriately selected according to the purpose of the camera. Therefore, it is possible to appropriately reduce the amount of stored data according to the importance determined by the use of the camera.

[0150] In the information processing apparatus according to the embodiment, the importance for each viewpoint is the importance based on the image capture target from the viewpoint. This makes it possible to select the data to be used for generating free viewpoint images based on the importance of the viewpoint, such as whether or not it captures the subject of the image as a noteworthy event within the field of view (for example, a player with the ball moving forward). Therefore, it becomes possible to select the data to be used in generating free viewpoint images so that the amount of stored data is reduced for viewpoints other than the viewpoint capturing the imaging subject that is considered important, and it is possible to appropriately reduce the amount of stored data depending on the importance of the imaging subject from the viewpoint.

[0151] Furthermore, in the information processing apparatus of the embodiment, the selection processing unit selects data according to importance based on image analysis of the captured image data. By analyzing the captured image data, it is possible to grasp the captured content, for example, whether or not a subject of high importance has been captured. Therefore, data can be appropriately selected according to the importance of the event and the importance of the viewpoint, and the amount of data stored for generating a free viewpoint image can be appropriately reduced.

[0152] Furthermore, in the information processing apparatus according to the embodiment, the selection processing unit determines the importance level based on a user operation input. This makes it possible to appropriately reduce the amount of data stored for generating free viewpoint images in accordance with the importance determined by the user.

[0153] In the information processing apparatus of the embodiment, the processing data includes silhouette image data of the subject. While the silhouette image data has a smaller data volume than the captured image data, it is possible to generate a free viewpoint image using three-dimensional data generated from the silhouette image data. Therefore, when silhouette image data is selected as data to be used for generating a free viewpoint image, it is possible to reduce the amount of data to be stored while enabling generation of a free viewpoint image.

[0154] Furthermore, in the information processing apparatus of the embodiment, the processed data includes three-dimensional data of the subject generated from the captured image data of the plurality of viewpoints by the volume intersection method. While the amount of three-dimensional data is smaller than the total amount of captured image data for each viewpoint, the use of three-dimensional data makes it possible to generate a free viewpoint image. Therefore, when three-dimensional data is selected as data to be used for generating a free viewpoint image, it is possible to reduce the amount of data to be stored while enabling the generation of a free viewpoint image.

[0155] Furthermore, in the information processing apparatus of the embodiment, the processed data includes polygon mesh data of the subject generated from image data captured from a plurality of viewpoints. While the amount of polygon mesh data is smaller than the total amount of captured image data for each viewpoint, the use of polygon mesh data makes it possible to generate a free viewpoint image. Therefore, when polygon mesh data is selected as data to be used for generating a free viewpoint image, it is possible to reduce the amount of data to be stored while still enabling the generation of a free viewpoint image.

[0156] In addition, in the information processing device of the embodiment, the selection processing unit selects, depending on the importance, whether to use the captured image data as data to generate a free viewpoint image or to use the processed data as data to generate a free viewpoint image. This makes it possible to select captured image data when the importance is high, and process data when the importance is low. When processed data is selected as data to be used for generating a free-viewpoint image, there is a risk that the degree of freedom in generating the free-viewpoint image and the image quality may be restricted compared to when captured image data is selected. Therefore, by being able to select captured image data when the importance is high and processed data when the importance is low as described above, it is possible to prevent restrictions on the degree of freedom in generating the free-viewpoint image when the importance is high, and to select processed data with a smaller amount of stored data when the importance is low, thereby reducing the amount of stored data. In other words, it is possible to reduce the amount of stored data while preventing the degree of freedom in generating the free-viewpoint image from being impaired when the importance is high.

[0157] Furthermore, in the information processing device of the embodiment, the importance includes the importance related to the viewpoint, and the selection processing unit selects the captured image data of some viewpoints as data to be used for generating a free viewpoint image according to the importance related to the viewpoint. This makes it possible to select captured image data for some viewpoints that are considered important among the multiple viewpoints as data to be used for generating a free viewpoint image. Therefore, it is possible to appropriately reduce the amount of stored data according to the importance of the viewpoint.

[0158] Furthermore, in the information processing device according to the embodiment, the selection processing unit selects the processing data for the viewpoints other than the part as data to be used for generating a free viewpoint image. This makes it possible to select captured image data for some viewpoints that are considered important among the multiple viewpoints, and processed data for other viewpoints that are considered unimportant, as data to be used in generating a free viewpoint image. Therefore, it is possible to appropriately reduce the amount of stored data according to the importance of the viewpoint. In addition, since processed data is used to generate free viewpoint images for viewpoints other than the important viewpoint, it is possible to improve the quality of the free viewpoint images compared to when only image data captured from the important viewpoint is used.

[0159] In addition, in the information processing device of the embodiment, the processing data includes polygon mesh data of the subject generated from image data captured from multiple viewpoints, and the selection processing unit selects the polygon mesh data as data to be used for generating the free viewpoint image according to its importance. For example, for a scene with low importance, polygon mesh data may be selected as data to be used for generating a free viewpoint image. This enables free-viewpoint image generation by CG (Computer Graphics) using polygon mesh data, while reducing the amount of data stored for generating the free-viewpoint image.

[0160] Furthermore, in the information processing device of the embodiment, the selection processing unit selects whether to use the captured image data as data to generate a free viewpoint image or to use the processed data as data to generate a free viewpoint image based on the importance of the event, and if the captured image data is selected to be data to generate a free viewpoint image, the selection processing unit selects which viewpoint's captured image data to use as data to generate a free viewpoint image based on the importance of the viewpoint (see Figure 19). This makes it possible to select only image data captured from viewpoints that are considered important in scenes that are considered important during an event as data to be used for generating a free viewpoint image. Therefore, depending on the importance of the event or viewpoint, it is possible to prevent data that is deemed unimportant from being saved for generating free viewpoint images, thereby reducing the amount of data that needs to be saved for generating free viewpoint images.

[0161] Furthermore, in the information processing device of the embodiment, the selection processing unit generates management information so that data selected according to importance from the selection target data recorded on one or more recording media is retained in its recorded state on one or more recording media. This makes it possible to reduce the amount of stored data appropriately according to importance, in accordance with specifications for generating free viewpoint images using captured image data and processed data that are kept in a recorded state on one or more recording media.

[0162] In addition, in the information processing device of the embodiment, the selection processing unit performs a process of outputting data selected from the selection target data recorded on one or more recording media according to importance to another one or more recording media. This allows the data used for generating free viewpoint images, among the captured image data and processed data recorded on one or more recording media, to be stored on a separate recording medium, thereby enabling an appropriate reduction in the amount of stored data according to the level of importance in accordance with the specifications for generating free viewpoint images. Specifically, in this case, it is possible to reduce the amount of stored data on the separate recording media.

[0163] Furthermore, in the information processing apparatus of the embodiment, the selection processing unit converts the resolution or the frame rate of at least one of the captured image data and the processed data in accordance with the importance. This allows the resolution and frame rate of less important data to be lowered, thereby reducing the amount of data to be stored.

[0164] In addition, the information processing method of the embodiment is an information processing method in which an information processing device selects data to be used for generating a free viewpoint image based on the importance of at least one of the event or the viewpoint, using multiple captured image data obtained by capturing an event from multiple viewpoints and processed data obtained by performing processing on the captured image data related to generating at least three-dimensional information of the subject as selection target data. According to this information processing method, it is possible to obtain the same functions and effects as those of the information processing device of the above embodiment.

[0165] Here, as an embodiment, a program that causes a CPU, a DSP (Digital Signal Processor), or a device including these to execute the processing by the selection processing unit 24 described in Figures 19 and 20, etc., can be considered. In other words, the program of the embodiment is a program that can be read by a computer device, and enables the computer device to realize the function of selecting data to be used for generating a free viewpoint image based on the importance of at least one of the event or the viewpoint, using multiple captured image data obtained by capturing an event from multiple viewpoints and processed data obtained by performing processing on the captured image data related to generating at least three-dimensional information of the subject as selection target data. By using such a program, the function of the selection processing unit 24 described above can be realized in the device serving as the information processing device 70.

[0166] The above-mentioned program can be recorded in advance on a HDD as a recording medium built into a device such as a computer device, or on a ROM in a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. Such a program can be installed onto a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0167] Furthermore, such a program is suitable for widely providing the selection processing unit 24 of the embodiment. For example, by downloading the program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, the personal computer or the like can function as a device that realizes the processing of the selection processing unit 24 of the present disclosure.

[0168] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0169] <12. This Technology> The present technology can also be configured as follows. (1) a selection processing unit that selects data to be used for generating a free viewpoint image based on the importance of at least one of the event and the viewpoints, using a plurality of captured image data obtained by capturing an event from a plurality of viewpoints and processed data obtained by performing processing related to generating three-dimensional information of the subject on the captured image data as selection target data; Information processing device. (2) The importance includes the importance of the scenes that make up the event. The information processing device according to (1) above. (3) The importance includes the importance for each of the viewpoints. The information processing device according to (1) or (2). (4) The importance of each viewpoint is based on the purpose of the camera arranged at each viewpoint. The information processing device according to (3) above. (5) The importance for each viewpoint is the importance based on the object being imaged from the viewpoint. The information processing device according to (3) or (4). (6) The selection processing unit The selection of data according to the importance is performed based on image analysis of the captured image data. The information processing device according to any one of (1) to (5). (7) The selection processing unit The importance is determined based on a user operation input. The information processing device according to claim 1. The information processing device according to (1) above. (8) The processed data includes silhouette image data of a subject. The information processing device according to any one of (1) to (7). (9) The processed data includes three-dimensional data of the subject generated from the captured image data of the plurality of viewpoints by a volume intersection method. The information processing device according to any one of (1) to (8). (10) The processing data includes polygon mesh data of the subject generated from the captured image data of the plurality of viewpoints. The information processing device according to any one of (1) to (9). (11) The selection processing unit Depending on the importance, a selection is made as to whether the captured image data is to be used for generating a free viewpoint image or whether the processed data is to be used for generating a free viewpoint image. The information processing device according to any one of (1) to (10). (12) the importance includes an importance related to the viewpoint, The selection processing unit The captured image data of some of the viewpoints is selected as data to be used for generating a free viewpoint image according to the importance of the viewpoints. The information processing device according to any one of (1) to (11). (13) The selection processing unit For the viewpoints other than the part, the processed data is selected as data to be used for generating a free viewpoint image. The information processing device according to (12) above. (14) the processing data includes polygon mesh data of the subject generated from the captured image data of the plurality of viewpoints, The selection processing unit The polygon mesh data is selected as data to be used for generating a free viewpoint image according to the importance. The information processing device according to any one of (1) to (13). (15) The selection processing unit selecting whether to use the captured image data for generating a free viewpoint image or the processed data for generating a free viewpoint image based on the importance of the event; When the captured image data is selected as data to be used for generating a free viewpoint image, a selection is made as to which of the captured image data of the viewpoints is to be used as data to generate a free viewpoint image based on the importance of the viewpoints. The information processing device according to any one of (1) to (14). (16) The selection processing unit Generate management information so that the data selected according to the importance from the selection target data recorded on one or more recording media is maintained in the recorded state on the one or more recording media. The information processing device according to any one of (1) to (15). (17) The selection processing unit a process of outputting data selected according to the importance from the selection target data recorded on one or more recording media to another one or more recording media; The information processing device according to any one of (1) to (15). (18) The selection processing unit The resolution or the frame rate of at least one of the captured image data and the processed data is converted according to the importance. The information processing device according to any one of (1) to (17). (19) The information processing device A plurality of captured image data obtained by capturing an event from a plurality of viewpoints and processed data obtained by performing processing related to generating at least three-dimensional information of a subject on the captured image data are used as selection target data, and data to be used for generating a free viewpoint image is selected according to the importance of at least one of the event and the viewpoints. Information processing methods. (20) A computer readable program, The computer device realizes a function of selecting data to be used for generating a free viewpoint image, using a plurality of captured image data obtained by capturing an event from a plurality of viewpoints and processed data obtained by performing processing related to generating at least three-dimensional information of a subject on the captured image data as selection target data, in accordance with the importance of at least one of the event and the viewpoints. program. [Explanation of symbols]

[0170] 1 Image Creation Controller 2. Free viewpoint image server 3,4,4A,4B,4C,4D Video Server 5. NAS 6 Switcher 7 Image conversion section 8 Utility Server 10. Imaging device 24 Selection processing section 32a Processing data generation unit 32b First FV generation section 32c Second FV generation section Gs generation operation screen Gg pass creation screen 41 Scene Window 42 Scene list display area 43 Camera Path Window 44 Camera path list display area 45 Parameter display section 46 Send Window 51 Preset list display area 52 Camera path list display area 53 Camera Path Window 54 Operation panel 55 Preview Window 70 Information processing equipment 71 CPU 72 ROM 73 RAM 74 Bus 75 Input / Output Interface 76 Input section 77 Display section 78 Audio output section 79 Memory section 80 Communications Department 81 Removable Recording Media 82 Drive P1 Foreground extraction processing P2 3D data generation processing P3 3D model generation processing P4 Texture generation process

Claims

1. a selection processing unit that selects data to be used for generating a free viewpoint image from a plurality of captured image data obtained by capturing an event from a plurality of viewpoints and processed data obtained by performing processing related to generating at least three-dimensional information of a subject on the captured image data as selection target data in accordance with the importance of each of the viewpoints; The selection processing unit The captured image data of some of the viewpoints is selected as data to be used for generating a free viewpoint image according to the importance of each of the viewpoints. Information processing device.

2. The importance of each viewpoint is based on the purpose of the camera arranged at each viewpoint. The information processing device according to claim 1 .

3. The importance for each viewpoint is the importance based on the object being imaged from the viewpoint. The information processing device according to claim 1 .

4. The selection processing unit The selection of data according to the importance is performed based on image analysis of the captured image data. The information processing device according to claim 1 .

5. The selection processing unit The importance is determined based on a user operation input. The information processing device according to claim 1 .

6. The processed data includes silhouette image data of a subject. The information processing device according to claim 1 .

7. The processed data includes three-dimensional data of the subject generated from the captured image data of the plurality of viewpoints by a volume intersection method. The information processing device according to claim 1 .

8. The processing data includes polygon mesh data of the subject generated from the captured image data of the plurality of viewpoints. The information processing device according to claim 1 .

9. The selection processing unit Depending on the importance, a selection is made as to whether the captured image data is to be used for generating a free viewpoint image or whether the processed data is to be used for generating a free viewpoint image. The information processing device according to claim 1 .

10. The selection processing unit For the viewpoints other than the part, the processed data is selected as data to be used for generating a free viewpoint image. The information processing device according to claim 1 .

11. the processing data includes polygon mesh data of the subject generated from the captured image data of the plurality of viewpoints, The selection processing unit The polygon mesh data is selected as data to be used for generating a free viewpoint image according to the importance. The information processing device according to claim 1 .

12. The selection processing unit Generate management information so that the data selected according to the importance from the selection target data recorded on one or more recording media is maintained in the recorded state on the one or more recording media. The information processing device according to claim 1 .

13. The selection processing unit a process of outputting data selected according to the importance from the selection target data recorded on one or more recording media to another one or more recording media; The information processing device according to claim 1 .

14. The selection processing unit The resolution or the frame rate of at least one of the captured image data and the processed data is converted according to the importance. The information processing device according to claim 1 .

15. The information processing device a selection process is performed in which a plurality of captured image data obtained by capturing an event from a plurality of viewpoints and processed data obtained by performing processing related to generating at least three-dimensional information of a subject on the captured image data are used as selection target data, and data to be used for generating a free viewpoint image is selected according to the importance of each of the viewpoints; In the selection process, The captured image data of some of the viewpoints is selected as data to be used for generating a free viewpoint image according to the importance of each of the viewpoints. Information processing methods.

16. A computer readable program, a selection function that selects data to be used for generating a free viewpoint image according to the importance of each viewpoint, using a plurality of captured image data obtained by capturing an event from a plurality of viewpoints and processed data obtained by performing processing related to generating at least three-dimensional information of a subject on the captured image data as selection target data; The selection function selects the captured image data of some of the viewpoints as data to be used for generating a free viewpoint image according to the importance of each of the viewpoints. program.

Images