Information processing device, image processing system, and information processing method

Abstract

Provided is an image processing system comprising a first information processing device and a second information processing device. The first information processing device performs a process of identifying, from a plurality of captured images captured simultaneously by a plurality of imaging devices, a generation target image section from which a free-viewpoint image is to be generated, performs control to cause image data of the generation target image section in each of the plurality of captured images to be transmitted as image data for use in generating a free-viewpoint image in the second information processing device, and generates an output image including the received free-viewpoint image. The second information processing device performs a process of acquiring image data of the generation target image section in each of the plurality of captured images, a process of generating a free-viewpoint image using the acquired image data, and control to transmit the generated free-viewpoint image to the first information processing device.

Description

[Technical field]

[0001] The present technology relates to an information processing device, an image processing system, and an information processing method, and in particular to a technology for processing a free viewpoint image that allows an imaged subject to be observed from any viewpoint in a three-dimensional space. [Background technology]

[0002] A technology is known that generates a free viewpoint image (also called a free viewpoint video, virtual viewpoint image (video), etc.) that corresponds to an observation image from any viewpoint in three-dimensional space, based on three-dimensional information that represents an image of an imaged subject in three-dimensional space. As a related prior art, the following Patent Document 1 can be mentioned. Patent Document 1 discloses a technology related to the generation of camerawork that can be said to be a movement trajectory of a viewpoint. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2018 / 030206 publication Summary of the Invention [Problem to be solved by the invention]

[0004] Free viewpoint images are also useful as broadcast content, and are used, for example, as replay images in sports broadcasts. For example, in soccer or basketball broadcasts, a few-second clip of a shooting scene or the like is created from images recorded in real time and broadcast as a replay image. In this disclosure, a "clip" refers to an image of a certain scene that is created by cutting out or further processing a recorded image.

[0005] Meanwhile, at the broadcasting site, operators of live broadcasting are required to, for example, create and broadcast clips for replay extremely quickly.

[0006] Therefore, the present technology proposes a technology that enables faster production of images such as clips that include free viewpoint images. [Means for solving the problem]

[0007] The information processing device according to the present technology includes a section identification processing unit that performs processing to identify a target image section to be generated as a target for generating a free viewpoint image from a plurality of captured images captured simultaneously by a plurality of imaging devices, a target image transmission control unit that controls the transmission of image data of the target image section in each of the plurality of captured images as image data to be used for generating a free viewpoint image in another information processing device, and an output image generation unit that generates an output image including the received free viewpoint image. This information processing device is an information processing device that functions as a main controller for image creation such as clips in an image processing system. This information processing device specifies a target image section for one or more frames for generating a free viewpoint image, and transmits image data of only the target image section in a plurality of captured images to another information processing device (a free viewpoint image server that actually creates the free viewpoint image). Then, the created free viewpoint image is received and an output image is generated.

[0008] In the information processing device related to the above-mentioned technology, the section identification processing unit performs a process of notifying the other information processing device of information on the identified image section to be generated, and the target image transmission control unit performs control of the transmission of image data of the image section to be generated in response to a request from the other information processing device. By notifying the other information processing device of the image section to be generated, the other information processing device can specify image data required for generating a free viewpoint image.

[0009] In the information processing device according to the present technology described above, it is considered that the section identification processing unit identifies a section of one frame of a captured image as the generation target image section. This is processing for generating a free viewpoint image in a generation target image section of one frame, that is, a free viewpoint image in which the viewpoint is changed while the image is still with time stopped.

[0010] In the information processing device according to the present technology described above, it is considered that the section identification processing unit identifies a section of a plurality of frames of a captured image as the generation target image section. This is processing for generating free viewpoint images in a generation target image section of a plurality of frames, that is, free viewpoint images in which the viewpoint is changed while the time is moving without stopping.

[0011] In the information processing device according to the present technology described above, it is considered that the output image generating unit generates an output image by combining a front image or a back image with a free viewpoint image on a time axis. That is, when an information processing device receives a free viewpoint image created by another information processing device, the information processing device connects the previous and next images to generate an output image.

[0012] In the information processing device related to the above-mentioned present technology, it is considered that the output image generation unit generates the output image as a virtual clip in which a free viewpoint image and a previous image or a next image are virtually combined on a time axis using playback list information. In other words, when a free viewpoint image created by another information processing device is received, the previous and next images are connected to create an output image, but the image combination is performed virtually as a connection on the playback list information that indicates the order in which the images are to be played back.

[0013] In the information processing device related to the above-mentioned present technology, it is conceivable that the output image generation unit acquires information specifying a front image or a back image to be connected to a free viewpoint image from the other information processing device, and generates an output image in which the front image or the back image is combined with the free viewpoint image on the time axis based on the information. That is, an information processing device receives a free viewpoint image created by another information processing device, and also receives information specifying images before and after the free viewpoint image to be connected from the other information processing device.

[0014] In the information processing device related to the above-mentioned present technology, the output image generation unit may acquire information specifying the time length of the front image or back image to be connected to the free viewpoint image from the other information processing device, prepare the front image or back image based on the information, and generate an output image in which the free viewpoint image is combined with the front image or back image on the time axis. That is, the information processing device receives a free viewpoint image created by another information processing device, and also receives information on the time lengths of images before and after the free viewpoint image to be connected from the other information processing device.

[0015] Another information processing device related to the present technology includes a target image acquisition unit that acquires image data of a target image section for generating a free viewpoint image in each of a plurality of captured images captured simultaneously by a plurality of imaging devices, an image generation processing unit that generates a free viewpoint image using the image data acquired by the target image acquisition unit, and a transmission control unit that controls transmission of the generated free viewpoint image to another information processing device. This information processing device is, for example, an information processing device that functions as a free viewpoint image server in an image processing system. In this information processing device, a plurality of captured images are acquired as a section of a scene for generating a free viewpoint image, and a free viewpoint image is generated based on the captured images, and the generated free viewpoint image is transmitted to the information processing device that functions as a main controller for image generation.

[0016] In the information processing device relating to the above-mentioned present technology, the target image acquisition unit may make a request to transmit multiple image data of the image section to be generated for each of the multiple captured images in response to notification of the image section to be generated. That is, by notifying the image section to be generated, the image section to be acquired is recognized and image data of a plurality of viewpoints for that section is requested to be transmitted.

[0017] In the information processing device according to the present technology described above, it is considered that the target image acquisition section acquires image data of one frame, which is set as the generation target image section, in each of a plurality of captured images. This is processing for generating a free viewpoint image in a generation target image section of one frame, that is, a free viewpoint image in which the viewpoint is changed while the image is still with time stopped.

[0018] In the information processing device according to the present technology described above, it is considered that the target image acquisition section acquires image data for a section of a plurality of frames in each of a plurality of captured images, the section being set as the generation target image section. This is processing for generating free viewpoint images in a generation target image section of a plurality of frames, that is, free viewpoint images in which the viewpoint is changed while the time is moving without stopping.

[0019] In the information processing device according to the present technology described above, it is considered that the image generation processing unit generates a free viewpoint image by using viewpoint movement trajectory information selected from pre-stored viewpoint movement trajectory information. A plurality of pieces of information indicating the trajectory of the viewpoint movement are prepared in advance, and can be selected by, for example, an operator's operation.

[0020] In the information processing device related to the above-mentioned present technology, it is considered that the transmission control unit performs control to transmit information specifying a front image or a back image to be connected to the generated free viewpoint image to the other information processing device. That is, the free viewpoint image is received by another information processing device functioning as a main controller for creating clips, and information on the images before and after being linked to the free viewpoint image is also transmitted so that the images can be used to generate output images in the other information processing device.

[0021] In the information processing device related to the above-mentioned present technology, it is considered that the transmission control unit controls the transmission of information specifying the time length of the front image or back image to be connected to the generated free viewpoint image to the other information processing device. That is, the free viewpoint image is received by another information processing device functioning as a main controller for creating clips, and information on the time lengths of the images before and after the free viewpoint image to be connected to the free viewpoint image is also transmitted so that the information can be used to generate output images in the other information processing device.

[0022] The information processing method related to the present technology is an information processing method in which an information processing device functioning as a main controller for image creation performs a section identification process for identifying a target image section to be generated as a target for generating a free viewpoint image for a plurality of captured images captured simultaneously by a plurality of imaging devices, a target image transmission control process for transmitting image data of the target image section in each of the plurality of captured images as image data to be used for generating a free viewpoint image in another information processing device, and an output image generation process for generating an output image including the received free viewpoint image. Another information processing method related to the present technology is an information processing method in which an information processing device functioning as a free viewpoint image server performs an image data acquisition process to acquire image data of a target image section for generating a free viewpoint image in each of a plurality of captured images captured simultaneously by a plurality of imaging devices, an image generation process to generate a free viewpoint image using the image data acquired in the image data acquisition process, and a transmission control process to control transmission of the generated free viewpoint image to another information processing device. In this way, each information processing device cooperates with each other to generate an output image including a free viewpoint image. [Brief description of the drawings]

[0023] [Figure 1] 1 is a block diagram of a system configuration according to an embodiment of the present technology; [Diagram 2] 1 is an explanatory diagram of an example of a camera arrangement for generating a free viewpoint image according to an embodiment; [Diagram 3] FIG. 1 is a block diagram of a hardware configuration of an information processing apparatus according to an embodiment. [Figure 4] FIG. 2 is an explanatory diagram of functions of an image creation controller according to an embodiment. [Diagram 5] FIG. 2 is an explanatory diagram of functions of a free viewpoint image server according to an embodiment. [Figure 6] 3A and 3B are explanatory diagrams of viewpoints in a free viewpoint image according to an embodiment. [Figure 7]11 is an explanatory diagram of a user interface screen on the free viewpoint image server side according to the embodiment. FIG. [Figure 8] FIG. 4 is an explanatory diagram of an output clip according to the embodiment. [Figure 9] 11 is an explanatory diagram of an output clip including a still image FV clip according to an embodiment. FIG. [Figure 10] 11 is an explanatory diagram of an output clip including a moving image FV clip according to an embodiment. FIG. [Figure 11] 11A and 11B are diagrams illustrating an example of an image of an output clip according to the embodiment. [Figure 12] FIG. 4 is an explanatory diagram of a procedure for creating a clip according to an embodiment; [Figure 13] 11 is a flowchart of a process of an image creation controller according to an embodiment. [Figure 14] 11 is a flowchart of a process of a free viewpoint image server according to an embodiment. [Figure 15] 11 is an explanatory diagram of a procedure for detecting camera movement according to an embodiment; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] The embodiments will be described below in the following order. <1. System configuration> <2. Configuration of Image Creation Controller and Free Viewpoint Image Server> <3. GUI> <4. Clips containing free viewpoint images> <5. Clip creation process> <6. Camera movement detection> <7. Summary and variations>

[0025] <1. System configuration> FIG. 1 shows an example of the configuration of an image processing system according to an embodiment of the present technology. The image processing system includes an image creation controller 1, a free viewpoint image server 2, a video server 3, multiple (e.g., four) video servers 4A, 4B, 4C, and 4D, a NAS (Network Attached Storage) 5, a switcher 6, an image conversion unit 7, a utility server 8, and multiple (e.g., 16) imaging devices 10. In the following description, the term "camera" refers to the imaging device 10. For example, the term "camera arrangement" refers to an arrangement of multiple imaging devices 10. Moreover, when the video servers 4A, 4B, 4C, and 4D are referred to collectively without distinction, they will be referred to as "video server 4." In this image processing system, based on captured images (e.g., image data V1 to V16) obtained from multiple imaging devices 10, free viewpoint images corresponding to images observed from any viewpoint in three-dimensional space can be generated, and an output clip including the free viewpoint images can be created.

[0026] In FIG. 1, the connection state of each part is indicated by a solid line, a dashed line, or a double line. The solid lines indicate connections for SDI (Serial Digital Interface), an interface standard for connecting broadcasting equipment such as cameras and switchers, and are compatible with 4K, for example. Image data is mainly sent and received between each device via SDI wiring.

[0027] The double lines indicate connections of communication standards for constructing a computer network, such as 10 Gigabit Ethernet. The image creation controller 1, free viewpoint image server 2, video server 3, 4A, 4B, 4C, 4D, NAS 5, and utility server 8 are connected by a computer network, enabling them to send and receive image data and various control signals to and from each other.

[0028] The dashed line between video servers 3 and 4 indicates that video servers 3 and 4 equipped with inter-server file sharing functionality are connected by, for example, a 10G network. This allows each video server between video server 3 and video servers 4A, 4B, 4C, and 4D to preview and transmit materials stored in other video servers. In other words, a system using multiple video servers is constructed, enabling efficient highlight editing and transmission.

[0029] Each imaging device 10 is configured as a digital camera device having an imaging element such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, and obtains captured images (image data V1 to V16) as digital data. In this example, each imaging device 10 obtains captured images as moving images.

[0030] In this example, each imaging device 10 captures an image of a game such as basketball or soccer being played, and each is disposed in a predetermined position and a predetermined orientation in the venue where the game is held. In this example, the number of imaging devices 10 is 16, but the number of imaging devices 10 may be at least two in order to enable the generation of a free viewpoint image. By increasing the number of imaging devices 10 and capturing images of the target subject from more angles, the accuracy of the three-dimensional reconstruction of the subject can be improved, and the image quality of the virtual viewpoint image can be improved.

[0031] Fig. 2 shows an example of the placement of imaging devices 10 around a basketball court. Let us assume that the circles represent imaging devices 10. This is an example of the camera placement when one wishes to focus on capturing images of the area near the goal on the left side of the drawing. Of course, the placement and number of cameras are just an example, and should be set according to the content and purpose of the filming or broadcasting.

[0032] The image creation controller 1 is configured by an information processing device. This image creation controller 1 can be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like. The image creation controller 1 controls and manages the operations of the video servers 3 and 4, and performs processing for creating clips. As an example, the image creation controller 1 is a device that can be operated by an operator OP1. The operator OP1, for example, selects clip contents and issues instructions for creation.

[0033] The free viewpoint image server 2 is configured as an information processing device that actually performs processing to create a free viewpoint image (an FV (Free View) clip to be described later) in response to an instruction from the image creation controller 1. This free viewpoint image server 2 can also be realized by using, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like. As an example, the free viewpoint image server 2 is a device that can be operated by an operator OP2. The operator OP2 performs, for example, a camerawork selection operation for creating an FV clip as a free viewpoint image.

[0034] The configurations and processes of the image creation controller 1 and the free viewpoint image server 2 will be described in detail later. In addition, although it is assumed that the operations are performed by operators OP1 and OP2, for example, the image creation controller 1 and the free viewpoint image server 2 may be arranged side by side and operated by a single operator.

[0035] Each of the video servers 3 and 4 is an image recording device, and includes a data recording unit such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), and a control unit that controls recording and playback of data in the data recording unit.

[0036] Each of the video servers 4A, 4B, 4C, and 4D can accept, for example, four input systems, and each of them simultaneously records images captured by the four imaging devices 10. For example, the video server 4A records image data V1, V2, V3, and V4. The video server 4B records image data V5, V6, V7, and V8. The video server 4C records image data V9, V10, V11, and V12. The video server 4D records image data V13, V14, V15, and V16. As a result, images captured by all 16 imaging devices 10 are recorded simultaneously. The video servers 4A, 4B, 4C, and 4D are assumed to perform continuous recording, for example, during a sports match to be broadcast.

[0037] The video server 3 is, for example, directly connected to the image creation controller 1, and is capable of, for example, two input systems and two output systems. Image data Vp and Vq are shown as the two input systems. As the image data Vp and Vq, it is possible to select images captured by any two of the image capture devices 10 (any two of the image data V1 to V16). Of course, images captured by other image capture devices may also be used.

[0038] The image data Vp, Vq can be displayed as a monitor image on a display by the image creation controller 1. The operator OP1 can check the status of a scene being shot and recorded for broadcast, for example, by the image data Vp, Vq input to the video server 3. In addition, since the video servers 3 and 4 are connected in a file sharing state, the image creation controller 1 can also display on the monitor the images captured by each imaging device 10 that are recorded in the video servers 4A, 4B, 4C, and 4D, allowing the operator OP1 to check them one by one.

[0039] In this example, a time code is added to the image captured by each imaging device 10, making it possible to achieve frame synchronization in the processing in the video servers 3, 4A, 4B, 4C, and 4D.

[0040] The NAS 5 is a storage device arranged on a network, and is composed of, for example, an SSD, an HDD, etc. In this example, the NAS 5 is a device that stores some frames of the image data V1, V2, ..., V16 recorded in the video servers 4A, 4B, 4C, and 4D for processing in the free viewpoint image server 2 when the frames are transferred for generating free viewpoint images, and stores the generated free viewpoint images.

[0041] The switcher 6 is a device that receives the images output via the video server 3 and selects the main line image PGMout to be finally selected and broadcast. For example, a broadcast director or the like performs the necessary operations.

[0042] The image conversion unit 7 performs, for example, resolution conversion and synthesis of image data from the imaging device 10, generates a monitoring image of the camera arrangement, and supplies it to the utility server 8. For example, 16-channel image data (V1 to V16) that is an 8K image is resolution-converted to a 4K image, and then the image is converted to four-channel images arranged in a tiled pattern, and supplied to the utility server 8.

[0043] The utility server 8 is a computer device capable of various related processes, but in this example, it is a device that performs a process of detecting camera movement for calibration in particular. For example, the utility server 8 detects camera movement by monitoring image data from the image conversion unit 7. Camera movement refers to movement of any of the positions of the imaging devices 10 arranged, for example, as shown in FIG. 2. Information on the position of the imaging devices 10 is an important element in generating a free viewpoint image, and if the position of the imaging devices 10 changes, it is necessary to reset the parameters. For this reason, the camera movement is monitored.

[0044] <2. Configuration of Image Creation Controller and Free Viewpoint Image Server> The image creation controller 1, the free viewpoint image server 2, the video servers 3 and 4, and the utility server 8 in the above configuration can be realized as an information processing device 70 having the configuration shown in FIG.

[0045] 3, a CPU 71 of an information processing device 70 executes various processes according to a program stored in a ROM 72 or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes as appropriate. The CPU 71, the ROM 72, and the RAM 73 are interconnected via a bus 74. To this bus 74, an input / output interface 75 is also connected.

[0046] The input / output interface 75 is connected to an input unit 76 including an operator and an operating device. For example, the input unit 76 may be various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71.

[0047] Further, the input / output interface 75 is connected, either integrally or separately, to a display unit 77 formed of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, or the like, and an audio output unit 78 formed of a speaker, or the like. The display unit 77 is a display unit that performs various displays, and is configured, for example, by a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, or the like. The display unit 77 displays images for various types of image processing, moving images to be processed, etc., on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface), based on instructions from the CPU 71.

[0048] The input / output interface 75 may be connected to a storage unit 79 configured with a hard disk or solid-state memory, or a communication unit 80 configured with a modem or the like. The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like.

[0049] A drive 82 is also connected to the input / output interface 75 as required, and a removable recording medium 81 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. The drive 82 allows data files such as image files MF and various computer programs to be read from the removable recording medium 81. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the sound output unit 78. Furthermore, the computer programs and the like read from the removable recording medium 81 are installed in the storage unit 79 as necessary.

[0050] In this information processing device 70, software can be installed via network communication by a communication unit 80 or via a removable recording medium 81. Alternatively, the software may be pre-stored in the ROM 72, the storage unit 79, or the like.

[0051] When the image creation controller 1 and the free viewpoint image server 2 are realized using such an information processing device 70, the processing functions as shown in FIGS. 4 and 5 are realized in the CPU 71 by software, for example.

[0052] FIG. 4 shows a section specification processing section 21, a target image transmission control section 22, and an output image generating section 23 as functions formed in a CPU 71 of an information processing device 70 serving as the image creation controller 1.

[0053] The section identification processing unit 21 performs a process of identifying a target image section for generating a free viewpoint image for a plurality of captured images (image data V1 to V16) captured simultaneously by a plurality of imaging devices 10. For example, in response to an operation by an operator OP1 to select a scene to be replayed within an image, the section identification processing unit 21 performs a process of identifying a time code for the scene, particularly for a section of the scene to be a free viewpoint image (target image section for generation), and notifying the free viewpoint image server 2 of the time code.

[0054] The image section to be generated here refers to the frame section that will actually be used as a free viewpoint image. When generating a free viewpoint image for a certain frame in a video, that frame becomes the image section to be generated. In this case, the in point / out point for the free viewpoint image will be the same time code. When generating a free viewpoint image for a section of multiple frames in a video, the multiple frames become the image section to be generated. In this case, the in point and out point for the free viewpoint image have different time codes. Although the structure of a clip will be described later, it is assumed that the in / out points of the image section to be generated will be different from the in / out points of the final output clip, because a previous clip and a next clip will be combined, as described later.

[0055] The target image transmission control unit 22 controls the transmission of image data of the image section to be generated in each of the multiple captured images, i.e., one or multiple frames of the image data V1 to V16, as image data used for generating a free viewpoint image in the free viewpoint image server 2. Specifically, the control unit 22 controls the transfer of the image data as the image section to be generated from the video servers 4A, 4B, 4C, and 4D to the NAS 5.

[0056] The output image generating unit 23 performs processing for generating an output image (output clip) including the free viewpoint image (FV clip) generated by the free viewpoint image server 2 and received. For example, the image creation controller 1 generates an output clip by combining, on the time axis, a previous clip, which is an actual moving image at a time point before the FV clip, which is a virtual image generated by the free viewpoint image server 2, and a subsequent clip, which is an actual moving image at a time point after the FV clip, through processing by the output image generating unit 23. In other words, the previous clip + FV clip + subsequent clip are treated as one output clip. Of course, the previous clip and the FV clip may be one output clip. Alternatively, the FV clip and the after clip may be one output clip. Furthermore, an output clip may be generated that is only an FV clip, without combining the preceding clip or the following clip. In either case, the image creation controller 1 generates an output clip including an FV clip and outputs it to the switcher 6 so that it can be used for broadcasting.

[0057] Next, FIG. 5 shows a target image acquisition unit 31, an image generation processing unit 32, and a transmission control unit 33 as functions formed in a CPU 71 of an information processing device 70 serving as the free viewpoint image server 2.

[0058] The target image acquisition unit 31 performs a process of acquiring image data of a target image section for generating a free viewpoint image in each of a plurality of captured images (image data V1 to V16) captured simultaneously by a plurality of imaging devices 10. That is, the image creation controller 1 acquires image data of one frame or multiple frames specified by the in-point / out-point of the target image section identified by the function of the section identification processing unit 21 from the video servers 4A, 4B, 4C, 4D via the NAS 5, so that the image data can be used for generating a free viewpoint image.

[0059] For example, the target image acquisition unit 31 acquires image data of one frame or multiple frames of the image section to be generated for all of the image data V1 to V16. The image data of the image section to be generated for all of the image data V1 to V16 is acquired in order to generate a high-quality free viewpoint image. As described above, it is possible to generate a free viewpoint image by using images captured by at least two or more imaging devices 10, but by increasing the number of imaging devices 10 (i.e., the number of viewpoints), it is possible to generate a more detailed 3D model and generate a high-quality free viewpoint image. Therefore, for example, when 16 imaging devices 10 are arranged, image data of the image section to be generated is acquired for all of the image data (V1 to V16) of the 16 imaging devices 10.

[0060] The image generation processing unit 32 has a function of generating a free viewpoint image, that is, an FV clip in this example, using the image data acquired by the target image acquisition unit 31. For example, the image generation processing unit 32 performs processes such as modeling processing including 3D model generation and subject analysis, and rendering for generating a free viewpoint image, which is a two-dimensional image, from the 3D model.

[0061] 3D model generation is a process of generating 3D model data that represents a subject in a three-dimensional space (i.e., that restores the three-dimensional structure of the subject from a two-dimensional image) based on images captured by each imaging device 10 and camera parameters for each imaging device 10 input from, for example, the utility server 8. Specifically, the 3D model data includes data that represents the subject in a three-dimensional coordinate system of (X, Y, Z). Based on 3D model data, subject analysis involves analyzing the position, orientation, and posture of the subject (player). Specifically, it estimates the subject's position, generates a simple model of the subject, and estimates the subject's orientation. A free viewpoint image is then generated based on the 3D model data and the subject analysis information. For example, a free viewpoint image is generated in which the viewpoint is moved for a 3D model of a stationary subject, a player.

[0062] The viewpoint of a free viewpoint image will be described with reference to FIG. 6A shows an image of a free viewpoint image in which a subject is captured from a required viewpoint set in a three-dimensional space. In this free viewpoint image, the subject HS1 is viewed substantially from the front, and the subject HS2 is viewed substantially from the back. Fig. 6B shows an image of a virtual viewpoint image when the viewpoint position is changed in the direction of arrow C in Fig. 6A and a viewpoint that views subject HS1 from approximately the back is set. In the free viewpoint image in Fig. 6B, subject HS2 is viewed approximately from the front, and subject HS3 and a basketball goal that were not shown in Fig. 6A are shown. For example, an image of about 1 to 2 seconds in which the viewpoint is gradually moved from the state of Fig. 6A in the direction of arrow C to the state of Fig. 6B is generated as a free viewpoint image (FV clip). Of course, the time length of the FV clip as a free viewpoint image and the trajectory of the viewpoint movement can be variously considered.

[0063] The transmission control unit 33 controls the transmission of the free viewpoint image (FV clip) generated by the image generation processing unit 32 as described above to the image creation controller 1 via the NAS 5. In this case, the transmission control unit 33 controls so that accompanying information for generating an output image is also transmitted to the image creation controller 1. The accompanying information is assumed to be information specifying the images of the previous clip and the next clip. That is, the accompanying information is information specifying which image from the image data V1 to V16 is to be used to create (cut out) the previous clip and the next clip. Information specifying the time length of the previous clip and the next clip is also assumed to be the accompanying information.

[0064] <3. GUI> As described above, the FV clip as a free viewpoint image is created in the free viewpoint image server 2 in response to the operation of the operator OP2.

[0065] In the free viewpoint image server 2, for example, a GUI screen 40 as shown in FIG. 7 is displayed on the display unit 77, allowing the operator OP2 to confirm and operate it.

[0066] In the scene window 41, for example, an image of the image section to be generated is displayed on a monitor, so that the operator OP2 can check the contents of the scene for which a free viewpoint image is to be generated. For example, a list of scenes designated as the image interval to be generated is displayed in the scene list display section 42. The operator OP2 can use the scene list display section 42 to select a scene to be displayed in the scene window 41.

[0067] The camerawork window 43 displays the positions of the imaging devices 10 that are arranged, the selected camerawork, or a plurality of selectable cameraworks. The camerawork is information that mainly indicates the movement trajectory of the viewpoint in a free viewpoint image. For example, when creating an FV clip in which the orientation and distance as the position (i.e., viewpoint) of the imaging device 10 are changed with respect to a subject for which a 3D model has been generated, the parameters required to form the trajectory of the viewpoint are regarded as camerawork information. As a display of the camerawork, for example, the trajectory of the viewpoint movement is displayed.

[0068] A list of various camera work information that has been created and stored in advance is displayed in the camera work list display section 44. The operator OP2 can select camera work to be used for generating an FV clip from among the camera work displayed in the camera work list display section 44. The parameter display section 45 displays various parameters related to the selected camera work.

[0069] The transmission window 46 displays information regarding the transmission of the created FV clip to the image creation controller 1.

[0070] <4. Clips containing free viewpoint images> Next, an output clip including an FV clip as a free viewpoint image will be described. FIG. 8 shows an example of an output clip that is configured by linking a previous clip, an FV clip, and a next clip.

[0071] For example, the previous clip is an actual moving image in the section from time code TC1 to TC2 in certain image data Vx among image data V1 to V16. The subsequent clip is an actual moving image in the section from time code TC5 to TC6 in certain image data Vy among the image data V1 to V16. It is generally assumed that the image data Vx is image data of the image capturing device 10 before the viewpoint movement due to the FV clip begins, and the image data Vy is image data of the image capturing device 10 at the end of the viewpoint movement due to the FV clip.

[0072] In this example, the previous clip is a video with a duration of t1, the FV clip is a free viewpoint image with a duration of t2, and the next clip is a video with a duration of t3. The total playback time of the output clip is t1+t2+t3. For example, a 5-second output clip may consist of a 1.5-second video, a 2-second free viewpoint image, and a 1.5-second video.

[0073] Here, the FV clip is shown as the section from time code TC3 to TC4, but this may or may not correspond to the number of frames in the actual video. In other words, with an FV clip, there are cases where the viewpoint is moved while the video time is stopped (TC3 = TC4), and cases where the viewpoint is moved without stopping the video time (TC3 ≠ TC4). For the sake of explanation, an FV clip in which the viewpoint is moved while the video time is frozen will be called a "still image FV clip," and an FV clip in which the viewpoint is moved without freezing the video time will be called a "video FV clip."

[0074] If a still image FV clip is shown based on video frames, it will look like Figure 9. In this example, the time codes TC1 and TC2 of the previous clip are the time codes of frames F1 and F81, and the time code of the following frame F82 is the time code TC3=TC4 in Figure 8. And the time codes TC5 and TC6 of the next clip are the time codes of frames F83 and F166. That is, this is the case where a free viewpoint image in which the viewpoint moves is generated for a single still image of frame F82.

[0075] On the other hand, a video FV clip will look like Figure 10. In this example, the time codes TC1 and TC2 of the previous clip become the time codes of frames F1 and F101, and the time codes of frames F102 and F302 become the time codes TC3 and TC4 in Figure 8. And the time codes TC5 and TC6 of the next clip become the time codes of frames F303 and F503. That is, this is the case where a free viewpoint image in which the viewpoint moves is generated for a video in a section of a plurality of frames from frame F102 to frame 302.

[0076] Therefore, the image section to be generated determined by the image creation controller 1 is the one-frame section of frame F82 when creating the still image FV clip of Figure 9, and is the multiple-frame section from frame F102 to frame 302 when creating the video FV clip of Figure 10.

[0077] For the example still image FV clip in FIG. 9, an example of the image content of the output clip is shown in FIG. For example, when creating a clip of a few seconds long that shows the scene before and after a basketball shot, an image that moves the viewpoint position and shows the moment of the shot as a free viewpoint image is inserted between the actual images before and after the shot. 11, the previous clip is the actual video from frame F1 to frame F81. The FV clip is a virtual image in which the viewpoint is moved in the scene of frame F81. The next clip is the actual video from frame F83 to frame F166. For example, an output clip including an FV clip is generated in this way and used as the image to be broadcast. In addition, the in / out points of an FV clip and the images between them may include both images taken from the viewpoint from the actual camera position and images taken from a viewpoint from a location other than the actual camera position, or they may be composed of multiple images taken from the viewpoint from the actual camera position or multiple images taken from a viewpoint from a location other than the actual camera position.

[0078] <5. Clip creation process> Below, a description will be given of an example of processing for creating an output clip performed in the image processing system of Fig. 1. The description will be focused mainly on the processing of the image creation controller 1 and the free viewpoint image server 2. First, the flow of processing including the operations of operators OP1 and OP2 will be described with reference to Fig. 12. Note that the processing of operator OP1 in Fig. 12 collectively shows the GUI processing of the image creation controller 1 and the operator operation. Also, the processing of operator OP2 collectively shows the GUI processing of the free viewpoint image server 2 and the operator operation.

[0079] Step S1: Select a scene When creating an output clip, the operator OP1 first selects a scene to be used as an FV clip. For example, the operator OP1 searches for a scene to be used as an FV clip while monitoring the captured images displayed on the display unit 77 on the image creation controller 1 side. Then, the operator OP1 selects an image section to be generated of one or more frames. This information on the image section to be generated is transmitted to the free viewpoint image server 2, and is made recognizable to the operator OP2 by the GUI on the display unit 77 on the free viewpoint image server 2 side. The information on the image section to be generated is specifically the information on the time codes TC3 and TC4 in Fig. 8. As described above, in the case of a still image FV clip, the time code TC3=TC4.

[0080] Step S2: Scene image transfer instruction The operator OP2 performs an operation to instruct the transfer of the image of the corresponding scene according to the designation of the image section to be generated. In response to this operation, the free viewpoint image server 2 transmits a transfer request for the image data of the section of time codes TC3 and TC4 to the image generation controller 1.

[0081] Step S3: Synchronous cutting In response to a request to transfer image data, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to extract the sections of time codes TC3 and TC4 from each of the 16 systems of image data from image data V1 to image data V16. Step S4: NAS transfer The image creation controller 1 then transfers to the NAS 5 all data in the sections of time codes TC3 and TC4 of image data V1 to V16.

[0082] Step S5: Thumbnail display The free viewpoint image server 2 displays thumbnails of the image data V1 to V16 in the section of the time codes TC3 and TC4 transferred to the NAS5. Step S6: Scene check An operator OP2 checks the scene contents in the section indicated by the time codes TC3 and TC4 on the GUI screen 40 of the free viewpoint image server 2. Step S7: Camerawork selection An operator OP2 selects camera work that he / she considers appropriate on the GUI screen 40 according to the contents of the scene. Step S8: Execute generation After selecting the camera work, the operator OP2 performs an operation to generate and execute an FV clip.

[0083] Step S9: Modeling The free viewpoint image server 2 generates a 3D model of the subject and performs subject analysis, etc., using frame data for the time codes TC3 and TC4 in each of the image data V1 to V16, and parameters such as the placement positions of each imaging device 10 that were previously input. Step S10: Rendering The free viewpoint image server 2 generates a free viewpoint image based on the 3D model data and the subject analysis information. At this time, the free viewpoint image is generated so that the viewpoint is moved based on the camera work selected in step S7.

[0084] Step S11: Transfer The free viewpoint image server 2 transfers the generated FV clip to the image creation controller 1. At this time, not only the FV clip but also information specifying the previous clip and the next clip and information specifying the time lengths of the previous clip and the next clip can be transmitted as accompanying information. Step S12: Quality check On the free viewpoint image server 2 side, the operator OP2 can check the quality before or after the transfer in step S11. That is, the free viewpoint image server 2 plays and displays the generated FV clip on the GUI screen 40 so that the operator OP2 can check it. In some cases, it may be possible for the operator OP2 to re-create the FV clip without executing the transfer.

[0085] Step S13: Playlist generation The image creation controller 1 generates an output clip using the transmitted FV clip. In this case, the output clip is generated by combining the FV clip with either or both of the preceding and succeeding clips on the time axis. This output clip may be generated as stream data in which each frame as the previous clip, each virtually generated frame as the FV clip, and each frame as the next clip are actually linked in chronological order, but in this processing example, they are virtually linked as a playlist. That is, by generating a playlist so that a frame section as the previous clip is played, followed by the FV clip, and then the frame section as the next clip is played, it becomes possible to play the output clip without generating actual linked stream data as the output clip.

[0086] Step S14: Quality check Playback based on the playlist is performed by the GUI on the image creation controller 1 side, and an operator OP1 checks the contents of the output clip. Step S15: Playback instruction The operator OP1 issues a playback instruction by a predetermined operation in response to the quality check. The image creation controller 1 recognizes the input of the playback instruction. Step S16: Playback In response to the playback instruction, the image creation controller 1 supplies the output clip to the switcher 6. This makes it possible to broadcast the output clip.

[0087] The process of the image creation controller 1 for realizing the above-mentioned processing procedure is shown in FIG. 13, and the process of the free viewpoint image server 2 is shown in FIG. FIG. 13 shows processing executed by the CPU 71 of the information processing device 70 which is made into the image creation controller 1 by the functions shown in FIG. FIG. 14 shows processing executed by the CPU 71 of the information processing device 70 which is made into the free viewpoint image server 2 by the functions shown in FIG.

[0088] In step S101 of Fig. 13, the image creation controller 1 performs captured image display control. For example, the image data Vp, Vq shown in Fig. 1 are displayed on the display unit 77 so that the operator OP1 can monitor them. 13, the image creation controller 1 monitors whether a selection operation for an image section to be created has been performed. If no selection operation has been performed, the process proceeds to another process.

[0089] When the image creation controller 1 detects at a certain point that an operator OP1 has selected the image section to be generated, the image creation controller 1 proceeds from step S102 to step S103, and performs a process of notifying the free viewpoint image server 2 of information on the image section to be generated, i.e., information on the in point / out point indicating the section of the scene for which an FV clip is to be generated. Then, the image creation controller 1 waits for a transfer instruction from the free viewpoint image server 2 in step S104.

[0090] On the free viewpoint image server 2 side, in step S201 of FIG. 14, the notification of the image section to be generated from the image creation controller 1 is checked. When the information on the image section to be generated is received, the free viewpoint image server 2 proceeds from step S201 to step S202 and performs processing for a transfer request of image data. For example, the free viewpoint image server 2 displays a notification of the image section to be generated on a GUI screen and presents it to the operator OP2, enabling the transfer request operation. Then, in response to the operation of the operator OP2, the free viewpoint image server 2 transmits a transfer request to the image creation controller 1. In the above description, the transfer request requires the operation of the operator OP2, but the transfer request may be made without requiring any particular operation.

[0091] After transmitting the transfer request, the free viewpoint image server 2 waits for the data of frames of the image section to be generated that is required in the image data (V1 to V16) to be transferred to the NAS 5 in step S203.

[0092] 13, the image creation controller 1 proceeds to step S105 to control the video servers 4A, 4B, 4C, and 4D to extract the image section to be generated and transfer it to the NAS 5. As a result, frames (one frame or multiple frames) of the in-point / out-point section of the image section to be generated in the image data V1 to V16 are transferred to the NAS 5. Then, the image creation controller 1 proceeds to step S106 and waits for the FV clip to be transmitted from the free viewpoint image server 2.

[0093] The free viewpoint image server 2 can acquire image data in a state where it can process the image data by transferring the image data to the NAS 5. When the image data transfer to the NAS 5 is completed and the free viewpoint image server 2 is in a state where it can process the image data, the process proceeds from step S203 to step S204 in Fig. 14, and the scene is displayed on the GUI screen 40 using the acquired image data. This allows the operator OP2 to check the contents of the scene for which an FV clip is to be created. Then, in step S205, the free viewpoint image server 2 monitors the camera work selection operation by the operator OP2.

[0094] When the camerawork selection operation of the operator OP2 is detected, the free viewpoint image server 2 proceeds from step S205 to step S206, determines the camerawork, and sets the parameters for processing. Then, in step S207, monitors the creation execution operation by the operator OP2. Of course, in practice, it is desirable to allow the operator OP2 to redo the camerawork selection.

[0095] When the creation execution operation by the operator OP2 is detected, the free viewpoint image server 2 proceeds from step S207 to step S208, and executes the above-mentioned modeling process. Then, in step S209, rendering is performed to generate an FV clip. In step S210, the free viewpoint image server 2 displays the generated FV clip on the GUI screen 40 so that the operator OP2 can confirm it. Then, in step S211, a transmission operation by the operator OP2 is monitored, and in response to the transmission operation, in step S212, the FV clip is transmitted to the image creation controller 1. At this time, information specifying the previous clip or the next clip, and information specifying the time lengths of the previous clip and the next clip may also be transmitted as accompanying information.

[0096] When the image creation controller 1 confirms reception of the FV clip in step S106 in Fig. 13, it proceeds to step S107 and sets the previous clip and the next clip. For example, the previous clip and the next clip are selected based on the accompanying information. Specifically, it is set which of the image data V1 to V16 is to be used for the previous clip and the next clip.

[0097] The previous clip is preferably an image captured by the imaging device 10 that matches the viewpoint of the start image of the FV clip. The next clip is preferably an image captured by the imaging device 10 that matches the viewpoint of the end image of the FV clip. The free viewpoint image server 2 knows which imaging device 10's images are used as the viewpoint position of the start image and the viewpoint position of the end image of the FV clip. That is, the information is about the imaging device 10 of the start image and the imaging device 10 of the end image of the FV clip, which are specified by the camera work used in the current FV clip generation process. Therefore, by having information specifying the previous clip and the next clip transmitted from the free viewpoint image server 2 as information indicating these as accompanying information, the image creation controller 1 can easily set the previous clip and the next clip.

[0098] The image creation controller 1 also sets the time lengths of the preceding and following clips, but these may also be set according to the accompanying information transmitted. For example, if the operator OP2 wants to determine the lengths of the preceding and following clips according to the content and time length of the FV clip and the camera work, it is desirable that the time lengths of the preceding and following clips are determined on the free viewpoint image server 2 side, and the image creation controller 1 sets the preceding and following clips accordingly. Of course, the image creation controller 1 may determine the time lengths of the preceding and following clips, or fixed time lengths may be determined.

[0099] In step S108, the image creation controller 1 creates an output clip by combining the previous clip, the FV clip, and the next clip. As described above, in this case, the output clip can be created by generating a playlist and virtually combining the clips.

[0100] In step S109, the image creation controller 1 performs processing to transfer the output clip to the switcher 6. In practice, the output clip is played back according to the playlist so that the operator OP1 can check the contents, and the clip is sent to the switcher 6 in response to the operation of the operator OP1.

[0101] As described above, the image creation controller 1 and the free viewpoint image server 2 perform the processes shown in Figs. 13 and 14, whereby an output clip including an FV clip is generated and broadcast or the like.

[0102] <6. Camera movement detection> In order to generate a free viewpoint image, a 3D model is generated using image data V1, V2, . . . V16, and therefore parameters including position information of each imaging device 10 are important. For example, if the position of a certain imaging device 10 is moved during a broadcast, it becomes necessary to calibrate the parameters accordingly. For this reason, in the image processing system of Fig. 1, the utility server 8 is configured to detect the camera movement. The processing procedure of the image creation controller 1 and the utility server 8 when detecting camera movement will be described with reference to Fig. 15. Note that Fig. 15 shows the processing procedure in the same format as Fig. 12, but the utility server 8 is also operated by an operator OP2 as an example.

[0103] · Step S30: HD output To detect camera movement, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to output image data to the image conversion unit 7. The images from the video servers 4A, 4B, 4C, and 4D, i.e., the images from the 16 imaging devices 10, are subjected to resolution conversion by the image conversion unit 7 and supplied to the utility server 8.

[0104] Step S31: Background generation The utility server 8 generates a background image based on the supplied image. Since the background image does not change unless the camera position is changed, for example, a background image excluding subjects such as players is generated for 16 systems of image data (V1 to V16). Step S32: Check the difference The background image is displayed in the GUI, allowing operator OP2 to confirm changes in the image. Step S33: Automatic movement detection Camera movement can also be automatically detected by comparing background images at each point in time.

[0105] Step S34: Camera movement detection As a result of step S32 or step S33 described above, the movement of a certain imaging device 10 is detected. Step S35: Image acquisition Calibration becomes necessary when the imaging device is moved. Therefore, the utility server 8 requests the image creation controller 1 to send image data in a state after the image capture device is moved. Step S36: Extract clip In response to a request for image acquisition from the utility server 8, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to execute clip extraction for the image data V1 to V16. Step S37: NAS transfer The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data cut out as clips to the NAS 5.

[0106] Step S38: Feature point correction The utility server 8 can refer to and display the image in the state after the camera has been moved by transferring the image to the NAS 5. The operator OP2 performs operations required for calibration, such as correcting feature points. · Step S39: Recalibration The utility server 8 re-executes calibration for creating a 3D model using image data (V1 to V16) in a state after the camera has been moved.

[0107] Step S40: Reacquire background After the calibration, in response to an operation by the operator OP2, the utility server 8 requests reacquisition of image data for the background image. Step S41: Extract clip In response to a request for image acquisition from the utility server 8, the image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to execute clip extraction for the image data V1 to V16. Step S42: NAS transfer The image creation controller 1 controls the video servers 4A, 4B, 4C, and 4D to transfer the image data cut out as clips to the NAS 5. Step S43: Background generation The utility server 8 generates a background image using the image data transferred to the NAS 5. This image is used as a reference background image for subsequent camera movement detection, for example.

[0108] For example, by performing camera movement detection and calibration according to the above procedure, even if the position of the imaging device 10 is moved during broadcasting, the parameters are corrected accordingly, making it possible to continuously generate accurate FV clips.

[0109] <7. Summary and variations> According to the above embodiment, the following effects can be obtained. The information processing device 70 serving as the image creation controller 1 of the embodiment includes a section identification processing unit 21 that performs processing to identify a target image section to be generated for generating a free viewpoint image from a plurality of captured images (e.g., image data V1 to V16) captured simultaneously by a plurality of imaging devices 10, a target image transmission control unit 22 that controls the transmission of image data of the target image section in each of the plurality of captured images (V1 to V16) as image data to be used for generating an FV clip in the free viewpoint image server 2, and an output image generation unit 23 that generates an output clip including the received FV clip. In addition, the information processing device 70 serving as the free viewpoint image server 2 in the embodiment includes a target image acquisition unit 31 that acquires image data of a target image section for generating a free viewpoint image in each of a plurality of captured images (V1 to V16) captured simultaneously by a plurality of imaging devices 10, an image generation processing unit 32 that generates a free viewpoint image using the acquired image data, and a transmission control unit 33 that transmits the generated free viewpoint image to the image creation controller 1. In such a configuration, the image creation controller 1 transfers only image data of the image section for which an FV clip is to be generated to the NAS 5. The free viewpoint image server 2 generates an FV clip based on the 16-channel image data transferred to the NAS 5, and transmits it to the image creation controller 1. The image creation controller 1 generates an output clip including the FV clip. Therefore, the amount of image data transferred to the NAS 5 (free viewpoint image server 2 side) is small, and the processing load on the free viewpoint image server 2 side is also small, making it possible to generate an FV clip in a short time. In addition, it is sufficient to transmit the FV clip to the image creation controller 1 side, which reduces the processing load on the free viewpoint image server 2 and is effective for rapid clip creation. Normally, when creating an output clip that includes an FV clip, it is assumed that, for example, images of the entire scene of the output clip are sent to the NAS 5 (the free viewpoint image server 2 side) and the free viewpoint image server 2 side performs processing to generate the entire output clip including the previous clip and the next clip, which requires a relatively long processing time. And, in the broadcasting field, there is a demand to broadcast a replay 10 or so seconds after a certain play, for example, but it is difficult to create a clip that includes a free viewpoint image in a short time. In contrast, in the case of this embodiment, the free viewpoint image server 2 can generate an FV clip from image data of a scene in the section of the FV clip. This makes the operation of this embodiment extremely useful for shortening the time required to generate an output clip. For example, in the case of live broadcasts of sports and other events, it may be necessary to create an output clip in about 10 seconds in order to broadcast an instant replay. This is also advantageous for such applications.

[0110] In addition, since the more the number of imaging devices 10, the more accurate the free viewpoint clip can be created, it is desirable to arrange as many imaging devices 10 as possible. The more the number of imaging devices 10, the larger the data amount of the FV clip section. Considering this, it is desirable to transfer the shorter the section of image data. When transferring images captured by, for example, 16 imaging devices 10 as in this embodiment, it is clear that transferring image data of the time code section (TC3 to TC4: TC3=TC4 or TC3≠TC4) of the FV clip for 16 devices to NAS 5 is extremely advantageous in terms of system processing efficiency, rather than transferring image data of the time code section (TC1 to TC6) of the entire output clip for 16 devices to NAS 5.

[0111] The image creation controller 1 of the embodiment performs a process of notifying the free viewpoint image server 2 of information on the identified image section to be generated, and also controls the transmission of image data for the image section to be generated in response to a request from the free viewpoint image server 2. Furthermore, in response to the notification of the image section to be generated, the free viewpoint image server 2 according to the embodiment makes a transmission request for a plurality of image data for the image section to be generated in each of the plurality of captured images. This allows the generation of an FV clip to be executed according to the convenience of the free viewpoint image server 2 or the operator OP2 after the image creation controller 1 notifies the target image section. This prevents processing confusion on the free viewpoint image server 2 side, and facilitates the generation processing of the FV clip and the work of the operator OP2.

[0112] In the embodiment, the section identification processing unit 21 of the image creation controller 1 identifies a section of one frame of a captured image as a target image section to be generated. The target image acquisition unit 31 of the free viewpoint image server 2 acquires image data of one frame that is determined as a target image section to be generated in each of a plurality of captured images. That is, this is the case where a still image FV clip is generated. In the case of a still image FV clip in which the viewpoint is changed while the time is stopped in a still image state, the free viewpoint image server 2 needs to acquire only one frame for each of the image data V1 to V16 from the 16 imaging devices 10. This allows the processing load to be significantly reduced in the transfer process to the NAS 5 and the processing of the free viewpoint image server 2.

[0113] In the embodiment, the section identification processing unit 21 of the image creation controller 1 identifies a section of a plurality of frames of a captured image as a target image section to be generated. The target image acquisition unit 31 of the free viewpoint image server 2 acquires image data of a section of a plurality of frames in each of the plurality of captured images that is determined as a target image section to be generated. That is, this is the case where a moving image FV clip is generated. In the case of a video FV clip in which the viewpoint is changed while the time is not stopped, the free viewpoint image server 2 only needs to acquire a plurality of frames of the video section to be the FV clip for each of the image data V1 to V16 from the 16 imaging devices 10. Since only the minimum necessary frame section is acquired, no unnecessary processing occurs in the transfer processing to the NAS 5 or the processing of the free viewpoint image server 2.

[0114] In the embodiment, the output image generating unit 23 of the image creation controller 1 generates an output image by combining a front image or a back image with a free viewpoint image on the time axis. That is, the output image generating unit 23 generates an output clip by combining a previous clip or a next clip with a received FV clip, so that an output clip in which the viewpoint is moved in the middle of a scene can be generated. By combining the previous clip and the next clip with the FV clip on the image creation controller 1 side, the processing load of the free viewpoint image server 2 is reduced, which is effective for quick clip creation.

[0115] In the embodiment, the output image generating unit 23 of the image creation controller 1 generates an output image as a virtual clip in which a free viewpoint image and a previous image or a next image are virtually combined on the time axis using the playlist information. That is, by forming a playlist that connects the previous clip and the next clip to the FV clip, it is possible to output the image of the output clip by playing back according to the playlist, without actually creating an output clip as an image data stream. This simplifies the processing on the image creation controller 1 side and shortens the time required to create the output clip, making it suitable for the demand of quick output clip creation.

[0116] In the embodiment, an example has been given in which the transmission control unit 33 of the free viewpoint image server 2 controls to transmit, as accompanying information, information specifying a front image or a back image to be connected to the generated free viewpoint image to the image creation controller 1. The output image generation unit 23 of the image creation controller 1 acquires information specifying a front image or a back image to be connected to the free viewpoint image from the free viewpoint image server 2, and generates an output image in which the front image or the back image is combined with the free viewpoint image on the time axis based on the information. An FV clip is an image that changes from a certain viewpoint, i.e., an image of a certain imaging device 10, to another viewpoint, i.e., an image of another imaging device 10. In other words, which imaging device 10 captured the images connected before and after the FV clip can be determined according to the camerawork selected by the free viewpoint image server 2. Therefore, by acquiring information on the previous clip and the next clip from the free viewpoint image server 2, the image creation controller 1 can easily prepare appropriate images as the previous clip and the next clip connected to the FV clip. Then, by combining these on the time axis, an output clip that is well suited to the positions before and after the viewpoint movement can be generated. Furthermore, the free viewpoint image server 2 can identify the previous clip and the next clip from the camera work information, so the processing load does not increase.

[0117] In the embodiment, an example has been given in which the transmission control unit 33 of the free viewpoint image server 2 controls to transmit, as accompanying information, information specifying the time length of a front image or a rear image to be connected to the generated free viewpoint image to the image creation controller 1. Then, the output image generation unit 23 of the image creation controller 1 acquires information specifying the time length of a front image or a rear image to be connected to the free viewpoint image from the free viewpoint image server 2, prepares a front image or a rear image based on the information, and generates an output image in which the free viewpoint image is combined with the front image or the rear image on the time axis. By specifying the duration of the previous clip and the next clip to be combined with the FV clip that expresses viewpoint movement by the free viewpoint image server 2, it becomes possible to create an output clip of the expected duration from the free viewpoint image server 2 side. For example, it becomes possible for the operator OP2 who creates the FV clip to set the duration of the output clip according to the contents of the FV clip.

[0118] The free viewpoint image server 2 of the embodiment has been described as an example in which the image generation processing unit 32 generates a free viewpoint image using camera work information selected from pre-stored camera work information (viewpoint movement trajectory information). A plurality of pieces of camera work information are stored as information showing the trajectory of viewpoint movement, and the operator OP2 selects camera work information according to the specified scene. This allows the operator OP2 to generate an FV clip simply by selecting camera work information according to the contents of the scene as the image section to be generated. Therefore, it becomes possible to create an FV clip with viewpoint movement (camera work) according to the scene contents in a short time.

[0119] As an embodiment, a program that causes a CPU, a DSP, or a device including these to execute the processing described with reference to FIG. 13 can be considered. In other words, the program of the embodiment is a program that causes an information processing device to execute a section identification process (S102, S103) that identifies a target image section from which a free viewpoint image is to be generated for a plurality of captured images (V1 to V16) captured simultaneously by a plurality of imaging devices 10, a target image transmission control process (S105) that causes image data of the target image section in each of the plurality of captured images to be transmitted as image data to be used for generating a free viewpoint image in another information processing device, and an output image generation process (S107, S108) that generates an output image including the received free viewpoint image. By using such a program, the image creation controller 1 described above can be realized in a device serving as the information processing device 70.

[0120] As an embodiment, a program that causes a CPU, a DSP, or a device including these to execute the processing described with reference to FIG. 14 can be considered. In other words, the program of the embodiment is a program that causes an information processing device to execute an image data acquisition process (S202, S203) that acquires image data of a target image section for generating a free viewpoint image in each of a plurality of captured images (V1 to V16) captured simultaneously by a plurality of imaging devices 10, an image generation process (S204 to S209) that generates a free viewpoint image using the image data acquired in the image data acquisition process, and a transmission control process (S212) that controls the transmission of the generated free viewpoint image to another information processing device. By using such a program, the above-mentioned free viewpoint image server 2 can be realized in a device serving as the information processing device 70.

[0121] These programs can be recorded in advance on a HDD as a recording medium built into a device such as a computer device, or on a ROM in a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0122] Moreover, such a program is suitable for widely providing the image creation controller 1 and the free viewpoint image server 2 of the embodiment. For example, by downloading the program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, the personal computer or the like can function as the image creation controller 1 and the free viewpoint image server 2 of the present disclosure.

[0123] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0124] The present technology can also be configured as follows. (1) a section identification processing unit that performs processing to identify a target image section from which a free viewpoint image is to be generated, for a plurality of captured images simultaneously captured by a plurality of imaging devices; a target image transmission control unit that controls transmission of image data of the target image section in each of the plurality of captured images as image data used for generating a free viewpoint image in another information processing device; an output image generating unit that generates an output image including the received free viewpoint image; Information processing device. (2) the section identification processing unit performs a process of notifying the other information processing device of information on the identified image section to be generated; The target image transmission control unit controls transmission of image data of the target image section in response to a request from the other information processing device. The information processing device according to (1) above. (3) The section identification processing unit identifies a section of one frame of a captured image as the generation target image section. The information processing device according to (1) or (2) above. (4) The section identification processing unit identifies a section of a plurality of frames of a captured image as the generation target image section. The information processing device according to (1) or (2) above. (5) The output image generating unit generates an output image by combining a front image or a back image with a free viewpoint image on a time axis. An information processing device according to any one of (1) to (4) above. (6) The output image generating unit generates an output image as a virtual clip in which a free viewpoint image and a previous image or a next image are virtually combined on a time axis according to playlist information. An information processing device according to any one of (1) to (5) above. (7) The output image generation unit acquires information specifying a front image or a back image to be connected to a free viewpoint image from the other information processing device, and generates an output image by combining the front image or the back image with the free viewpoint image on a time axis based on the information. An information processing device according to any one of (1) to (6) above. (8) The output image generation unit acquires information specifying a time length of a front image or a rear image to be connected to a free viewpoint image from the other information processing device, prepares a front image or a rear image based on the information, and generates an output image in which the front image or the rear image is combined with the free viewpoint image on a time axis. An information processing device according to any one of (1) to (7) above. (9) a target image acquisition unit that acquires image data of a target image section that is set as a target for generating a free viewpoint image in each of a plurality of captured images captured simultaneously by a plurality of imaging devices; an image generation processing unit that generates a free viewpoint image using the image data acquired by the target image acquisition unit; A transmission control unit that controls transmission of the generated free viewpoint image to another information processing device. Information processing device. (10) The target image acquisition unit issues a transmission request for a plurality of image data of the image section to be generated in each of the plurality of captured images in response to the notification of the image section to be generated. The information processing device according to (9) above. (11) The target image acquisition unit acquires image data of one frame that is determined as the generation target image section in each of a plurality of captured images. The information processing device according to (9) or (10) above. (12) The target image acquisition unit acquires image data for a section of a plurality of frames in each of a plurality of captured images, the section being set as the target image section for generation. The information processing device according to (9) or (10) above. (13) The image generation processing unit includes: A free viewpoint image is generated using viewpoint movement trajectory information selected from pre-stored viewpoint movement trajectory information. An information processing device according to any one of (9) to (12) above. (14) The transmission control unit The other information processing device is controlled to transmit information specifying a front image or a rear image to be connected to the generated free viewpoint image. An information processing device according to any one of (9) to (13) above. (15) The transmission control unit The other information processing device is controlled to transmit information specifying the time length of the front image or the back image to be connected to the generated free viewpoint image. An information processing device according to any one of (9) to (14) above. (16) As an image processing system having a first information processing device and a second information processing device and outputting an output image including a free viewpoint image, The first information processing device, a section identification processing unit that performs processing to identify a target image section from which a free viewpoint image is to be generated, for a plurality of captured images simultaneously captured by a plurality of imaging devices; a target image transmission control unit that controls transmission of image data of the target image section in each of the plurality of captured images as image data used for generating a free viewpoint image in the second information processing device; an output image generating unit that generates an output image including the received free viewpoint image; Equipped with The second information processing device is a target image acquisition unit that acquires image data of the target image section that is set as a target for generating a free viewpoint image in each of the plurality of captured images; an image generation processing unit that generates a free viewpoint image using the image data acquired by the target image acquisition unit; a transmission control unit that controls transmission of the generated free viewpoint image to the first information processing device; An image processing system comprising: (17) An information processing device, A section identification process for identifying a target image section from which a free viewpoint image is to be generated, from a plurality of captured images simultaneously captured by a plurality of imaging devices; a target image transmission control process for transmitting image data of the target image section in each of the plurality of captured images as image data used for generating a free viewpoint image in another information processing device; an output image generation process for generating an output image including the received free viewpoint image; An information processing method for performing the above. (18) An information processing device, an image data acquisition process for acquiring image data of a target image section, which is a target for generating a free viewpoint image, in each of a plurality of captured images simultaneously captured by a plurality of imaging devices; an image generation process for generating a free viewpoint image using the image data acquired in the image data acquisition process; A transmission control process for controlling transmission of the generated free viewpoint image to another information processing device; An information processing method for performing the above. [Explanation of symbols]

[0125] 1 Image Creation Controller 2. Free viewpoint image server 3,4,4A,4B,4C,4D Video Server 5. NAS 6. Switcher 7 Image conversion section 8 Utility Server 10. Imaging device 21 Section specific processing unit 22 Target image transmission control section 23 Output image generation unit 31 Target image acquisition unit 32 Image generation processing section 33 Transmission control section 70 Information processing equipment 71 CPU 77 Display section

Claims

1. a section identification processing unit that performs processing to identify a target image section from which a free viewpoint image is to be generated, for a plurality of captured images simultaneously captured by a plurality of imaging devices; a target image transmission control unit that controls transmission of image data of the target image section in each of the plurality of captured images as image data used for generating a free viewpoint image in another information processing device; an output image generating unit that generates an output image including the received free viewpoint image; Information processing device.

2. the section identification processing unit performs a process of notifying the other information processing device of information on the identified image section to be generated; The target image transmission control unit controls transmission of image data of the target image section in response to a request from the other information processing device. The information processing device according to claim 1 .

3. The section identification processing unit identifies a section of one frame of a captured image as the generation target image section. The information processing device according to claim 1 .

4. The section identification processing unit identifies a section of a plurality of frames of a captured image as the generation target image section. The information processing device according to claim 1 .

5. The output image generating unit generates an output image by combining a front image or a rear image with a free viewpoint image on a time axis. The information processing device according to claim 1 .

6. The output image generating unit generates an output image as a virtual clip in which a free viewpoint image and a previous image or a next image are virtually combined on a time axis according to playlist information. The information processing device according to claim 1 .

7. The output image generation unit acquires information specifying a front image or a back image to be connected to a free viewpoint image from the other information processing device, and generates an output image by combining the front image or the back image with the free viewpoint image on a time axis based on the information. The information processing device according to claim 1 .

8. The output image generation unit acquires information specifying a time length of a front image or a rear image to be connected to a free viewpoint image from the other information processing device, prepares a front image or a rear image based on the information, and generates an output image in which the front image or the rear image is combined with the free viewpoint image on a time axis. The information processing device according to claim 1 .

9. a target image acquisition unit that acquires image data of a target image section that is set as a target for generating a free viewpoint image in each of a plurality of captured images captured simultaneously by a plurality of imaging devices; an image generation processing unit that generates a free viewpoint image using the image data acquired by the target image acquisition unit; a transmission control unit that controls transmission of the generated free viewpoint image to another information processing device; Equipped with The target image acquisition unit issues a transmission request for a plurality of image data of the image section to be generated in each of the plurality of captured images in response to the notification of the image section to be generated. Information processing device.

10. a target image acquisition unit that acquires image data of a target image section that is set as a target for generating a free viewpoint image in each of a plurality of captured images captured simultaneously by a plurality of imaging devices; an image generation processing unit that generates a free viewpoint image using the image data acquired by the target image acquisition unit; a transmission control unit that controls transmission of the generated free viewpoint image to another information processing device; Equipped with The transmission control unit The other information processing device is controlled to transmit information specifying a front image or a rear image to be connected to the generated free viewpoint image. Information processing device.

11. a target image acquisition unit that acquires image data of a target image section that is set as a target for generating a free viewpoint image in each of a plurality of captured images captured simultaneously by a plurality of imaging devices; an image generation processing unit that generates a free viewpoint image using the image data acquired by the target image acquisition unit; a transmission control unit that controls transmission of the generated free viewpoint image to another information processing device; Equipped with The transmission control unit The other information processing device is controlled to transmit information specifying the time length of the front image or the back image to be connected to the generated free viewpoint image. Information processing device.

12. The target image acquisition unit acquires image data of one frame that is determined as the generation target image section in each of a plurality of captured images. The information processing device according to any one of claims 9 to 11.

13. The target image acquisition unit acquires image data for a section of a plurality of frames in each of a plurality of captured images, the section being set as the target image section for generation. The information processing device according to any one of claims 9 to 11.

14. The image generation processing unit includes: A free viewpoint image is generated using viewpoint movement trajectory information selected from pre-stored viewpoint movement trajectory information. The information processing device according to any one of claims 9 to 11.

15. An image processing system having a first information processing device and a second information processing device and outputting an output image including a free viewpoint image, The first information processing device, a section identification processing unit that performs processing to identify a target image section from which a free viewpoint image is to be generated, for a plurality of captured images simultaneously captured by a plurality of imaging devices; a target image transmission control unit that controls transmission of image data of the target image section in each of the plurality of captured images as image data used for generating a free viewpoint image in the second information processing device; an output image generating unit that generates an output image including the received free viewpoint image; Equipped with The second information processing device is a target image acquisition unit that acquires image data of the target image section that is set as a target for generating a free viewpoint image in each of the plurality of captured images; an image generation processing unit that generates a free viewpoint image using the image data acquired by the target image acquisition unit; a transmission control unit that controls transmission of the generated free viewpoint image to the first information processing device; An image processing system comprising:

16. An information processing device, a section identification process for identifying a target image section from which a free viewpoint image is to be generated, for a plurality of captured images simultaneously captured by a plurality of imaging devices; a target image transmission control process for transmitting image data of the target image section in each of the plurality of captured images as image data used for generating a free viewpoint image in another information processing device; an output image generation process for generating an output image including the received free viewpoint image; An information processing method for performing the above.

17. An information processing device, an image data acquisition process for acquiring image data of a target image section, which is a target for generating a free viewpoint image, in each of a plurality of captured images simultaneously captured by a plurality of imaging devices; an image generation process for generating a free viewpoint image using the image data acquired in the image data acquisition process; A transmission control process for controlling transmission of the generated free viewpoint image to another information processing device; Do the following: In the image data acquisition process, in response to the notification of the image section to be generated, a transmission request is made for a plurality of image data of the image section to be generated for each of the plurality of captured images. Information processing methods.

Images