Information processing apparatus, image processing system, and information processing method

Abstract

An image processing system is provided, including a first information processing device and a second information processing device. The first information processing device performs a process of identifying a generation target image interval from which a free view point image is generated, from a plurality of captured images captured simultaneously by a plurality of image devices; performs a control of causing image data of the generation target image interval in each of the plurality of captured images to be transferred as image data for generating a free view point image in the second information processing device; and generates an output image including the received free view point image. The second information processing device performs a process of acquiring image data of the generation target image interval in each of the plurality of captured images, a process of generating a free view point image by using the acquired image data, and a control of transferring the generated free view point image to the first information processing device.

Description

Technical Field

[0001] This technology relates to information processing apparatus, image processing system, and information processing method, and more particularly to technology related to the processing of free-viewpoint images of a subject whose image has been captured, allowing the subject to be viewed from any viewpoint in three-dimensional space. Background Technology

[0002] The following techniques are known: generating free-viewpoint images (also known as free-viewpoint videos or virtual viewpoint images (videos), etc., based on the three-dimensional information of the subject whose image has been captured in three-dimensional space, representing its image.

[0003] As a relevant conventional technique, Patent Document 1 can be mentioned below. Patent Document 1 discloses a technique related to the generation of a shooting style (camerawork) that can be regarded as a path of viewpoint movement.

[0004] Citation List

[0005] Patent documents

[0006] Patent Document 1: WO2018 / 030206A Summary of the Invention

[0007] The problem to be solved by the present invention

[0008] Free-viewpoint images can also be used as broadcast content and are also used as replay images, for example, in sports broadcasts. For instance, in football or basketball broadcasts, a few-second clip, such as a shooting scene, is created from live-recorded images and broadcast as a replay image. In this disclosure, a "clip" is an image of a scene created by cutting out or further processing recorded images.

[0009] Meanwhile, in live broadcast settings, operators need to create and broadcast clips for playback very quickly.

[0010] In view of the above, this technology proposes a technique that enables faster image generation, including editing of free-viewpoint images.

[0011] Solution to the problem

[0012] The information processing apparatus according to the present technology includes: an interval identification processing unit configured to perform processing for identifying a generation target image interval for generating a free viewpoint image from multiple captured images simultaneously captured by multiple image capture devices; a target image transmission control unit configured to perform control for transmitting image data of the generation target image interval in each of the multiple captured images as image data for generating a free viewpoint image in another information processing apparatus; and an output image generation unit configured to generate an output image including the received free viewpoint image.

[0013] The information processing device is, for example, used as a main controller for creating images such as clips in an image processing system. The information processing device identifies a target image region for generating one or more frames of a free-viewpoint image and transmits image data of only the target image region from multiple captured images to other information processing devices (the free-viewpoint image server that actually creates the free-viewpoint image). The created free-viewpoint image is then received to generate an output image.

[0014] In the aforementioned information processing apparatus according to the present technology, it is conceivable that the interval recognition processing unit performs processing to notify other information processing devices of information about the identified target image interval, and the target image transmission control unit performs transmission control of image data of the target image interval in response to a request from the other information processing devices.

[0015] The target image region is generated and then communicated to other information processing devices, which enables these devices to specify the image data required to generate the free-viewpoint image.

[0016] In the aforementioned information processing apparatus according to the present technology, it is conceivable that the interval recognition processing unit recognizes an interval of a frame of the captured image as the interval for generating the target image.

[0017] This is the process of generating a free viewpoint image of the target image range for a frame (i.e., a free viewpoint image in the case of changing the viewpoint while in a still image state at a stop time).

[0018] In the aforementioned information processing apparatus according to the present technology, it is conceivable that the interval recognition processing unit recognizes intervals of multiple frames of the captured image as intervals for generating the target image.

[0019] This is the processing of generating free viewpoint images of the target image range for multiple frames (i.e., free viewpoint images where the viewpoint is changed while the motion image is in motion without stopping time).

[0020] In the aforementioned information processing apparatus according to the present technology, it is conceivable that the output image generation unit generates an output image obtained by combining a previous or subsequent image with a free viewpoint image on a time axis.

[0021] That is, when an information processing device receives a free viewpoint image created by another information processing device, the information processing device connects the previous image and the subsequent image to the free viewpoint image to obtain an output image.

[0022] In the aforementioned information processing apparatus according to the present technology, it is conceivable that the output image generation unit generates an output image as a virtual clip obtained by virtually combining a free viewpoint image with a previous or subsequent image on the timeline based on playback list information.

[0023] That is, when an information processing device receives a free viewpoint image created by another information processing device, the previous image and the subsequent image are concatenated to form an output image, and the concatenation of the images is virtually performed as a concatenation on the playback list information indicating the playback order.

[0024] In the aforementioned information processing apparatus according to the present technology, it is conceivable that the output image generation unit obtains information from other information processing apparatus about a previous or subsequent image to be connected to a free viewpoint image, and generates an output image by combining the previous or subsequent image with the free viewpoint image on a time axis based on the obtained information.

[0025] That is, the information processing device receives a free viewpoint image created by another information processing device, and also receives information from the other information processing device specifying previous and subsequent images to be connected to the free viewpoint image.

[0026] In the aforementioned information processing apparatus according to the present technology, it is conceivable that the output image generation unit obtains information from other information processing apparatus regarding the time length of a previous or subsequent image to be connected to a free viewpoint image, prepares the previous or subsequent image based on the obtained information, and generates an output image obtained by combining the previous or subsequent image with the free viewpoint image on a time axis.

[0027] That is, the information processing device receives a free viewpoint image created by another information processing device, and also receives information from the other information processing device about the time length of previous and subsequent images to be connected to the free viewpoint image.

[0028] Other information processing apparatuses according to the present technology include: a target image acquisition unit configured to acquire image data of a generation target image region for generating a free viewpoint image from each of a plurality of captured images simultaneously captured by a plurality of image capture devices; an image generation processing unit configured to generate a free viewpoint image by using the image data acquired by the target image acquisition unit; and a transmission control unit configured to perform control of transmitting the generated free viewpoint image to other information processing apparatuses.

[0029] The information processing device is, for example, an information processing device used as a free-viewpoint image server in an image processing system. The information processing device acquires multiple captured images as intervals of a scene for which a free-viewpoint image is generated, generates a free-viewpoint image based on the captured images, and transmits the free-viewpoint image to the information processing device used as a main controller for image creation.

[0030] In the aforementioned information processing apparatus according to the present technology, it is conceivable that the target image acquisition unit, in response to a notification of generating a target image interval, requests the transmission of multiple sets of image data for generating the target image interval in each of a plurality of captured images.

[0031] That is, by generating a notification of the target image region, the image region to be acquired is identified, and image data of multiple viewpoints of the region is requested to be transmitted.

[0032] In the information processing apparatus according to the present technology described above, it is conceivable that the target image acquisition unit acquires image data of a frame corresponding to the range of the target image generated from each of a plurality of captured images.

[0033] This is the process of generating a free viewpoint image of the target image range for a frame (i.e., a free viewpoint image in the case of changing the viewpoint while in a still image state at a stop time).

[0034] In the information processing apparatus according to the present technology described above, it is conceivable that the target image acquisition unit acquires image data of each of a plurality of captured images, corresponding to a range of frames in which the target image range is generated.

[0035] This is the processing of generating free viewpoint images of the target image range for multiple frames (i.e., free viewpoint images where the viewpoint is changed while the motion image is in motion without stopping time).

[0036] In the aforementioned information processing apparatus according to the present technology, it is conceivable that the image generation processing unit generates a free viewpoint image by using viewpoint movement path information selected from pre-stored viewpoint movement path information.

[0037] Multiple pieces of information indicating the path of viewpoint movement are prepared in advance and selected, for example, through operator actions.

[0038] In the aforementioned information processing apparatus according to the present technology, it is conceivable that the transmission control unit performs control to transmit information specifying a previous or subsequent image to be connected to the generated free viewpoint image to other information processing apparatuses.

[0039] That is, the free viewpoint image is received by other information processing devices that serve as the main controller for clip creation, and information about previous and subsequent images to be connected to the free viewpoint image is also transmitted for use in generating output images in other information processing devices.

[0040] In the aforementioned information processing apparatus according to the present technology, it is conceivable that the transmission control unit performs control to transmit information specifying the time length of a previous or subsequent image to be connected to the generated free viewpoint image to other information processing apparatuses.

[0041] That is, the free viewpoint image is received by other information processing devices that serve as the main controller for clip creation, and information about the time length of the previous and subsequent images to be connected to the free viewpoint image is also transmitted for use in generating the output image in other information processing devices.

[0042] The information processing method according to this technology is such that an information processing device, acting as a main controller for image creation, performs the following processes: interval identification processing for identifying a generation target image interval for generating a free viewpoint image from multiple captured images simultaneously captured by multiple image capture devices; target image transmission control processing for transmitting image data of the generation target image interval in each of the multiple captured images as image data for generating a free viewpoint image in other information processing devices; and output image generation processing for generating an output image including the received free viewpoint image.

[0043] Another information processing method according to the present technology is an information processing method in which an information processing device acting as a free-viewpoint image server performs the following processes: image data acquisition processing for acquiring image data of a generation target image range for generating a free-viewpoint image from each of a plurality of captured images simultaneously captured by a plurality of image capture devices; image generation processing for generating a free-viewpoint image by using the image data acquired in the image data acquisition processing; and transmission control processing for performing control to transmit the generated free-viewpoint image to other information processing devices.

[0044] Therefore, the various information processing devices work together to generate an output image that includes a free viewpoint image. Attached Figure Description

[0045] Figure 1 This is a block diagram of a system configuration according to an embodiment of the present technology.

[0046] Figure 2 This is an illustrative diagram illustrating an example of a camera configured to generate free-viewpoint images according to an embodiment.

[0047] Figure 3 This is a block diagram of the hardware configuration of an information processing apparatus according to an embodiment.

[0048] Figure 4 This is an explanatory diagram illustrating the function of the image creation controller according to an embodiment.

[0049] Figure 5 This is an explanatory diagram illustrating the function of a free-viewpoint image server according to an embodiment.

[0050] Figure 6 This is an explanatory diagram of a viewpoint in a free-viewpoint image according to an embodiment.

[0051] Figure 7 This is an explanatory diagram of the user interface screen on the free-viewpoint image server side according to an embodiment.

[0052] Figure 8 This is an explanatory diagram of the output clip according to an embodiment.

[0053] Figure 9 This is an explanatory diagram of an output clip including a still image FV clip according to an embodiment.

[0054] Figure 10 This is an explanatory diagram of an output clip including a motion picture (FV) clip according to an embodiment.

[0055] Figure 11 This is an illustrative diagram illustrating an example of an output clipped image according to an embodiment.

[0056] Figure 12 This is an illustrative diagram illustrating the workflow for clip creation according to an embodiment.

[0057] Figure 13 This is a flowchart of the process for creating a controller based on an embodiment.

[0058] Figure 14 This is a flowchart of the processing of a free-viewpoint image server according to an embodiment.

[0059] Figure 15 This is an illustrative diagram illustrating the workflow for camera motion detection according to an embodiment. Detailed Implementation

[0060] The embodiments are described below in the following order.

[0061] <1. System Configuration>

[0062] <2. Configuration of Image Creation Controller and Free Viewpoint Image Server>

[0063] <3.GUI>

[0064] <4. Clipping including free-viewpoint images>

[0065] <5. Clip Creation Processing>

[0066] <6. Camera Motion Detection>

[0067] <7. Conclusions and Examples of Changes>

[0068] <1. System Configuration>

[0069] Figure 1 An example configuration of an image processing system according to an embodiment of the present technology is shown.

[0070] The image processing system includes an image creation controller 1, a free-viewpoint image server 2, a video server 3, multiple (e.g., four) video servers 4A, 4B, 4C and 4D, a network attached storage (NAS) 5, a switcher 6, an image conversion unit 7, a utility server 8, and multiple (e.g., sixteen) image capture devices 10.

[0071] Hereinafter, the term "camera" refers to the image capture device 10. For example, "camera arrangement" means the arrangement of multiple image capture devices 10.

[0072] In addition, as a general term without distinction between video servers 4A, 4B, 4C and 4D, "video server 4" is used.

[0073] In the image processing system, a free viewpoint image corresponding to an image observed from any viewpoint in three-dimensional space can be generated based on captured images (e.g., image data V1 to V16) acquired from multiple image capture devices 10, and an output clip including the free viewpoint image can be created.

[0074] exist Figure 1 In the diagram, the connection status between individual units is represented by solid lines, dashed lines, and double lines.

[0075] Solid lines represent Serial Digital Interface (SDI) connections, an interface standard used for connecting broadcast equipment such as cameras and switchers, and assumed to be, for example, 4K compatible. Image data is primarily transmitted and received between individual devices via SDI cabling.

[0076] Double lines indicate connections using communication standards for building computer networks (such as 10G Ethernet). The image creation controller 1, free-viewpoint image server 2, video server 3, 4A, 4B, 4C and 4D, NAS 5, and utility server 8 are connected via a computer network, enabling them to send and receive image data and various control signals to each other.

[0077] The dashed line between video servers 3 and 4 indicates that video servers 3 and 4, equipped with inter-server file sharing capabilities, are connected via, for example, a 10G network. As a result, each video server can preview and send material from other video servers, including video servers 4A, 4B, 4C, and 4D. That is, a system using multiple video servers is constructed, thereby achieving efficient highlight editing and sending.

[0078] Each of the image capture devices 10 is, for example, a digital camera device having an image capture element such as a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor, and acquires captured images (image data V1 to V16) as digital data. In this example, each of the image capture devices 10 acquires captured images as moving images.

[0079] In this example, it is assumed that each of the image capture devices 10 captures an image of a basketball or football game in progress, and each of the image capture devices 10 is positioned at a predetermined location within the game venue with a predetermined orientation. In this example, the number of image capture devices 10 is 16, but having at least two or more image capture devices 10 is sufficient to generate free-viewpoint images. Increasing the number of image capture devices 10, and capturing images of the target subject from multiple angles, improves the accuracy of the 3D reconstruction of the subject and enhances the image quality of the virtual viewpoint image.

[0080] Figure 2 An example of setting up an image capture device 10 around a basketball court is shown. Assume that the circle (○) represents the image capture device 10. For example, Figure 2 This example illustrates a camera setup for capturing images with emphasis on the area near the goal on the left side of the image. Of course, the camera setup and number of cameras are merely examples and should be determined based on the content and purpose of image capture and broadcasting.

[0081] The image creation controller 1 is implemented using an information processing device. The image creation controller 1 can be implemented using, for example, a dedicated workstation, a general-purpose personal computer, or a mobile terminal.

[0082] Image creation controller 1 performs control / action management for video servers 3 and 4, as well as processing for clip creation.

[0083] As an example, image creation controller 1 is a device that can be operated by operator OP1. Operator OP1 gives instructions such as selecting or creating clip content.

[0084] The free-viewpoint image server 2 is implemented as an information processing device that performs processing for the actual creation of free-viewpoint images (free-viewpoint (FV) clips described later) according to instructions from the image creation controller 1, etc. The free-viewpoint image server 2 can also be implemented using, for example, a dedicated workstation, a general-purpose personal computer, or a mobile terminal.

[0085] As an example, the free-viewpoint image server 2 is a device that can be operated by operator OP2. Operator OP2 performs operations such as selecting the shooting style for creating FV clips as free-viewpoint images.

[0086] The configuration and processing of image creation controller 1 and free-viewpoint image server 2 are described later. Furthermore, operators OP1 and OP2 perform operations; however, for example, image creation controller 1 and free-viewpoint image server 2 can be set up side-by-side and operated by a single operator.

[0087] Each of the video servers 3 and 4 is an image recording device and includes a data recording unit such as a solid-state drive (SSD) or hard disk drive (HDD) and a control unit that performs data recording / playback control on the data recording unit.

[0088] Video servers 4A, 4B, 4C, and 4D can each receive input from, for example, four systems, and each can simultaneously record images captured by four image capture devices 10.

[0089] For example, video server 4A records image data V1, V2, V3, and V4. Video server 4B records image data V5, V6, V7, and V8. Video server 4C records image data V9, V10, V11, and V12. Video server 4D records image data V13, V14, V15, and V16.

[0090] This allows for the simultaneous recording of all images captured by the sixteen image capture devices 10.

[0091] Video servers 4A, 4B, 4C, and 4D continuously record, for example, during a sporting event to be broadcast.

[0092] Video server 3 is connected directly to image creation controller 1, for example, and can, for example, perform inputs and outputs for both systems. Image data Vp and Vq are shown as inputs for both systems. As image data Vp and Vq, any two images captured by image capture device 10 (any two sets of image data V1 to V16) can be selected. Of course, image data Vp and Vq can be images captured by other image capture devices.

[0093] The image creation controller 1 can display image data Vp and Vq on a monitor as a monitor image. Operator OP1 can, for example, view the scene being imaged and recorded for broadcast based on the image data Vp and Vq input to video server 3.

[0094] Furthermore, since video servers 3 and 4 are connected to enable file sharing, image creation controller 1 can monitor and display images captured by each of the image capture devices 10 recorded in video servers 4A, 4B, 4C, and 4D, which allows operator OP1 to view the captured images from each other.

[0095] Note that in this example, each captured image in the image capture device 10 is time-coded, and frame synchronization can be achieved in the processing of video servers 3, 4A, 4B, 4C, and 4D.

[0096] NAS 5 is a storage device on the network and includes, for example, SSDs or HDDs. In this example, NAS 5 is a device that stores, in the case of some frames of image data V1, V2, ..., V16 recorded in video servers 4A, 4B, 4C, and 4D for the generation and transmission of free-viewpoint images, the same frames for processing in free-viewpoint image server 2, or stores the created free-viewpoint images.

[0097] Switcher 6 is a device that receives input images output via video server 3 and selects the main image PGMout to be ultimately selected and broadcast. For example, broadcast directors and others perform the necessary operations.

[0098] Image conversion unit 7, for example, converts the resolution of image data from image capture device 10 and combines the image data to generate a surveillance image of the camera arrangement, and provides the surveillance image to utility server 8. For example, 16 system image data (V1 to V16) as 8K images are converted into 4 system images arranged in a tiled shape after the resolution is converted to 4K images, and the 4 system images are supplied to utility server 8.

[0099] The utility server 8 is a computer device capable of performing various related processes, and in this example, the utility server 8 is a device that performs processes for detecting camera movement used for calibration. For example, the utility server 8 monitors image data supplied from the image conversion unit 7 to detect camera movement. Camera movement is, for example, as... Figure 2 The arrangement position of any one of the image capture devices 10, as shown, can be changed. Information about the arrangement position of the image capture devices 10 is an important element for generating free-viewpoint images, and changes in the arrangement position require parameter resetting. Therefore, the movement of the monitoring camera is necessary.

[0100] <2. Configuration of Image Creation Controller and Free Viewpoint Image Server>

[0101] For example, the image creation controller 1, free-viewpoint image server 2, video servers 3 and 4, and utility server 8, configured as described above, can be implemented as having Figure 3 The information processing device 70 is configured as shown.

[0102] exist Figure 3 In this information processing device 70, the CPU 71 performs various processes according to the program stored in the ROM 72 or the program downloaded from the storage unit 79 to the RAM 73. The RAM 73 also appropriately stores data and other data required by the CPU 71 to perform various processes.

[0103] CPU 71, ROM 72 and RAM 73 are interconnected via bus 74. Input / output interface 75 is also connected to bus 74.

[0104] The input unit 76, including the operator and operating equipment, is connected to the input / output interface 75.

[0105] The input unit 76 can be, for example, various operators and operating devices such as keyboards, mice, keys, dial pads, touch panels, touchpads, and remote controls.

[0106] The input unit 76 detects user operations, and the CPU 71 parses the signals corresponding to the input operations.

[0107] In addition, the input / output interface 75 is integrated or separately connected to a display unit 77, including a liquid crystal display (LCD) or an organic electroluminescent (EL) panel, and to an audio output unit 78, including a speaker.

[0108] The display unit 77 is a display unit that displays various instructions, and is implemented by, for example, a display device installed in the housing of the information processing device 70 or a separate display device connected to the information processing device 70.

[0109] The display unit 77 is used to display images for various types of image processing and motion images to be processed on the display screen based on commands from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, and messages based on commands from the CPU 71, i.e., it displays a graphical user interface (GUI).

[0110] In some cases, the storage unit 79, implemented by a hard disk or solid-state storage, and the communication unit 80, implemented by a modem, are connected to the input / output interface 75.

[0111] The communication unit 80 performs communication processing via transmission lines such as the Internet, wired / wireless communication with various devices, and bus communication.

[0112] The drive 82 is also connected to the input / output interface 75 as needed, and a removable recording medium 81, such as a disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately attached thereto.

[0113] The drive 82 can read data files such as image files (MF) and various computer programs from the removable recording medium 81. The read data files are stored in the storage unit 79, and the images and audio contained in the data files are output by the display unit 77 and the audio output unit 78. Furthermore, computer programs and other data read from the removable recording medium 81 can be installed in the storage unit 79 as needed.

[0114] In the information processing apparatus 70, software can be installed via network communication through the communication unit 80 or via the removable recording medium 81. Alternatively, the software can be pre-stored in the ROM 72 or the storage unit 79, etc.

[0115] When implementing the image creation controller 1 or the free-viewpoint image server 2 using this information processing device 70, for example, it is implemented in the CPU 71 via software. Figure 4 and Figure 5 The processing functions shown.

[0116] As a function formed in the CPU 71 of the information processing device 70, which serves as the image creation controller 1, Figure 4 The diagram shows the interval recognition processing unit 21, the target image transmission control unit 22, and the output image generation unit 23.

[0117] Regarding the multiple captured images (image data V1 to V16) simultaneously captured by multiple image capture devices 10, the interval recognition processing unit 21 performs processing for identifying the target image interval for generating a free viewpoint image. For example, in response to operator OP1 performing an operation to select a scene to be played back in the image, the interval recognition processing unit 21 performs processing for identifying the time code of the scene, especially the interval of the scene to become a free viewpoint image (the target image interval), and processing for notifying the time code to the free viewpoint image server 2.

[0118] Here, the target image range refers to the frame range actually used as the free viewpoint image. In the case of generating a free viewpoint image for a single frame in a moving image, that single frame corresponds to the target image range. In this case, the in / out points of the free viewpoint image have the same timecode.

[0119] Furthermore, when generating free-viewpoint images for intervals of multiple frames in a moving image, where multiple frames correspond to intervals of the generated target image, the in / out points of the free-viewpoint images have different timecodes.

[0120] Note that although the structure of the clips will be described later, the in / out points of the target image region are expected to be different from the in / out points of the output clip to be ultimately generated. This is because the previous and subsequent clips described later are coupled.

[0121] The target image transmission control unit 22 executes control to transmit image data (i.e., one or more frames of image data V1 to V16) from each of the multiple captured images as image data for generating free-viewpoint images in the free-viewpoint image server 2. Specifically, the target image transmission control unit 22 executes control to transmit image data as generated target image intervals from video servers 4A, 4B, 4C, and 4D to NAS 5.

[0122] The output image generation unit 23 performs processing to generate an output image (output clip) that includes the received free viewpoint image (FV clip) generated by the free viewpoint image server 2.

[0123] For example, through the processing of the output image generation unit 23, the image creation controller 1 combines a previous clip, which is an actual motion image at a previous time point, and a subsequent clip, which is an actual motion image at a subsequent time point, with an FV clip, which is a virtual image generated by the free viewpoint image server 2, on the time axis to obtain an output clip. That is, the previous clip + FV clip + subsequent clip are set as one output clip.

[0124] Of course, the previous clip + FV clip can be set as an output clip.

[0125] Alternatively, an FV clip plus a subsequent clip can be set as an output clip.

[0126] Alternatively, you can generate an output clip with only FV clips without combining previous and subsequent clips.

[0127] In any case, the image creation controller 1 generates an output clip including an FV clip to output the output clip to the switcher 6 so that the output clip can be used for broadcasting.

[0128] Next, as a function formed in the CPU 71 of the information processing device 70, which serves as the free viewpoint image server 2, Figure 5 The target image acquisition unit 31, the image generation and processing unit 32, and the transmission and control unit 33 are shown.

[0129] The target image acquisition unit 31 performs processing to acquire image data for generating a target image interval from each of the multiple captured images (image data V1 to V16) simultaneously captured by the multiple image capture devices 10, for which a free-viewpoint image is generated. That is, the target image acquisition unit 31 obtains image data of one or more frames specified by the in-point / out-point of the target image interval identified by the interval recognition processing unit 21 through the NAS 5 from the video servers 4A, 4B, 4C, and 4D, and can use the image data for generating a free-viewpoint image.

[0130] For example, for all groups of image data V1 to V16, the target image acquisition unit 31 acquires image data of one or more frames for generating the target image interval. To generate a high-quality free-viewpoint image, image data for generating the target image interval for all groups of image data V1 to V16 is acquired. As described above, a free-viewpoint image can be generated using images captured by at least two or more image capture devices 10; however, a more detailed 3D model and a high-quality free-viewpoint image can be generated by increasing the number of image capture devices 10 (i.e., the number of viewpoints). Therefore, for example, in the case of setting sixteen image capture devices 10, image data for generating the target image interval is acquired for all groups of image data (V1 to V16) from the sixteen image capture devices 10.

[0131] The image generation processing unit 32 is a function that generates a free viewpoint image (i.e., an FV clip in this example) by using image data acquired by the target image acquisition unit 31.

[0132] For example, the image generation processing unit 32 performs modeling processes including 3D model generation and subject analysis, as well as rendering processes such as generating a free-viewpoint image from the 3D model as a two-dimensional image.

[0133] 3D model generation is a process used to generate 3D model data (i.e., the three-dimensional structure of the subject recovered from a two-dimensional image) representing a subject in three-dimensional space, based on images captured by each of the image capture devices 10 and camera parameters for each of the image capture devices 10 input from, for example, a utility server 8. Specifically, the 3D model data includes data representing the subject in a three-dimensional coordinate system using (X, Y, Z).

[0134] In subject analysis, the position, orientation, and pose of the subject (a person / athlete) are analyzed based on 3D model data. Specifically, this includes estimating the subject's position, generating a simple model of the subject, and estimating the subject's orientation.

[0135] Then, a free-viewpoint image is generated based on the 3D model data and subject analysis information. For example, a free-viewpoint image is generated such that the viewpoint moves relative to a stationary 3D model of an athlete as the subject.

[0136] refer to Figure 6 Describes the viewpoint of a free-viewpoint image.

[0137] Figure 6 Image A shows a free-viewpoint image of a subject captured from a necessary viewpoint set in three-dimensional space. In this free-viewpoint image, subject HS1 is viewed from approximately the front, and subject HS2 is viewed from approximately the back.

[0138] Figure 6 B indicates the position at the viewpoint. Figure 6 The direction of arrow C in A is changed, and a virtual viewpoint image is set in the case of viewing the subject HS1 from approximately behind. Figure 6 In the free-viewpoint image of B, the subject HS2 is viewed from approximately the front, and it is shown that... Figure 6 The subject HS3 and the basketball goal are not shown in A.

[0139] For example, the generated viewpoint gradually moves in the direction of arrow C from... Figure 6 A's state to Figure 6 A 1-2 second image in state B is used as a free viewpoint image (FV clip). Of course, the duration of the FV clip used as a free viewpoint image and the path of viewpoint movement can be considered differently.

[0140] The transfer control unit 33 controls the transmission of the free-viewpoint image (FV clip) generated by the image generation processing unit 32 as described above to the image creation controller 1 via the NAS 5. In this case, the transfer control unit 33 also controls the transmission of supplementary information used to generate the output image to the image creation controller 1. Assume that the supplementary information specifies the images of the previous and subsequent clips. That is, the supplementary information specifies which image from image data V1 to V16 is used to create (cut) the previous and subsequent clips. Furthermore, the supplementary information may specify the duration of the previous or subsequent clip.

[0141] <3.GUI>

[0142] As described above, in response to the operation of operator OP2, an FV clip is created in the free viewpoint image server 2 as a free viewpoint image.

[0143] In the free-viewpoint image server 2, for example, the image displayed in the display unit 77 is as follows: Figure 7 The GUI screen 40 shown allows operator OP2 to view and perform operations.

[0144] In scene window 41, for example, the image used to generate the target image range is displayed in the monitor to allow operator OP2 to view the content of the scene used to generate the free viewpoint image.

[0145] The scene list display section 42 displays, for example, a list of scenes specified as the target image range for generation. Operator OP2 can select the scene to be displayed in the scene window 41 on the scene list display section 42.

[0146] The shooting style window 43 displays the location of the image capture device 10, the selected shooting style, and multiple selectable shooting styles.

[0147] Shooting style is information that primarily indicates the path of viewpoint movement in a free-viewpoint image. For example, in the case of creating an FV clip where the orientation or distance of the position (i.e., viewpoint) of the image capture device 10 changes relative to the subject for which a 3D model has been generated, the parameters required to form the path of the viewpoint are used as information about the shooting style. As a display of the shooting style, for example, the path of viewpoint movement is shown.

[0148] Shooting Style List Display Section 44 displays a list of various information about pre-created and stored shooting styles. Operator OP2 can select the shooting style to use for FV clip generation from the shooting styles displayed in Shooting Style List Display Section 44.

[0149] In the parameter display section 45, various parameters related to the selected shooting style are displayed.

[0150] In the transfer window 46, information about transferring the created FV clip to the image creation controller 1 is displayed.

[0151] <4. Clipping including free-viewpoint images>

[0152] Next, the description includes FV clipping as the output clip of a free viewpoint image.

[0153] As an example, Figure 8 This shows the state of the output clip formed by connecting the previous clip, the FV clip, and the subsequent clip.

[0154] For example, the previous clip is the actual motion image in the timecode TC1 to TC2 interval of a certain image data Vx in image data V1 to image data V16.

[0155] Furthermore, the subsequent editing is the actual motion image in the timecode TC5 to TC6 interval of a certain image data Vy in image data V1 to image data V16.

[0156] It is generally assumed that image data Vx is the image data of the image capture device 10 before the start of the viewpoint movement of the FV clip, and that image data Vy is the image data of the image capture device 10 at the end of the viewpoint movement of the FV clip.

[0157] In this example, the previous clip is a motion picture with a duration of t1, the FV clip is a free-viewpoint clip with a duration of t2, and the subsequent clip is a motion picture with a duration of t3. The playback duration of the entire output clip is equal to t1 + t2 + t3. For example, a 5-second output clip might consist of 1.5 seconds of motion pictures, 2 seconds of free-viewpoint clips, and 1.5 seconds of motion pictures, etc.

[0158] Here, FV clips are shown as the timecode interval TC3 to TC4, but this may or may not correspond to the number of frames in the actual motion picture.

[0159] Specifically, as an FV edit, there are cases where the viewpoint moves while the motion image is stopped (TC3 = TC4) and cases where the viewpoint moves while the motion image is not stopped (TC3 ≠ TC4).

[0160] For the purpose of description, an FV clip used to move the viewpoint in a time state where the moving image is stopped is called a "still image FV clip", and an FV clip used to move the viewpoint in a time state where the moving image is not stopped is called a "moving image FV clip".

[0161] Figure 9 A still image FV clip of a frame representing a reference moving image is shown. In this example, the timecodes TC1 and TC2 of the previous clip are the timecodes of frames F1 and F81, and the timecode of the subsequent frame F82 is... Figure 8 The timecodes TC3 and TC4 are in the first frame. Then, the timecodes TC5 and TC6 of the subsequent clips are the timecodes of frames F83 and F166.

[0162] That is, this is the case of generating a free viewpoint image in which the viewpoint moves relative to a still image of a frame (frame F82).

[0163] Meanwhile, motion picture FV editing, such as Figure 10 As shown. In this example, the timecodes TC1 and TC2 of the previously clipped frames are the timecodes of frames F1 and F101, and the timecodes of frames F102 and F302 are... Figure 8 The timecodes TC3 and TC4 are used in the first clip. Then, the timecodes TC5 and TC6 for subsequent clips are the timecodes for frames F303 and F503.

[0164] That is, this is the case of generating a free viewpoint image in which the viewpoint moves relative to the motion images of multiple frames (from frame F102 to frame 302).

[0165] Therefore, the target image range determined by the image creation controller 1 is used in the creation process. Figure 9 In the case of still image FV clipping, it is a frame interval (frame F82), while in the creation Figure 10 In the case of motion picture FV clipping, it is an interval of multiple frames (from frame F102 to frame 302).

[0166] Figure 11 Show Figure 9 An example of the output clip image content of a still image FV clip.

[0167] For example, in the case of creating a scene before and after a shot using a few seconds of clip, an image of the viewpoint position shifting is inserted as a free-viewpoint image at the moment of the shot between the actual images before and after the shot.

[0168] exist Figure 11 In the sequence, the previous clip is the actual motion image from frame F1 to frame F81. The FV clip is a virtual image that moves the viewpoint within the scene at frame F81. The subsequent clip is the actual motion image from frame F83 to frame F166.

[0169] For example, as described above, an output clip including an FV clip is generated and used as an image to be broadcast.

[0170] Note that the in / out points of an FV clip and the images in between can include both images from a viewpoint at the actual camera position and images from a viewpoint at a location other than the actual camera position. It can include multiple images from a viewpoint at the actual camera position and multiple images from a viewpoint at a location other than the actual camera position.

[0171] <5. Clip Creation Processing>

[0172] The following is a description of... Figure 1 This is an example of output clip creation processing performed in an image processing system. It primarily describes the processing of image creation controller 1 and free-viewpoint image server 2.

[0173] First, refer to Figure 12 Describe the processing flow for the operations performed by operators OP1 and OP2. Note that... Figure 12 The processing of operator OP1 summarizes the GUI processing and operator operations of the image creation controller 1. Furthermore, the processing of operator OP2 summarizes the GUI processing and operator operations of the free-viewpoint image server 2.

[0174] Step S1: Select Scene

[0175] To create an output clip, operator OP1 first selects the scene to be used as the FV clip. For example, operator OP1 searches for the scene to be used as the FV clip while monitoring the captured image displayed on the display unit 77 on the image creation controller 1 side. Then, a range of one or more frames of the target image to be generated is selected.

[0176] Information about the generated target image region is transmitted to the free viewpoint image server 2, allowing operator OP2 to recognize it using the GUI in the display unit 77 on the free viewpoint image server 2 side.

[0177] Specifically, the information about the generated target image region is about Figure 8 The information in timecodes TC3 and TC4. As mentioned above, in the case of still image FV clips, timecode TC3 = TC4.

[0178] Step S2: Instruct scene image transmission

[0179] In response to the specified target image range, operator OP2 performs an operation to instruct the transfer of images for the corresponding scene. In response to this operation, free-viewpoint image server 2 sends a transfer request to image creation controller 1 for the image data ranges corresponding to timecodes TC3 and TC4.

[0180] Step S3: Simultaneous cutting

[0181] In response to a request to transmit image data, the image creation controller 1 controls video servers 4A, 4B, 4C, and 4D to cut timecodes TC3 and TC4 intervals in the 16 system image data from image data V1 to image data V16.

[0182] Step S4: NAS Transfer

[0183] Then, the image creation controller 1 transmits the data from the timecode intervals TC3 and TC4 of all groups of image data V1 to image data V16 to the NAS 5.

[0184] Step S5: Display thumbnails

[0185] Free Viewpoint Image Server 2 displays thumbnails of image data V1 to image data V16 transmitted to NAS 5 in timecodes TC3 and TC4.

[0186] Step S6: View the scene

[0187] Operator OP2 views the scene content of the timecode TC3 and TC4 intervals through the GUI screen 40 of the free viewpoint image server 2.

[0188] Step S7: Select Shooting Style

[0189] Operator OP2 selects the appropriate shooting style in the GUI screen 40 based on the scene content.

[0190] Step S8: Execute generation

[0191] After selecting the shooting style, operator OP2 performs the operation to generate the FV clip.

[0192] Step S9: Modeling

[0193] The free viewpoint image server 2 generates a 3D model of the subject and analyzes the subject by using the frame data in the timecode intervals TC3 and TC4 of each group of image data in image data V1 to V16, as well as parameters such as the arrangement position of each of the pre-input image capture devices 10.

[0194] Step S10: Rendering

[0195] The free-viewpoint image server 2 generates a free-viewpoint image based on 3D model data and subject analysis information. At this point, generating the free-viewpoint image enables viewpoint movement based on the shooting style selected in step S7.

[0196] Step S11: Transfer

[0197] The free-viewpoint image server 2 transmits the generated FV clip to the image creation controller 1. At this time, not only the FV clip can be transmitted, but also supplementary information, such as information specifying the previous and subsequent clips, as well as information specifying the duration of the previous and subsequent clips.

[0198] Step S12: Check the quality

[0199] Note that on the free-viewpoint image server 2 side, operator OP2 can perform a quality check before or after the transmission in step S11. Specifically, the free-viewpoint image server 2 replays and displays the generated FV clip in the GUI screen 40, allowing operator OP2 to view the FV clip. In some cases, operator OP2 may also regenerate the FV clip without transmission.

[0200] Step S13: Generate playlist

[0201] Image creation controller 1 uses the FV clip thus transmitted to generate an output clip. In this case, one or both of the previous and subsequent clips are combined with the FV clip on the timeline to generate the output clip.

[0202] The output clip can be generated as streaming data, in which frames as previous clips, frames virtually generated as FV clips, and frames as subsequent clips are actually connected in time sequence; however, in the processing example, the frames are virtually connected as a playlist.

[0203] Specifically, a playlist is generated such that after playing back the frame intervals of the previous clip, the FV clip is played back, and then the frame intervals of the subsequent clip are played back, so that the output clip can be played back without generating the actual stream data that is connected to the output clip.

[0204] Step S14: Check the quality

[0205] The GUI on the image creation controller 1 side allows for playback based on playlists, and operator OP1 can view the contents of the output clips.

[0206] Step S15: Instruct playback

[0207] Operator OP1 issues a playback command based on a pre-defined operation after quality inspection. Image creation controller 1 recognizes the input playback command.

[0208] Step S16: Playback

[0209] In response to a playback command, the image creation controller 1 supplies the output clip to the switcher 6. This enables the broadcasting of the output clip.

[0210] Figure 13 The processing of image creation controller 1 is shown, and, Figure 14 The processing of the free-viewpoint image server 2 used to implement the above-described processing is shown.

[0211] Figure 13 Showing through Figure 4 The functions shown are processed by the CPU 71 of the information processing device 70, which serves as the image creation controller 1.

[0212] Figure 14 Showing through Figure 5 The functions shown are performed by the CPU 71 of the information processing unit 70, which serves as the free viewpoint image server 2.

[0213] exist Figure 13 In step S101, the image creation controller 1 executes captured image display control. For example, Figure 1 The image data Vp and Vq shown are displayed in display unit 77 and can be monitored by operator OP1.

[0214] exist Figure 13 In step S102, the image creation controller 1 monitors the operation used to select the target image region for generation. If no selection operation is performed, the process proceeds to another step.

[0215] When operator OP1 selects a target image region at a certain time point, the processing of image creation controller 1 proceeds from step S102 to step S103, and image creation controller 1 performs processing to notify free viewpoint image server 2 of information about the target image region (i.e., information about the in / out point of the region indicating the scene for which FV clips are generated).

[0216] Then, in step S104, the image creation controller 1 waits for a transmission instruction from the free viewpoint image server 2.

[0217] exist Figure 14 In step S201, the free viewpoint image server 2 views the notification from the image creation controller 1 regarding the generated target image range.

[0218] Upon receiving information regarding the generation of a target image region, the processing of the free-viewpoint image server 2 proceeds from step S201 to step S202, which involves processing a request to transmit image data. For example, the free-viewpoint image server 2 displays a notification of the generation of the target image region on a GUI screen and presents this notification to operator OP2, enabling the transmission request operation. Then, in response to operator OP2's operation, the transmission request is sent to the image creation controller 1.

[0219] Note that in the above description, the transfer request involves the operation of operator OP2; however, a transfer request can be made without any specific operation.

[0220] When a transmission request is sent, the free viewpoint image server 2 waits in step S203 for the data of the frames required to generate the target image region in the image data (V1~V16) to be transmitted to the NAS 5.

[0221] When a transmission request is received from the free-viewpoint image server 2, the image creation controller 1 processes the request from... Figure 13 Step S104 proceeds to step S105, and the image creation controller 1 executes control of video servers 4A, 4B, 4C, and 4D to cut and generate target image intervals and transmit them to NAS 5. Thus, frames (one or more frames) of the in-point / out-point intervals of the generated target image intervals in image data V1 to V16 are transmitted to NAS 5.

[0222] Then, the image creation controller 1 proceeds to step S106 and waits for the FV clip to be sent from the free viewpoint image server 2.

[0223] Freeview Image Server 2 transmits image data to NAS 5, enabling image data acquisition in a processable state. Once the image data has been fully transmitted to NAS 5 and Freeview Image Server 2 is ready to process it, processing begins from... Figure 14 The process proceeds from step S203 to step S204, where the scene is displayed on the GUI screen 40 using the acquired image data. This allows operator OP2 to confirm the content of the scene for which an FV clip is to be created.

[0224] Then, in step S205, the free viewpoint image server 2 monitors the shooting style selection operation of operator OP2.

[0225] When operator OP2's shooting style selection operation is detected, the processing of the free viewpoint image server 2 proceeds from step S205 to step S206, which determines the shooting style and sets the processing parameters. Then, in step S207, the creation execution operation of operator OP2 is monitored. Note that in practice, it is desirable to allow operator OP2 to select the shooting style again.

[0226] When operator OP2's creation execution operation is detected, the processing of the free viewpoint image server 2 proceeds from step S207 to step S208, where the aforementioned modeling processing is performed. Then, in step S209, rendering is performed to generate an FV clip.

[0227] In step S210, the free-viewpoint image server 2 displays the generated FV clip on the GUI screen 40, allowing operator OP2 to view the FV clip. Then, in step S211, the transmission operation of operator OP2 is monitored, and in step S212, the FV clip is sent to the image creation controller 1 in response to the transmission operation. Furthermore, at this time, in some cases, information specifying the previous and subsequent clips, as well as information specifying the duration of the previous and subsequent clips, is transmitted as supplementary information.

[0228] When in Figure 13 When the reception of the FV clip is confirmed in step S106, the processing of the image creation controller 1 proceeds to step S107, which sets the previous clip and the subsequent clip. For example, the previous clip and the subsequent clip are selected based on supplementary information. Specifically, it is set which of the image data V1 to V16 will be used for the previous clip and the subsequent clip.

[0229] The previous clip is expected to be an image captured by the image capture device 10 at the viewpoint corresponding to the start image of the FV clip. Similarly, the subsequent clip is expected to be an image captured by the image capture device 10 at the viewpoint corresponding to the end image of the FV clip. The information known to the free-viewpoint image server 2 is which image capture device 10 is used to capture the images at the viewpoint positions of the start and end images of the FV clip. That is, it is information about the image capture device 10 corresponding to the start and end images of the FV clip, defined by the shooting style used in the current FV clip generation process. Therefore, the image creation controller 1 can easily set the previous and subsequent clips by configuring the free-viewpoint image server 2 to send information specifying the previous and subsequent clips as supplementary information.

[0230] Furthermore, the image creation controller 1 sets the duration of the previous and subsequent clips, and only needs to set their duration based on the supplementary information transmitted. For example, in order for the operator OP2 to determine the duration of the previous and subsequent clips based on the content and duration of the FV clip and the shooting style, it is expected that the free viewpoint image server 2 will determine the duration of the previous and subsequent clips, and the image creation controller 1 will set the previous and subsequent clips accordingly.

[0231] Note that, of course, the image creation controller 1 can determine the duration of previous and subsequent clips, or it can determine a fixed duration.

[0232] In step S108, the image creation controller 1 creates an output clip by combining the previous clip, the FV clip, and the subsequent clip. As described above, in this case, the output clip can be created by generating a playlist and virtually combining the playlist.

[0233] In step S109, the image creation controller 1 performs processing to transmit the output clip to the switcher 6. In practice, the output clip is played back according to the playlist, allowing operator OP1 to view the content, and is transmitted to the switcher 6 in response to operator OP1's operation.

[0234] As described above, image creation controller 1 and free viewpoint image server 2 perform... Figure 13 and Figure 14 The processing enables the generation and broadcasting of output clips, including FV clips, and so on.

[0235] <6. Camera Motion Detection>

[0236] Note that in order to generate free-viewpoint images, 3D models are generated using image data V1, V2, ... and V16, therefore parameters including position information of each of the image capture devices 10 are important.

[0237] For example, if the position of an image capture device 10 moves during broadcasting, the parameters need to be calibrated accordingly. Therefore, in Figure 1 In the image processing system, server 8 is used to detect camera movement.

[0238] refer to Figure 15 Describe the processing of the image creation controller 1 and utility server 8 when detecting camera movement. Figure 15 With similar Figure 12 The format shows the processing procedure, but Figure 15 This shows an example of operator OP2 also operating utility server 8.

[0239] Step S30: HD Output

[0240] Image creation controller 1 controls video servers 4A, 4B, 4C, and 4D to output image data to image conversion unit 7 for use in camera motion detection. Images from video servers 4A, 4B, 4C, and 4D (i.e., images from the sixteen image capture devices 10) undergo resolution conversion by image conversion unit 7, and the results are supplied to utility server 8.

[0241] Step S31: Generate background

[0242] The utility server 8 generates a background image based on the image provided therefrom. Since the background image is an image that does not change unless the camera position is changed, for example, for 16 system image data (V1 to V16), a background image that does not include subjects such as athletes is generated.

[0243] Step S32: View the differences

[0244] The background image is displayed in the GUI, allowing operator OP2 to view changes in the image.

[0245] Step S33: Automatically detect movement

[0246] Camera movement can be automatically detected by performing comparison processing on the background image at each time point.

[0247] Step S34: Detect camera movement

[0248] As a result of step S32 or step S33 above, movement of an image capture device 10 is detected.

[0249] Step S35: Acquire image

[0250] As the image capture device moves, calibration is required. Therefore, utility server 8 requests image data in its post-movement state from image creation controller 1.

[0251] Step S36: Cut and edit

[0252] In response to an image acquisition request from utility server 8, image creation controller 1 controls video servers 4A, 4B, 4C and 4D to perform clipping on image data V1 to V16.

[0253] Step S37: NAS Transfer

[0254] Image creation controller 1 controls video servers 4A, 4B, 4C, and 4D to transmit image data, which is thus cut into clips, to NAS 5.

[0255] Step S38: Correct feature points

[0256] The transmission to NAS 5 allows the utility server 8 to reference and also displays images in the state after camera movement. Operator OP2 performs the necessary calibration operations, such as feature point correction.

[0257] Step S39: Recalibrate

[0258] The utility server 8 performs calibration again to create a 3D model using image data (V1-V16) from the state after the camera has moved.

[0259] Step S40: Reacquire the background

[0260] After calibration, in response to operator OP2's operation, utility server 8 requests to reacquire image data for the background image.

[0261] Step S41: Cut and edit

[0262] In response to an image acquisition request from utility server 8, image creation controller 1 controls video servers 4A, 4B, 4C and 4D to perform clipping on image data V1 to V16.

[0263] Step S42: NAS Transfer

[0264] Image creation controller 1 controls video servers 4A, 4B, 4C, and 4D to transmit image data, which is thus cut into clips, to NAS 5.

[0265] Step S43: Generate background

[0266] The utility server 8 generates a background image using image data transmitted to the NAS 5. This is used, for example, as a background image to serve as a reference for subsequent detection of camera movement.

[0267] For example, camera movement detection and calibration are performed as described above, so that, for example, even if the position of the image capture device 10 moves during broadcasting, the parameters are corrected accordingly, making it possible to continuously generate FV clips with good accuracy.

[0268] <7. Conclusions and Examples of Changes>

[0269] The above embodiments have the following effects.

[0270] The information processing apparatus 70 used as the image creation controller 1 according to the embodiment includes: an interval recognition processing unit 21, configured to perform processing for identifying a generation target image interval for generating a free viewpoint image from a plurality of captured images (e.g., image data V1 to V16) simultaneously captured by a plurality of image capture devices 10; a target image transmission control unit 22, configured to perform control to transmit image data of the generation target image interval in each of the plurality of captured images (V1 to V16) as image data for generating an FV clip in the free viewpoint image server 2; and an output image generation unit 23, configured to generate an output clip including the received FV clip.

[0271] Furthermore, the information processing apparatus 70 of the free-viewpoint image server 2 according to this embodiment includes: a target image acquisition unit 31, configured to acquire image data of a generation target image range for generating a free-viewpoint image from each of a plurality of captured images (V1 to V16) simultaneously captured by a plurality of image capture devices 10; an image generation processing unit 32, configured to generate a free-viewpoint image using the acquired image data; and a transmission control unit 33, configured to execute control to send the generated free-viewpoint image to the image creation controller 1.

[0272] In this configuration, Image Creation Controller 1 transmits only the image data of the target image region for FV clip generation to NAS 5. Free Viewpoint Image Server 2 generates an FV clip based on the 16 system image data transmitted to NAS 5 and sends the FV clip to Image Creation Controller 1. Image Creation Controller 1 generates an output clip that includes the FV clip.

[0273] Therefore, the amount of image data transmitted to NAS 5 (Free Viewpoint Image Server 2) is small, and the processing load on Free Viewpoint Image Server 2 is low, which enables the generation of FV clips in a short time. Furthermore, only the FV clips need to be sent to Image Creation Controller 1, reducing the processing load on Free Viewpoint Image Server 2, which is effective for rapid clip creation.

[0274] Typically, for example, in creating an output clip that includes FV clips, assuming the images of the entire scene for the output clip are sent to NAS 5 (Free Viewpoint Image Server 2), and processing to generate the entire output clip including previous and subsequent clips is performed on Free Viewpoint Image Server 2, this requires a relatively long processing time. For example, in a broadcast site, there is a request to broadcast a replay after 10 seconds of playback, but it is difficult to generate a clip including free viewpoint images in a short time.

[0275] On the other hand, in this embodiment, the free-viewpoint image server 2 has sufficient image data of the scene within the range of the FV clip to create the FV clip. This makes the operation of the embodiment very useful for shortening the time required to create the output clip.

[0276] For example, in the case of live sports broadcasts, it might be necessary to create an output clip within approximately 10 seconds, or to enable immediate broadcast playback. This is also advantageous for supporting such applications.

[0277] Furthermore, as the number of image capture devices 10 increases, free-viewpoint clips with higher precision can be created; therefore, it is desirable to configure as many image capture devices 10 as possible. With the increase in the number of image capture devices 10, the amount of data in the intervals used for FV clips increases. Therefore, it is more desirable to transmit shorter intervals of image data. For example, in the case of transmitting images captured by sixteen image capture devices 10 as in this embodiment, it is clear that transmitting image data in the timecode intervals (TC3-TC4: TC3=TC4 or TC3≠TC4) of the FV clip to the sixteen NAS 5 is extremely advantageous in terms of system processing efficiency compared to transmitting image data in the timecode intervals (TC1-TC6) of the entire output clip to the sixteen NAS 5.

[0278] According to this embodiment, the image creation controller 1 performs processing to notify the free viewpoint image server 2 of information about the identified target image region to be generated, and in response to a request from the free viewpoint image server 2, performs transmission control on the image data of the target image region to be generated.

[0279] Furthermore, in response to a notification to generate a target image region, the free viewpoint image server 2 of this embodiment requests multiple sets of image data for the generated target image region in each of the multiple captured images.

[0280] As a result, after the image creation controller 1 notifies the target image region, an FV clip can be generated based on the convenience of either the free-viewpoint image server 2 or the operator OP2. This prevents the processing on the free-viewpoint image server 2 side from becoming complicated and simplifies the FV clip generation process and the operation of operator OP2.

[0281] In this embodiment, an example is described in which the interval recognition processing unit 21 of the image creation controller 1 identifies an interval of a frame of a captured image as the interval for generating the target image. The target image acquisition unit 31 of the free viewpoint image server 2 acquires image data from each of the multiple captured images that corresponds to a frame of the interval for generating the target image. That is, this is the case of generating a still image FV clip.

[0282] In the case of still image FV clipping with a changed viewpoint while the image is in a still image state at a stopped time, the free viewpoint image server 2 only needs to acquire one frame each for image data V1 to image data V16 through the sixteen image capture devices 10. This can greatly reduce the processing load in the transmission processing to the NAS 5 and the processing in the free viewpoint image server 2.

[0283] In this embodiment, an example is described in which the interval recognition processing unit 21 of the image creation controller 1 identifies intervals of multiple frames of captured images as intervals for generating target images. The target image acquisition unit 31 of the free viewpoint image server 2 acquires image data of intervals of multiple frames corresponding to the intervals for generating target images from each of the multiple captured images. That is, this is the case of generating motion picture (FV) clips.

[0284] In the case of moving image FV clips where the viewpoint is changed while the motion image is in a state of continuous motion, the free-viewpoint image server 2 only needs to acquire multiple frames from the interval of motion image that is only required to become an FV clip for each of the sixteen image capture devices 10 for image data V1 to image data V16. Since the minimum necessary frame interval is obtained, unnecessary processing is not performed in the transmission processing to NAS 5 and the processing of the free-viewpoint image server 2.

[0285] In an embodiment, the output image generation unit 23 of the image creation controller 1 generates an output image by combining a previous or subsequent image with a free viewpoint image on the time axis.

[0286] Specifically, the output image generation unit 23 generates an output clip in which a previous or subsequent clip is combined with a received FV clip, such that, for example, an output clip in which the viewpoint moves in the middle of the scene can be generated. Furthermore, combining the previous and subsequent clips with the FV clip on the image creation controller 1 side reduces the processing load on the free-viewpoint image server 2, which is effective for rapid clip creation.

[0287] In an embodiment, the output image generation unit 23 of the image creation controller 1 generates an output image as a virtual clip obtained by virtually combining a free viewpoint image with a previous or subsequent image on the timeline based on playback list information.

[0288] Specifically, a playlist is created in which previous and subsequent clips are connected to FV clips. This allows the images of the clips to be output by playing them back along the playlist, without actually creating the output clip as an image data stream. This simplifies the processing on the image creation controller 1 side and reduces the time required to create the output clip, making it suitable for requests for rapid output clip creation.

[0289] In this embodiment, an example is described in which the transmission control unit 33 of the free viewpoint image server 2 executes control to send information specifying a previous or subsequent image to be connected to the generated free viewpoint image as supplementary information to the image creation controller 1. Then, the output image generation unit 23 of the image creation controller 1 obtains the information specifying the previous or subsequent image to be connected to the free viewpoint image from the free viewpoint image server 2, and generates an output image obtained by combining the previous or subsequent image with the free viewpoint image on the timeline based on the information.

[0290] An FV clip is an image in which the viewpoint changes from one viewpoint (i.e., an image from one image capture device 10) to another viewpoint (i.e., an image from another image capture device 10). That is, depending on the shooting style selected by the free-viewpoint image server 2, it can determine which image captured by which image capture device 10 will be used as the preceding and following image to be connected to the FV clip. Therefore, the image creation controller 1 can easily prepare appropriate images as preceding or following clips to be connected to the FV clip by obtaining information about the preceding or following clip from the free-viewpoint image server 2. These are then combined with each other on the timeline, allowing the generation of an output clip with good compatibility before and after the viewpoint movement.

[0291] Furthermore, since previous and subsequent clips can be identified based on shooting style information, the processing load on the Free Viewpoint Image Server 2 is not increased.

[0292] In this embodiment, an example is described in which the transmission control unit 33 of the free-viewpoint image server 2 executes control to send information specifying the time length of a previous or subsequent image to be connected to the generated free-viewpoint image as supplementary information to the image creation controller 1. Then, the output image generation unit 23 of the image creation controller 1 obtains the information specifying the time length of the previous or subsequent image to be connected to the free-viewpoint image from the free-viewpoint image server 2, prepares the previous or subsequent image based on the obtained information, and generates an output image obtained by combining the previous or subsequent image with the free-viewpoint image on the timeline.

[0293] Free Viewpoint Image Server 2 specifies the time lengths of the previous and subsequent clips to be connected to the FV clip representing viewpoint movement. This allows the generation of output clips with assumed time lengths from the Free Viewpoint Image Server 2 side. For example, operator OP2, who creates the FV clip, can set the time length of the output clip based on the content of the FV clip.

[0294] An example has been described in which, in the free viewpoint image server 2 according to this embodiment, the image generation processing unit 32 generates a free viewpoint image by using shooting style information selected from pre-stored shooting style information (viewpoint movement path information).

[0295] Multiple shooting style information entries are stored as indicators of viewpoint movement paths, and operator OP2 selects the shooting style information based on the specified scene. This allows FV clips to be generated solely by operator OP2 selecting shooting style information based on the content of the scene, which serves as the target image range. Therefore, FV clips with viewpoint movement (shooting style) based on the scene content can be created in a short time.

[0296] As an example, a device that enables a CPU, DSP, or other device including a CPU and DSP to perform the operation can be used. Figure 13 The processing procedure described in [the document].

[0297] Specifically, the program according to this embodiment is a program that causes the information processing device to perform the following processes: interval identification processing (S102, S103), for identifying a generation target image interval for generating a free viewpoint image from multiple captured images (V1 to V16) captured simultaneously by multiple image capture devices 10; target image transmission control processing (S105), for transmitting the image data of the generation target image interval in each of the multiple captured images as image data for generating a free viewpoint image in other information processing devices; and output image generation processing (S107, S108), for generating an output image including the received free viewpoint image.

[0298] Through such a procedure, the image creation controller 1 described above can be implemented in the device 70, which is an information processing device.

[0299] Furthermore, as an example, a device that enables a CPU, DSP, or includes a CPU and DSP can be used to perform the operation. Figure 14 The processing procedure described in [the document].

[0300] Specifically, the program according to this embodiment is a program that causes the information processing device to perform the following processes: image data acquisition processing (S202, S203), for acquiring image data of a generation target image interval for generating a free viewpoint image from each of a plurality of captured images (V1 to V16) simultaneously captured by a plurality of image capture devices 10; image generation processing (S204 to S209), for generating a free viewpoint image by using the image data acquired in the image data acquisition processing; and transmission control processing (S212), for performing control to transmit the generated free viewpoint image to other information processing devices.

[0301] Through such a procedure, the aforementioned free-viewpoint image server 2 can be implemented in the device that serves as the information processing unit 70.

[0302] These programs can be pre-recorded in an HDD as a recording medium built into devices such as computer equipment or ROM in a microcomputer with a CPU.

[0303] Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium, such as a floppy disk, optical disc read-only memory (CD-ROM), magneto-optical disc (MO), digital versatile optical disc (DVD), Blu-ray disc (registered trademark), magnetic disk, semiconductor memory, or memory card. Such a removable recording medium can be provided as a so-called software package.

[0304] Furthermore, such programs can be installed on personal computers from removable recording media, or downloaded from download sites via networks such as local area networks (LANs) or the Internet.

[0305] Furthermore, such a program is suitable for widespread provision to the image creation controller 1 and the free-viewpoint image server 2 of this embodiment. For example, by downloading the program to a personal computer, portable information processing device, mobile phone, gaming device, video device, personal digital assistant (PDA), etc., the personal computer, etc., can be used as the image creation controller 1 or the free-viewpoint image server 2 of this disclosure.

[0306] Note that the effects described in this manual are merely illustrative and not limiting, and other effects may also exist.

[0307] Note that this technology can also be configured as follows. (1)

[0309] An information processing device, comprising:

[0310] The interval recognition processing unit is configured to perform processing on multiple captured images simultaneously captured by multiple image capture devices to identify the generated target image interval for generating a free viewpoint image.

[0311] A target image transmission control unit is configured to perform control to transmit image data of the generated target image region in each of the plurality of captured images as image data for the generation of a free-viewpoint image in other information processing devices; and

[0312] The output image generation unit is configured to generate an output image that includes the received free viewpoint image. (2)

[0314] According to the information processing apparatus described in (1) above, wherein,

[0315] The interval recognition processing unit performs processing to notify the other information processing devices of information about the recognized interval for generating the target image, and

[0316] The target image transmission control unit, in response to a request from the other information processing device, performs transmission control of image data for generating the target image region. (3)

[0318] According to the information processing apparatus described in (1) or (2) above, wherein,

[0319] The interval recognition processing unit identifies an interval of a frame of the captured image as the interval for generating the target image. (4)

[0321] According to the information processing apparatus described in (1) or (2) above, wherein,

[0322] The interval recognition processing unit identifies intervals from multiple frames of the captured image as intervals for generating the target image. (5)

[0324] The information processing apparatus according to any one of (1) to (4) above, wherein,

[0325] The output image generation unit generates an output image by combining a previous or subsequent image with a free viewpoint image on the time axis. (6)

[0327] The information processing apparatus according to any one of (1) to (5) above, wherein,

[0328] The output image generation unit generates an output image as a virtual clip obtained by virtually combining a free-viewpoint image with a previous or subsequent image on the timeline based on playback list information. (7)

[0330] The information processing apparatus according to any one of (1) to (6) above, wherein,

[0331] The output image generation unit obtains information from the other information processing device about a previous or subsequent image to be connected to the free viewpoint image, and generates an output image by combining the previous or subsequent image with the free viewpoint image on the time axis based on the obtained information. (8)

[0333] The information processing apparatus according to any one of (1) to (7) above, wherein,

[0334] The output image generation unit obtains information from the other information processing device regarding the time length of a previous or subsequent image to be connected to the free viewpoint image, prepares the previous or subsequent image based on the obtained information, and generates an output image obtained by combining the previous or subsequent image with the free viewpoint image on the time axis. (9)

[0336] An information processing device, comprising:

[0337] The target image acquisition unit is configured to acquire image data of the target image region for generating a free viewpoint image from each of a plurality of captured images captured simultaneously by a plurality of image capture devices.

[0338] The image generation processing unit is configured to generate a free-viewpoint image using image data acquired by the target image acquisition unit; and

[0339] The transmission control unit is configured to perform control over transmitting the generated free-viewpoint image to other information processing devices. (10)

[0341] According to the information processing apparatus described in (9) above, wherein,

[0342] The target image acquisition unit, in response to a notification to generate a target image interval, requests multiple sets of image data for the generated target image interval from each of the plurality of captured images. (11)

[0344] According to the information processing apparatus described in (9) or (10) above, wherein,

[0345] The target image acquisition unit acquires image data of a frame corresponding to the range of the target image generated from each of the plurality of captured images. (12)

[0347] According to the information processing apparatus described in (9) or (10) above, wherein,

[0348] The target image acquisition unit acquires image data of each of the plurality of captured images, corresponding to a range of frames in the range of generated target images. (13)

[0350] The information processing apparatus according to any one of (9) to (12) above, wherein,

[0351] The image generation processing unit generates a free viewpoint image by using viewpoint movement path information selected from pre-stored viewpoint movement path information. (14)

[0353] The information processing apparatus according to any one of (9) to (13) above, wherein,

[0354] The transmission control unit performs control to transmit information specifying a previous or subsequent image to be connected to the generated free-viewpoint image to the other information processing device. (15)

[0356] The information processing apparatus according to any one of (9) to (14) above, wherein,

[0357] The transmission control unit performs control to transmit information specifying the time length of a previous or subsequent image to be connected to the generated free-viewpoint image to the other information processing device. (16)

[0359] An image processing system includes a first information processing unit and a second information processing unit to output an output image including a free viewpoint image, wherein,

[0360] The first information processing device includes:

[0361] The interval recognition processing unit is configured to perform processing on multiple captured images simultaneously captured by multiple image capture devices to identify the generated target image interval for generating a free viewpoint image.

[0362] The target image transmission control unit is configured to perform control to transmit image data of the target image region in each of the plurality of captured images as image data for generating a free viewpoint image in the second information processing device; and

[0363] The output image generation unit is configured to generate an output image including the received free viewpoint image, and

[0364] The second information processing device includes:

[0365] The target image acquisition unit is configured to acquire image data of the generation target image range for generating a free viewpoint image of each of the plurality of captured images;

[0366] The image generation processing unit is configured to generate a free-viewpoint image using image data acquired by the target image acquisition unit; and

[0367] The transmission control unit is configured to perform control of transmitting the generated free-viewpoint image to the first information processing device. (17)

[0369] An information processing method, comprising:

[0370] The information processing device performs interval recognition processing, which is used to identify the generation target image interval for generating free viewpoint images from multiple captured images captured simultaneously by multiple image capture devices.

[0371] The information processing device performs target image transmission control processing, which transmits image data of the target image region generated in each of the plurality of captured images as image data for generating free-viewpoint images in other information processing devices; and

[0372] The information processing device performs an output image generation process to generate an output image that includes the received free viewpoint image. (18)

[0374] An information processing method, comprising:

[0375] The image data acquisition process is performed by an information processing device. The image data acquisition process is used to acquire the image data of the target image range for generating a free viewpoint image from each of a plurality of captured images captured simultaneously by a plurality of image capture devices.

[0376] The information processing device performs image generation processing, which generates a free-viewpoint image using image data acquired in the image data acquisition processing; and

[0377] The transmission control process is performed by an information processing device, which controls the transmission of the generated free-viewpoint image to other information processing devices.

[0378] List of reference numerals

[0379] 1 Image Creation Controller

[0380] 2 Free Viewpoint Image Server

[0381] 3, 4, 4A, 4B, 4C, 4D video servers

[0382] 5NAS

[0383] 6 Switch

[0384] 7 Image Conversion Units

[0385] 8 Practical Servers

[0386] 10 Image Capture Devices

[0387] 21-interval identification processing unit

[0388] 22 Target Image Transmission Control Unit

[0389] 23 Output Image Generation Unit

[0390] 31 Target Image Acquisition Unit

[0391] 32 Image Generation and Processing Units

[0392] 33 Transmission Control Unit

[0393] 70 Information Processing Device

[0394] 71 CPU

[0395] 77 display units

Claims

1. An information processing apparatus, comprising: The interval recognition processing unit is configured to perform processing on multiple captured images simultaneously captured by multiple image capture devices to identify the generated target image interval for generating a free viewpoint image. The target image transmission control unit is configured to perform control to transmit image data of the generated target image region in each of the plurality of captured images as image data for the generation of a free viewpoint image in other information processing devices; and The output image generation unit is configured to generate an output image that includes the received free viewpoint image.

2. The information processing apparatus according to claim 1, wherein, The interval recognition processing unit performs processing to notify the other information processing devices of information about the recognized intervals for generating the target image, and The target image transmission control unit responds to a request from the other information processing device by performing transmission control of image data for generating the target image region.

3. The information processing apparatus according to claim 1, wherein, The interval recognition processing unit identifies an interval of a frame of the captured image as the interval for generating the target image.

4. The information processing apparatus according to claim 1, wherein, The interval recognition processing unit identifies intervals from multiple frames of the captured image as intervals for generating the target image.

5. The information processing apparatus according to claim 1, wherein, The output image generation unit generates an output image by combining a previous or subsequent image with a free viewpoint image on the time axis.

6. The information processing apparatus according to claim 1, wherein, The output image generation unit generates an output image as a virtual clip obtained by virtually combining a free-viewpoint image with a previous or subsequent image on the timeline based on playback list information.

7. The information processing apparatus according to claim 1, wherein, The output image generation unit obtains information from the other information processing device about a previous or subsequent image to be connected to the free viewpoint image, and generates an output image by combining the previous or subsequent image with the free viewpoint image on the time axis based on the obtained information.

8. The information processing apparatus according to claim 1, wherein, The output image generation unit obtains information from the other information processing device about the time length of a previous or subsequent image to be connected to the free viewpoint image, prepares the previous or subsequent image based on the obtained information, and generates an output image by combining the previous or subsequent image with the free viewpoint image on the time axis.

9. An image processing system, comprising a first information processing unit and a second information processing unit to output an output image including a free viewpoint image, wherein, The first information processing device includes: The interval recognition processing unit is configured to perform processing on multiple captured images simultaneously captured by multiple image capture devices to identify the generated target image interval for generating a free viewpoint image. The target image transmission control unit is configured to perform control to transmit image data of the target image region in each of the plurality of captured images as image data for generating a free viewpoint image in the second information processing device; and The output image generation unit is configured to generate an output image including the received free viewpoint image, and The second information processing device includes: The target image acquisition unit is configured to acquire image data of the generation target image range for generating a free viewpoint image of each of the plurality of captured images; The image generation processing unit is configured to generate a free-viewpoint image using image data acquired by the target image acquisition unit; and The transmission control unit is configured to perform control of transmitting the generated free-viewpoint image to the first information processing device.

10. An information processing method, comprising: The information processing device performs interval recognition processing, which is used to identify the generation target image interval for generating free viewpoint images from multiple captured images captured simultaneously by multiple image capture devices. The target image transmission control process is executed by the information processing device, which is used to transmit the image data of the target image generation interval in each of the plurality of captured images as image data for the generation of free viewpoint images in other information processing devices. and The information processing device performs an output image generation process to generate an output image that includes the received free viewpoint image.

Images