Information processing apparatus, information processing method, and program

Abstract

The information processing apparatus is provided with a display processing unit for executing a process of filtering a plurality of pieces of information to display, from among them, a screen corresponding to photographing style information of user input information as a photographing style specification screen that receives a specification operation of photographing style information that indicates a movement track of at least a viewpoint in a free viewpoint image.

Description

Technical Field

[0001] This technology relates to information processing equipment, information processing methods and programs, and in particular to technology for processing free-viewpoint images of a captured subject that can be observed from any viewpoint in three-dimensional space. Background Technology

[0002] The following techniques are known: generating free viewpoint images (also known as free viewpoint videos or virtual viewpoint images (videos) based on three-dimensional information representing the captured subject in three-dimensional space.

[0003] As relevant prior art, Patent Document 1 can be cited below. Patent Document 1 discloses a technique related to the generation of a shooting style (camerawork) that can be called a viewpoint movement trajectory.

[0004] Citation List

[0005] Patent documents

[0006] Patent Document 1: WO 2018 / 030206A Summary of the Invention

[0007] The problem that this invention aims to solve

[0008] Free-viewpoint images can also be used as broadcast content and are also used as replay images, for example, in sports broadcasts. For instance, in football or basketball broadcasts, a few-second clip, such as a shot, is created from live-recorded images and broadcast as a replay. Note that in this disclosure, "clip" refers to an image of a scene created by cutting out or further processing a scene from recorded images.

[0009] Simultaneously, in broadcast settings, especially live broadcasts, operators need to quickly create and broadcast clips for playback. For example, they might need to broadcast a 10-second replay after a previous playback. This requirement is similarly applied to the creation of clips including free-viewpoint images; therefore, the creation of free-viewpoint images needs to be performed rapidly.

[0010] In view of the above, this technology is proposed, and the purpose of this technology is to enable the rapid creation of free viewpoint images.

[0011] Solution to the problem

[0012] The information processing device according to this technology includes: a display processing unit, which performs processing to display a screen showing a shooting style information corresponding to user input information from multiple shooting style information through filtering, as a shooting style specified screen for receiving shooting style information for a specified operation, wherein the shooting style information is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image.

[0013] By filtering and displaying shooting style information based on user input, it is easy to find the shooting style information the user wants and shorten the time required to specify the shooting style information.

[0014] In the information processing device according to the present technology, the display processing unit can be configured to perform processing of filtering and displaying shooting style information corresponding to keywords as input information on a shooting style specified screen.

[0015] Therefore, it is possible to perform appropriate filtering of shooting style information that reflects the user's intent.

[0016] In the information processing device described above according to the present technology, it is possible to configure it such that filter condition information indicating filter conditions for shooting style information is displayed on the shooting style specified screen, and the display processing unit performs processing to filter and display shooting style information as input information according to the filter conditions indicated by the selected filter condition information.

[0017] As a result, the operations required to filter and display shooting style information can be reduced to simply selecting filter condition information.

[0018] In the information processing apparatus described above according to the present technology, the display processing unit can be configured to perform processing to display information obtained by visualizing the movement trajectory of the viewpoint on a screen specified by the shooting style.

[0019] By displaying information that visualizes the movement of the viewpoint, users can easily envision the shooting style.

[0020] In the information processing apparatus according to the present technology, the display processing unit can be configured to perform processing of displaying camera arrangement position information indicating the arrangement positions of multiple cameras on a shooting style specified screen, the multiple cameras performing imaging for generating free viewpoint images.

[0021] By displaying information indicating the placement of each camera, users can easily deduce which type of image should be generated as a free-viewpoint image.

[0022] In the information processing apparatus according to the present technology, the display processing unit can be configured to perform processing that displays, on a shooting style specified screen, starting point arrangement position information and ending point arrangement position information of a camera among a plurality of cameras that serves as the starting point of the viewpoint movement and a camera that serves as the ending point of the viewpoint movement.

[0023] As a result, it allows users to control where the viewpoint movement begins and ends in a shooting style.

[0024] In the information processing device according to the present technology, the display processing unit can be configured to perform processing to display, in different modes, the starting point arrangement location information and the ending point arrangement location information, as well as the arrangement location information of cameras other than the camera used as the starting point of movement and the camera used as the ending point of movement among a plurality of cameras.

[0025] As a result, it allows users to intuitively understand from which camera position the movement of the viewpoint begins and ends in the shooting style.

[0026] In the information processing apparatus described above according to the present technology, the display processing unit can be configured to perform processing to display information obtained by visualizing the movement speed of the viewpoint on a screen specified by the shooting style.

[0027] The time period during which the speed of viewpoint movement changes is an important factor in the rendering of free viewpoint images.

[0028] In the information processing device according to the present technology, the display processing unit can be configured to perform processing of information indicating a period of decreased movement speed as information obtained by visualizing the movement speed of the viewpoint.

[0029] The period during which the viewpoint's movement speed decreases is an important factor in the rendering of free-viewpoint images.

[0030] In the information processing apparatus described above according to the present technology, the display processing unit can be configured to perform processing to display information obtained by visualizing the field of view from the viewpoint on a screen specified by the shooting style.

[0031] Because the field of view is visually indicated, it makes it easier for users to grasp the shooting style.

[0032] In the information processing apparatus according to the present technology, the display processing unit can be configured to perform processing of displaying a target on a screen with a shooting style specified, which defines the direction of the line of sight from the viewpoint.

[0033] As a result, it allows users to easily determine the location of the subject in three-dimensional space as the target of the shooting style.

[0034] The information processing device described above according to the present technology can be configured to include a shooting style editing processing unit that updates the target position information in the shooting style information according to the change of the target position on the screen specified by the shooting style.

[0035] As a result, when you want to edit the shooting style information at a stage where you specify the shooting style information used to generate free-viewpoint images, it is not necessary to start the software used to generate the shooting style information.

[0036] In the information processing apparatus described above according to the present technology, the display processing unit can be configured to perform processing to display an image obtained by observing three-dimensional space from a viewpoint on a screen specified by the shooting style.

[0037] As a result, images similar to free-viewpoint images generated based on shooting style information can be displayed to users as previews, facilitating the understanding of shooting styles.

[0038] In the information processing apparatus described above according to the present technology, the display processing unit can be configured to perform processing to display an image obtained by rendering a virtual three-dimensional model of real space as an image obtained by viewing three-dimensional space from a viewpoint.

[0039] As a result, when a preview display of the observed image from the viewpoint is achieved, it is not necessary to perform rendering processing using a 3D model generated from a captured image from the target's real space.

[0040] In the information processing apparatus according to the present technology, the display processing unit can be configured to perform processing to display information of a camera among a plurality of cameras that has detected a change in field of view.

[0041] When generating free-viewpoint images, in order to accurately generate 3D information from images captured by multiple cameras, each camera must maintain an assumed position and orientation beforehand, and the parameters used to generate 3D information need to be calibrated if any camera's position or orientation changes. By notifying the camera that detects a change in the field of view, as described above, the user can be notified of the cameras that need calibration.

[0042] The information processing method according to this technology is as follows: an information processing device performs processing to display a screen that shows the shooting style information corresponding to the user's input information from multiple shooting style information through filtering, as a shooting style specified screen for receiving the specified operation of the shooting style information, wherein the shooting style information is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image.

[0043] Furthermore, by using this information processing method according to the present technology, the same effect as the information processing device according to the present technology described above can be achieved.

[0044] The program according to this technology is a program that can be read by a computer device, and is a function that enables the computer device to perform the following processing: displaying a screen that indicates shooting style information corresponding to user input information from multiple shooting style information through filtering as a shooting style specified screen for receiving the specified operation of the shooting style information, wherein the shooting style information is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image.

[0045] This procedure enables the implementation of the information processing device based on this technology. Attached Figure Description

[0046] Figure 1 This is a block diagram of a system configuration according to an embodiment of the present technology.

[0047] Figure 2 This is an illustrative diagram illustrating an example configuration of a camera for generating free-viewpoint images according to an embodiment.

[0048] Figure 3 This is a block diagram of the hardware configuration of an information processing device according to an embodiment.

[0049] Figure 4 This is an explanatory diagram illustrating the function of the image creation controller according to an embodiment.

[0050] Figure 5 This is an explanatory diagram illustrating the function of a free-viewpoint image server according to an embodiment.

[0051] Figure 6 A and Figure 6 B is an explanatory diagram of the viewpoint in a free-viewpoint image according to an embodiment.

[0052] Figure 7 This is an explanatory diagram outlining the shooting style specified in the embodiment.

[0053] Figure 8 This is an explanatory diagram summarizing the creation operation screen according to an embodiment.

[0054] Figure 9 This is an explanatory diagram of the output clip according to an embodiment.

[0055] Figure 10 This is an explanatory diagram of an output clip including a still image FV clip according to an embodiment.

[0056] Figure 11 This is an explanatory diagram of an output clip including a motion picture (FV) clip according to an embodiment.

[0057] Figure 12 This is an illustrative diagram illustrating an example of an output clipped image according to an embodiment.

[0058] Figure 13 This is an illustrative diagram illustrating the workflow of creating a clip based on an embodiment.

[0059] Figure 14 This is an explanatory diagram illustrating the working process of camera change detection according to an embodiment.

[0060] Figure 15 This is a diagram showing the initial screen of the creation operation screen according to an embodiment.

[0061] Figure 16 This is an illustration illustrating an example of the operation used to obtain a list of camera presets.

[0062] Figure 17 This is an illustration of the changes in the background 3D model.

[0063] Figure 18 Similarly, there is an explanatory diagram showing the changes in the background 3D model.

[0064] Figure 19 This is an explanatory image for adding a shooting style entry.

[0065] Figure 20 Similarly, there are illustrative diagrams showing examples of the field of view of a specified camera and the observed image.

[0066] Figure 21 Similarly, there are illustrative diagrams showing examples of the field of view of a specified camera and the observed image.

[0067] Figure 22 This is an illustrative diagram illustrating an example of a method for specifying an internal camera.

[0068] Figure 23 This is an illustration showing an example of the screen display when a specific camera is used.

[0069] Figure 24 This is an illustrative diagram illustrating an example of a method for specifying an external camera.

[0070] Figure 25 This is an illustration showing an example of the screen display in the case of a specified external camera.

[0071] Figure 26 This is an explanatory diagram showing the changes in the screen display based on the operation of the search bar in the timeline operation unit.

[0072] Figure 27 This is an illustrative diagram illustrating an example of a method for specifying a viewpoint via a point.

[0073] Figure 28 Similarly, an illustrative diagram illustrating an example of a method for specifying a viewpoint via a point.

[0074] Figure 29This is an explanatory diagram illustrating an example of screen display when the viewpoint is specified via a point.

[0075] Figure 30 This is an explanatory diagram illustrating a screen display example when multiple transit points are specified.

[0076] Figure 31 This is an example diagram of a method used to describe the shape type of a movement trajectory for a specified viewpoint.

[0077] Figure 32 Similarly, here is an example diagram of a method for describing the shape type of a movement trajectory for a specified viewpoint.

[0078] Figure 33 This is an explanatory diagram illustrating an example of screen display when the shape type of the movement trajectory of a specified viewpoint is used.

[0079] Figure 34 This is a diagram illustrating an example of a method used to describe the movement speed of a specified viewpoint.

[0080] Figure 35 This is an explanatory diagram illustrating an example of screen display when the viewpoint moves at a specified speed.

[0081] Figure 36 This is an explanatory diagram illustrating the significance of the target in the embodiment.

[0082] Figure 37 Similarly, there is an illustrative diagram illustrating the meaning of the target in the embodiments.

[0083] Figure 38 This is an explanatory diagram specifying the movement of the target according to an embodiment.

[0084] Figure 39 Similarly, there are specified illustrations of the movement of the target according to the embodiments.

[0085] Figure 40 Similarly, there are specified illustrations of the movement of the target according to the embodiments.

[0086] Figure 41 Similarly, there are specified illustrations of the movement of the target according to the embodiments.

[0087] Figure 42 Similarly, there are specified illustrations of the movement of the target according to the embodiments.

[0088] Figure 43 This is an illustrative diagram illustrating an exemplary operation that specifies a target-oriented time period.

[0089] Figure 44 Similarly, there is an illustrative diagram illustrating an exemplary operation that specifies a target-oriented time period.

[0090] Figure 45 Similarly, there is an illustrative diagram illustrating an exemplary operation that specifies a target-oriented time period.

[0091] Figure 46 This is a diagram showing an example of a preview image displaying a specified target in motion.

[0092] Figure 47 Similarly, this is an example of a preview image showing a specified target motion.

[0093] Figure 48 Similarly, this is an example of a preview image showing a specified target motion.

[0094] Figure 49 Similarly, this is an example of a preview image showing a specified target motion.

[0095] Figure 50 This is a diagram illustrating another example of a target-oriented time period.

[0096] Figure 51 This is an illustrative diagram illustrating examples of specifying time periods for multiple added targets.

[0097] Figure 52 It is a flowchart of the process related to the generation and display of the movement trajectory of the viewpoint specified by the in-camera and the out-camera.

[0098] Figure 53 This is a flowchart illustrating the processes related to the generation and display of a movement trajectory based on a specified viewpoint via a point.

[0099] Figure 54 This is a diagram showing the initial frame of the shooting style specified according to the embodiment.

[0100] Figure 55 This is an illustrative diagram illustrating an example of an image import method used to generate free-viewpoint images.

[0101] Figure 56 Similarly, an illustrative diagram illustrating an example of an image import method for generating free-viewpoint images.

[0102] Figure 57 This is an illustration showing an example of how the screen displays after an image is imported.

[0103] Figure 58 This is a diagram showing an example of an image displayed from the X-axis viewpoint.

[0104] Figure 59 This is a diagram showing an example of an image displayed from a Y-axis viewpoint.

[0105] Figure 60 This is a diagram showing an example of an image from a Z-axis viewpoint.

[0106] Figure 61 This is a diagram showing an example of an image displayed from the Pe viewpoint.

[0107] Figure 62 This is a diagram showing an example of an image from viewpoint Ca.

[0108] Figure 63 This is a diagram illustrating an example of the process for filtering and displaying shooting styles.

[0109] Figure 64 This is an illustration showing examples of filtering and displaying shooting styles.

[0110] Figure 65 This is an explanatory diagram of the reset button of the filtering operation unit according to an embodiment.

[0111] Figure 66 This is an explanatory diagram illustrating a change example of the filtering operation unit.

[0112] Figure 67 This is an explanatory diagram of another variation of the filtering operation unit.

[0113] Figure 68 This is a diagram illustrating a display example of visual information showing the speed of viewpoint movement.

[0114] Figure 69 This is another example of a display showing the visualization of the viewpoint's movement speed.

[0115] Figure 70 This is an explanatory diagram illustrating the editing of the target position on the screen according to the shooting style of this embodiment.

[0116] Figure 71 Similarly, this is an explanatory diagram illustrating the editing of the target position on the screen according to the shooting style of this embodiment.

[0117] Figure 72 This is an illustration showing an example of a notification message from a camera that has detected a change.

[0118] Figure 73 This is a flowchart illustrating the processing related to filtering based on the label information displayed on the screen.

[0119] Figure 74 This is a flowchart illustrating the processing related to filtering based on the shooting style corresponding to the input keywords.

[0120] Figure 75This is a flowchart illustrating the process related to a notification that a camera needs calibration. Detailed Implementation

[0121] The following exemplary embodiments are described in the following order.

[0122] <1. System Configuration>

[0123] <2. Configuration of Image Creation Controller and Free Viewpoint Image Server>

[0124] <3. Overview of the GUI>

[0125] <4. Clipping including free-viewpoint images>

[0126] <5. Clip Creation Processing>

[0127] <6. Camera Change Detection>

[0128] <7. GUI for creating shooting styles>

[0129] <8. GUI for creating free-viewpoint images>

[0130] <9. Change Example>

[0131] <10. Overview of the Implementation Examples>

[0132] <11. This technology>

[0133] <1. System Configuration>

[0134] Figure 1 An example configuration of an image processing system according to an embodiment of the present technology is shown.

[0135] The image processing system includes an image creation controller 1, a free-viewpoint image server 2, a video server 3, multiple (e.g., four) video servers 4A, 4B, 4C and 4D, a network attached storage (NAS) 5, a switcher 6, an image conversion unit 7, a utility server 8, and multiple (e.g., sixteen) imaging devices 10.

[0136] Hereinafter, the term "camera" refers to imaging device 10. For example, "camera arrangement" means the arrangement of multiple imaging devices 10.

[0137] In addition, when video servers 4A, 4B, 4C and 4D are collectively referred to without any particular distinction, they are called "video server 4".

[0138] In this image processing system, a free viewpoint image corresponding to an observation image from any viewpoint in three-dimensional space can be generated based on captured images (e.g., image data V1 to V16) acquired from multiple imaging devices 10, and an output clip including the free viewpoint image can be created.

[0139] exist Figure 1 In the diagram, the connection status of each part is represented by solid lines, dashed lines, and double lines.

[0140] Solid lines represent Serial Digital Interface (SDI) connections, an interface standard used to connect broadcast equipment such as cameras and switchers, and supporting, for example, 4K. Image data is primarily transmitted and received between devices via SDI cabling.

[0141] Double lines indicate connections using communication standards for building computer networks (such as 10G Ethernet). The image creation controller 1, free-viewpoint image server 2, video server 3, 4A, 4B, 4C and 4D, NAS 5, and utility server 8 are connected by a computer network, enabling them to send and receive image data and various control signals to each other.

[0142] The dashed line between video servers 3 and 4 indicates that video servers 3 and 4, equipped with inter-server file sharing capabilities, are connected via, for example, a 10G network. As a result, each video server can preview and send materials from other video servers, including video servers 4A, 4B, 4C, and 4D. That is, a system using multiple video servers can be built, enabling efficient highlighting, editing, and sending.

[0143] Each imaging device 10 is configured, for example, as a digital camera device including imaging elements such as a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor, and acquires captured images (image data V1 to V16) as digital data. In this example, each imaging device 10 acquires captured images as moving images.

[0144] In this example, each imaging device 10 captures images of a game being played, such as basketball or football, and each imaging device is arranged in a predetermined orientation at a predetermined position within the playing field. In this example, the number of imaging devices 10 is sixteen, but the number can be at least two or more to enable the generation of free-viewpoint images. By increasing the number of imaging devices 10 and imaging the target subject from more angles, the accuracy of the 3D reconstruction of the subject can be improved, and the image quality of the virtual viewpoint image can be enhanced.

[0145] Figure 2An example of the arrangement of imaging device 10 around a basketball court is shown. Let the circle (○) represent imaging device 10. For example, this is an example of camera arrangement in the case where a shot is expected near the goal on the left side of the image. Of course, the arrangement and number of cameras are examples and should be determined according to the content and purpose of the photography and broadcasting.

[0146] The image creation controller 1 includes an information processing device. For example, the image creation controller 1 can be implemented using a dedicated workstation, a general-purpose personal computer, or a mobile terminal device.

[0147] Image creation controller 1 performs control / action management of video servers 3 and 4, as well as processing for creating clips.

[0148] As an example, image creation controller 1 is a device that can be operated by operator OP1. For example, operator OP1 selects clip content and gives instructions to create the clip.

[0149] The free-viewpoint image server 2 is an information processing device configured to actually perform the processing of creating free-viewpoint images (free-viewpoint (FV) clips described later) according to instructions from the image creation controller 1, etc. The free-viewpoint image server 2 can also be implemented using, for example, a dedicated workstation, a general-purpose personal computer, or a mobile terminal device.

[0150] As an example, the free-viewpoint image server 2 is a device that can be operated by operator OP2. Operator OP2 performs tasks related to, for example, creating FV clips as free-viewpoint images. Specifically, operator OP2 performs a specified operation (selection operation) for generating the shooting style of the free-viewpoint image. In this example, operator OP2 also performs the task of creating the shooting style.

[0151] The configuration and processing of Image Creation Controller 1 and Free Viewpoint Image Server 2 are described in detail below. Additionally, operators OP1 and OP2 perform operations, but for example, Image Creation Controller 1 and Free Viewpoint Image Server 2 can be arranged side-by-side and operated by a single operator.

[0152] Each of the video servers 3 and 4 is an image recording device and includes a data recording unit such as a solid-state drive (SSD) or hard disk drive (HDD) and a control unit for controlling the data recording and reproduction of the data recording unit.

[0153] Each of the video servers 4A, 4B, 4C, and 4D can receive input from, for example, four systems, and each video server simultaneously records images captured by the four imaging devices 10.

[0154] For example, video server 4A records image data V1, V2, V3, and V4. Video server 4B records image data V5, V6, V7, and V8. Video server 4C records image data V9, V10, V11, and V12. Video server 4D records image data V13, V14, V15, and V16.

[0155] As a result, all images captured by the sixteen imaging devices 10 are recorded simultaneously.

[0156] Video servers 4A, 4B, 4C, and 4D, for example, perform continuous recording during a sporting event to be broadcast.

[0157] Video server 3 is connected directly to image creation controller 1, for example, and can receive input from both systems and output data to both systems. Image data Vp and Vq are shown as inputs to both systems. As image data Vp and Vq, images captured by any two imaging devices 10 (any two of image data V1 to V16) can be selected. Of course, the captured image can be an image captured by another imaging device.

[0158] The image creation controller 1 can display image data Vp and Vq on a monitor as a monitoring image. Operator OP1 can, for example, view the scene captured and recorded for broadcast by inputting the image data Vp and Vq into the video server 3.

[0159] In addition, since video servers 3 and 4 are connected to a file-sharing system, image creation controller 1 can monitor and display images captured by imaging device 10 and recorded on video servers 4A, 4B, 4C and 4D, and operator OP1 can view the captured images sequentially.

[0160] Note that in this example, timecode is appended to the images captured by each imaging device 10, and frame synchronization can be achieved during processing in video servers 3, 4A, 4B, 4C, and 4D.

[0161] NAS 5 is a storage device deployed on the network and includes, for example, SSDs or HDDs. In this example, NAS 5 is a device that stores or stores the free-viewpoint images created for processing in the free-viewpoint image server 2 when transmitting some frames of image data V1, V2, ..., V16 recorded in video servers 4A, 4B, 4C, and 4D to generate free-viewpoint images.

[0162] Switcher 6 is a device that takes in images output via video server 3 and selects the main image PGMout to be ultimately selected and broadcast. For example, broadcast directors and others perform necessary operations.

[0163] The image conversion unit 7 performs, for example, resolution conversion and synthesis of image data from the imaging device 10, to generate a monitoring image of the camera arrangement, and supplies the monitoring image to the utility server 8. For example, image data (V1 to V16) of sixteen systems intended to be 8K images are converted into four-system images arranged in a tiled shape after the resolution is converted to 4K images, and the four-system images are supplied to the utility server 8.

[0164] The utility server 8 is a computer device capable of performing various related processes. In this example, the utility server 8 is a device that performs the process of detecting camera movement for calibration. For example, the utility server 8 monitors image data from the image conversion unit 7 to detect camera movement. Camera movement is, for example, as... Figure 2 The movement of any one of the imaging devices 10 arranged as shown is monitored. Information about the arrangement position of the imaging devices 10 is an important factor in generating free-viewpoint images, and parameter settings need to be redone when the arrangement position changes. Therefore, camera movement is monitored.

[0165] <2. Configuration of Image Creation Controller and Free Viewpoint Image Server>

[0166] For example, the image creation controller 1, free-viewpoint image server 2, video servers 3 and 4, and utility server 8, configured as described above, can be implemented as having Figure 3 The information processing device 70 is configured as shown.

[0167] exist Figure 3 In the information processing device 70, the CPU 71 performs various processes according to the program stored in the ROM 72 or the program loaded into the RAM 73 from the storage unit 79. The RAM 73 also appropriately stores data and other data required by the CPU 71 to perform various processes.

[0168] CPU 71, ROM 72 and RAM 73 are interconnected via bus 74. Input / output interface 75 is also connected to bus 74.

[0169] The input unit 76, including the operator and operating equipment, is connected to the input / output interface 75.

[0170] For example, as input unit 76, various operators and operating devices are assumed, such as keyboards, mice, keys, dials, touch panels and touchpads and remote controls.

[0171] The user's operation is detected by the input unit 76, and the signal corresponding to the input operation is parsed by the CPU 71.

[0172] Furthermore, a display unit 77 configured with a liquid crystal display (LCD) or an organic electroluminescent (EL) panel and a sound output unit 78 configured with a speaker are integrated or separately connected to the input / output interface 75.

[0173] The display unit 77 is a display unit that performs various displays, and includes, for example, a display device disposed in the housing of the information processing device 70 or a separate display device connected to the information processing device 70.

[0174] The display unit 77 performs various types of image processing and displays motion images to be processed on the display screen based on instructions from the CPU 71. Additionally, the display unit 77 displays various operation menus, icons, and messages; that is, it displays a graphical user interface (GUI) based on instructions from the CPU 71.

[0175] In some cases, storage units 79, including hard disks or solid-state storage, and communication units 80 configured by modems or the like, are connected to input / output interfaces 75.

[0176] The communication unit 80 performs communication processing via transmission paths such as the Internet, wired / wireless communication with various devices, and bus communication.

[0177] The drive 82 is also connected to the input / output interface 75 as needed, and a removable recording medium 81, such as a disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately installed.

[0178] Data files such as image files (MF) and various computer programs can be read from the removable recording medium 81 via drive 82. The read data files are stored in storage unit 79, and the images and sounds contained in the data files are output by display unit 77 and sound output unit 78. Furthermore, computer programs and the like read from the removable recording medium 81 can be installed in storage unit 79 as needed.

[0179] In the information processing device 70, software can be installed via network communication through the communication unit 80 or the removable recording medium 81. Alternatively, the software can be pre-stored in the ROM 72 and the storage unit 79, etc.

[0180] When the image creation controller 1 and the free-viewpoint image server 2 are implemented using this information processing device 70, for example, it is implemented in the CPU 71 via software. Figure 4 and Figure 5 The processing functions shown.

[0181] Figure 4The diagram shows an interval recognition processing unit 21, a target image transmission control unit 22, and an output image generation unit 23, which are formed in the CPU 71 of the information processing device 70 that serves as the image creation controller 1.

[0182] The interval recognition processing unit 21 performs processing to identify the target image interval as the target of generating a free-viewpoint image from multiple captured images (image data V1 to V16) simultaneously captured by multiple imaging devices 10. For example, in response to operator OP1 performing an operation to select a scene to be played back in the image, it performs processing to specify the time code of the scene, in particular the interval of the scene to be become a free-viewpoint image (the target image interval), and notifies the free-viewpoint image server 2 of the time code.

[0183] Here, the target image interval refers to the frame interval actually used as the free viewpoint image. In the case of generating a free viewpoint image for a frame in a moving image, that frame is the target image interval. In this case, the in-point and out-point of the free viewpoint image have the same timecode.

[0184] Furthermore, when generating free-viewpoint images for intervals of multiple frames in a moving image, these multiple frames are the intervals of the target image to be generated. In this case, the in-point and out-point of the free-viewpoint image are different timecodes.

[0185] Note that although the structure of the clips is described later, it is assumed that the in / out points of the generated target image region are different from the in / out points of the final generated output clip. This is because the pre-clip and post-clip described later are coupled.

[0186] The target image transmission control unit 22 executes control to transmit image data (i.e., one or more frames of image data V1 to V16) from each of the multiple captured images 10 as image data for generating free-viewpoint images in the free-viewpoint image server 2. Specifically, control is executed to transmit image data as generated target image intervals from video servers 4A, 4B, 4C, and 4D to NAS 5.

[0187] The output image generation unit 23 performs the process of generating an output image (output clip), which includes a free viewpoint image (FV clip) generated and received by the free viewpoint image server 2.

[0188] For example, through the processing of the output image generation unit 23, the image creation controller 1 combines the pre-cut of the actual motion image at the previous time point and the post-cut of the actual motion image at the subsequent time point with the FV clip of the virtual image generated by the free viewpoint image server 2 on the time axis to obtain the output clip. That is, the pre-cut + FV clip + post-cut is set as one output clip.

[0189] Of course, pre-edit + FV edit can be a single output edit.

[0190] Alternatively, FV editing + post-editing can be an output clip.

[0191] In addition, output clips with only FV clips can be generated without coupling the pre- and post-cut clips.

[0192] In any case, the image creation controller 1 generates an output clip including an FV clip, outputs the output clip to the switcher 6, and can use the output clip for broadcasting.

[0193] Figure 5 The target image acquisition unit 31, image generation processing unit 32, transmission control unit 33, and shooting style generation processing unit 34 are shown as functions formed in the CPU 71 of the information processing device 70, which serves as the free viewpoint image server 2.

[0194] The target image acquisition unit 31 performs processing to acquire image data of the target image region from each of the multiple captured images (image data V1 to V16) simultaneously captured by multiple imaging devices 10, which serves as the target for generating a free-viewpoint image. That is, image data of one or more frames specified by the in-point / out-point of the target image region specified by the image creation controller 1 through the function of the region recognition processing unit 21 can be acquired from video servers 4A, 4B, 4C, and 4D via NAS 5 and used to generate a free-viewpoint image.

[0195] For example, the target image acquisition unit 31 acquires image data of one or more frames for generating the target image interval for all image data V1 to V16. To generate a high-quality free-viewpoint image, image data for generating the target image interval is acquired for all image data V1 to V16. As described above, a free-viewpoint image can be generated using images captured by at least two or more imaging devices 10. However, by increasing the number of imaging devices 10 (i.e., the number of viewpoints), a more detailed 3D model and a high-quality free-viewpoint image can be generated. Therefore, for example, in the case of sixteen imaging devices 10, image data for generating the target image interval is acquired for all image data (V1 to V16) from the sixteen imaging devices 10.

[0196] The image generation processing unit 32 is a function that generates a free viewpoint image (i.e., an FV clip in this example) by using image data acquired by the target image acquisition unit 31.

[0197] For example, the image generation processing unit 32 performs modeling processes including 3D model generation and subject analysis, as well as rendering processes such as generating a free-viewpoint image from the 3D model as a two-dimensional image.

[0198] 3D model generation is a process that generates 3D model data representing the subject in three-dimensional space (i.e., reconstructing the three-dimensional structure of the subject from a two-dimensional image) based on images captured by each imaging device 10 and camera parameters of each imaging device 10 input from a utility server 8, etc. Specifically, the 3D model data includes data representing the subject in a three-dimensional coordinate system (X, Y, Z).

[0199] In subject analysis, the position, orientation, and pose of the subject (athlete) are analyzed based on 3D model data. Specifically, this includes subject position estimation, generation of a simple model of the subject, and subject orientation estimation.

[0200] Then, a free-viewpoint image is generated based on the 3D model data and subject analysis information. For example, a free-viewpoint image is generated such that the viewpoint moves relative to the stationary 3D model of the athlete as the subject.

[0201] refer to Figure 6 A and Figure 6 B describes the viewpoint of a free-viewpoint image.

[0202] Figure 6 Image A shows a free-viewpoint image of a subject captured from a desired viewpoint set in three-dimensional space. In this free-viewpoint image, subject S1 is viewed from approximately the front, and subject S2 is viewed from approximately the back.

[0203] Figure 6 B indicates the position at the viewpoint. Figure 6 The direction of arrow C in A is changed, and a virtual viewpoint image is set for viewing the subject S1 from approximately behind. Figure 6 In the free-viewpoint image of B, the subject S2 is viewed from approximately the front, and it is shown that... Figure 6 A. Subject S3 (not shown) and a basketball goal.

[0204] For example, viewpoint from Figure 6 The state of A gradually moves in the direction of arrow C, and generates the state that leads to... Figure 6A 1- to 2-second image of state B is taken as a free viewpoint image (FV clip). Of course, the duration of the FV clip as a free viewpoint image and the trajectory of the viewpoint movement can be considered differently.

[0205] Here, the free viewpoint image server 2 (CPU 71) in this example has the function of display processing unit 32a as part of the function of image generation processing unit 32.

[0206] The display processing unit 32a performs processing to display a shooting style specifying screen Gs that receives shooting style information for generating a free-viewpoint image. Note that details of the shooting style and shooting style specifying screen Gs related to the free-viewpoint image will be described again later.

[0207] In addition, the free viewpoint image server 2 in this example, as part of the function of the image generation processing unit 32, also has the function of the shooting style editing processing unit 32b, but the function of the shooting style editing processing unit 32b will also be described later.

[0208] The transfer control unit 33 performs control over the transfer of the free-viewpoint image (FV clip) generated by the image generation processing unit 32 to the image creation controller 1 via the NAS 5. In this case, the transfer control unit 33 also controls the transfer of accompanying information for generating the output image to the image creation controller 1. It is assumed that the accompanying information specifies the images for the pre-clip and post-clip. That is, it specifies which image from image data V1 to V16 is used to create (cut out) the pre-clip and post-clip. Additionally, it is assumed that information specifying the duration of the pre-clip or post-clip is also included as accompanying information.

[0209] The shooting style generation processing unit 34 performs processing related to generating shooting style information for generating free-viewpoint images. When creating a free-viewpoint image, multiple candidate shooting styles are pre-created to address various scenarios. To achieve this pre-creation of shooting styles, a software program for creating shooting styles is installed in the free-viewpoint image server 2 in this example. The shooting style generation processing unit 34 is a function implemented by this software program and performs shooting style generation processing based on user input.

[0210] The shooting style generation processing unit 34 also functions as a display processing unit 34a. The display processing unit 34a performs processing to display the creation operation screen Gg, receiving various operation inputs from the user (operator OP2 in this example) to create the shooting style.

[0211] <3. Overview of the GUI>

[0212] refer to Figure 7 and Figure 8 This describes an overview of the shooting style specification screen Gs and the creation operation screen Gg used to create free-viewpoint images. In this example, the shooting style specification screen Gs and the creation operation screen Gg are displayed, for example, on display unit 77 in the free-viewpoint image server 2, and can be viewed and operated by operator OP2.

[0213] exist Figure 7 The shooting style specified screen Gs shown includes a scene window 41, a scene list display unit 42, a shooting style window 43, a shooting style list display unit 44, a parameter display unit 45, and a transmission window 46.

[0214] In scene window 41, for example, the monitoring display of the image generated in the target image range is performed, and operator OP2 can view the content of the scene that generates the free viewpoint image.

[0215] For example, a list of scenes designated as the target image range is displayed on the scene list display unit 42. Operator OP2 can select the scene to be displayed in the scene window 41 on the scene list display unit 42.

[0216] The shooting style window 43 displays the position of the imaging device 10, the selected shooting style, or multiple shooting styles that can be selected.

[0217] Here, shooting style information refers to information that indicates at least the movement trajectory of a viewpoint in a free-viewpoint image. For example, in the case of creating an FV clip where the position of the viewpoint, the direction of the gaze, and the angle of view (focal length) change relative to a subject with a generated 3D model, the parameters required to define the movement trajectory of the viewpoint, the pattern of change of the gaze direction, and the pattern of change of the angle of view are shooting style information.

[0218] In the shooting style window 43, at least information that visualizes and indicates the movement trajectory of the viewpoint is displayed as a shooting style display.

[0219] The shooting style list display unit 44 displays a list of information on various types of shooting styles that have been pre-created and stored. Operator OP2 can select and specify the shooting style to be used for FV clip generation from the shooting styles displayed on the shooting style list display unit 44.

[0220] Various parameters related to the selected shooting style are displayed on the parameter display unit 45.

[0221] In the transfer window 46, information related to transferring the created FV clip to the image creation controller 1 is displayed.

[0222] Next, the description Figure 8 The creation operation screen is Gg.

[0223] On the creation operation screen Gg, there are preset list display unit 51, shooting style list display unit 52, shooting style window 53, operation panel unit 54 and preview window 55.

[0224] The preset list display unit 51 can selectively display the preset list of the camera, the preset list of the target, and the preset list of the 3D model.

[0225] The camera preset list is a list of position information (position information in three-dimensional space) of each camera preset by the user for the on-site camera layout. As described below, when the camera preset list is selected, the information indicating the position of each identification information of the camera (e.g., camera1, camera2, ..., camera16) is displayed in a list form on the preset list display unit 51.

[0226] Furthermore, in the preset list of targets, a target refers to the location of a target whose viewing direction is determined from the viewpoint in the free viewpoint image. When generating the free viewpoint image, the viewing direction from the viewpoint is determined to be towards the target.

[0227] When a preset list of targets is selected, the preset list display unit 51 displays a list of multiple identification information about the targets preset by the user and information indicating the location of the targets.

[0228] In the following text, the target from which the line of sight is determined by the viewpoint in the free viewpoint image, as described above, is referred to as the "target Tg".

[0229] The preset list of 3D models is a list of preset 3D models to be displayed as the background of the shooting style window 43. When the preset list of 3D models is selected, the preset list display unit 51 displays a list of multiple identification information of the preset 3D models.

[0230] The shooting style list display unit 52 can display a list of multiple shooting styles created through the creation operation screen Gg, as well as information on newly created shooting styles through the creation operation screen Gg (entries described later).

[0231] In the shooting style window 53, at least information that visualizes and indicates the movement trajectory of the viewpoint is displayed as a shooting style display.

[0232] The operation panel unit 54 is an area for receiving various operation inputs for creating shooting styles.

[0233] In the preview window 55, the observation image from the viewpoint is displayed. When an operation to move the viewpoint along the movement trajectory is performed, the observation images from each viewpoint position along the movement trajectory are displayed sequentially in the preview window 55. Additionally, as described later, when an operation to specify a camera from the camera's preset list is performed while the preset list of cameras is displayed on the preset list display unit 51, the observation image from the camera's placement position is displayed in the preview window 55 in this example.

[0234] Note that this will be described again later. Figure 7 The details of the shooting style specification in the image Gs and the specific process of specifying the shooting style are shown. Figure 8 The creation process shown in the screenshot displays details of the Gg creation process and the specific steps involved in creating the shooting style.

[0235] <4. Clipping including free-viewpoint images>

[0236] Next, the description includes FV clipping as the output clip of a free viewpoint image.

[0237] Figure 10 The output clip is shown as an example of the state connecting the pre-clip, FV clip, and post-clip.

[0238] For example, the pre-edit is the actual motion image in the timecode interval TC1 to TC2 of a certain image data Vx in image data V1 to V16.

[0239] Furthermore, post-editing is the actual motion image within the timecode TC5 to TC6 interval of a certain image data Vy in image data V1 to V16.

[0240] It is generally assumed that image data Vx is the image data of the imaging device 10 before the start of the viewpoint movement via FV clipping, and image data Vy is the image data of the imaging device 10 at the end of the viewpoint movement via FV clipping.

[0241] In this example, the pre-cut is a motion picture with a time length of t1, the FV clip is a free-viewpoint image with a time length of t2, and the post-cut is a motion picture with a time length of t3. The total playback time of the entire output clip is t1 + t2 + t3. For example, as a 5-second output clip, one could consider 1.5 seconds of motion picture, 2 seconds of free-viewpoint image, and 1.5 seconds of motion picture, etc.

[0242] Here, FV clips are shown as the timecode interval TC3 to TC4, but this may or may not correspond to the number of frames in the actual motion picture.

[0243] That is, as an FV clip, there are cases where the viewpoint moves while the motion image is stopped (TC3=TC4) and cases where the viewpoint moves without stopping the motion image (TC3≠TC4).

[0244] For the purposes of this explanation, an FV clip in which the viewpoint moves while the motion image is stationary is called a "still image FV clip," and an FV clip in which the viewpoint moves without stopping the motion image is called a "moving image FV clip."

[0245] Figure 10 The image shows a still image FV clip of a frame that is a reference moving image. In this example, the timecodes TC1 and TC2 of the previous clip are the timecodes of frames F1 and F81, while the timecode of the subsequent frame F82 is... Figure 9 The timecodes TC3 and TC4 in the middle are the timecodes for frames F83 and F166, respectively. The timecodes TC5 and TC6 in the later edits are the timecodes for frames F83 and F166.

[0246] That is, this is the generation of a free viewpoint image where the viewpoint moves relative to a still image of a frame of frame F82.

[0247] Meanwhile, motion picture FV editing, such as Figure 11 As shown. In this example, the timecodes TC1 and TC2 of the pre-cut are the timecodes of frames F1 and F101, and the timecodes of frames F102 and F302 are... Figure 9 The timecodes TC3 and TC4 are in the middle. The timecodes TC5 and TC6 in the later edit are the timecodes for frames F303 and F503.

[0248] That is, this is the case of generating a free viewpoint image with viewpoint movement for motion images in the interval of multiple frames from frame F102 to frame 302.

[0249] Therefore, the target image region determined by the image creation controller 1 is used in the creation process. Figure 10 In the case of still image FV clips, it is a frame interval of frame F82, and in the creation Figure 11 In the case of motion picture FV clipping, it is a range of multiple frames from frame F102 to frame 302.

[0250] Figure 12 Show Figure 10 An example of the output clip image content of a still image FV clip.

[0251] exist Figure 12 In the sequence, the pre-cut is the actual motion image from frame F1 to frame F81. The FV cut is a virtual image showing the viewpoint moving within the scene at frame F81. The post-cut is the actual motion image from frame F83 to frame F166.

[0252] For example, output clips, including FV clips, are generated in this way and used as images to be broadcast.

[0253] <5. Clip Creation Processing>

[0254] The following is a description of... Figure 1 This is an example of output clip creation processing performed in an image processing system. It primarily describes the processing of image creation controller 1 and free-viewpoint image server 2.

[0255] First, refer to Figure 13 The description includes the workflow for operations performed by operators OP1 and OP2. Note that... Figure 13 The processing of operator OP1 in the diagram illustrates the GUI processing and operator operations of the image creation controller 1. Similarly, the processing of operator OP2 illustrates the GUI processing and operator operations of the free-viewpoint image server 2.

[0256] Step S1: Scene Selection

[0257] When creating an output clip, operator OP1 first selects the scene to be used as an FV clip. For example, operator OP1 searches for the scene to be used as an FV clip while monitoring the captured image displayed on display unit 77 on the image creation controller 1 side. Then, one or more frames of the target image range for generation are selected.

[0258] The information for generating the target image region is transmitted to the free viewpoint image server 2, and the operator OP2 can identify the generated target image region through the GUI on the display unit 77 on the free viewpoint image server 2 side.

[0259] Specifically, the information about the generated target image region is about Figure 9 The information in timecodes TC3 and TC4. As mentioned above, in the case of still image FV clips, the timecodes are TC3 = TC4.

[0260] Step S2: Scene image transfer command

[0261] In response to the specification of the target image range, operator OP2 executes an operation that gives instructions to transmit the images of the corresponding scene. In response to this operation, free-viewpoint image server 2 transmits a request to image creation controller 1 to transmit image data in the timecode TC3 and TC4 ranges.

[0262] Step S3: Simultaneous Cutout

[0263] In response to an image data transmission request, the image creation controller 1 controls video servers 4A, 4B, 4C, and 4D, and causes the video servers to cut out intervals of timecodes TC3 and TC4 for each of the sixteen system image data from image data V1 to image data V16.

[0264] Step S4: NAS Transfer

[0265] Then, the image creation controller 1 transmits the data in the intervals of all timecodes TC3 and TC4 of image data V1 to V16 to NAS 5.

[0266] Step S5: Thumbnail Display

[0267] Free Viewpoint Image Server 2 displays thumbnails of image data V1 to V16 in the timecode intervals TC3 and TC4 transmitted to NAS 5.

[0268] Step S6: On-site inspection

[0269] Operator OP2 views the scene content on the shooting style specified frame Gs via the free viewpoint image server 2, within the interval indicated by timecodes TC3 and TC4.

[0270] Step S7: Select Shooting Style

[0271] Operator OP2 selects (specifies) a suitable shooting style on the shooting style specification screen Gs based on the scene content.

[0272] Step S8: Generate Execution

[0273] After selecting the shooting style, operator OP2 performs the operation to generate the FV clip.

[0274] Step S9: Modeling

[0275] The free viewpoint image server 2 generates a 3D model of the subject and analyzes the subject by using frame data in the timecodes TC3 and TC4 intervals in the image data V1 to V16, as well as parameters such as the pre-inputted arrangement positions of each imaging device 10.

[0276] Step S10: Rendering

[0277] The free-viewpoint image server 2 generates a free-viewpoint image based on 3D model data and subject analysis information. At this point, generating the free-viewpoint image enables viewpoint movement based on the shooting style selected in step S7.

[0278] Step S11: Transfer

[0279] The free-viewpoint image server 2 transmits the generated FV clip to the image creation controller 1. At this time, as accompanying information, not only the FV clip can be transmitted, but also the specified information of the preceding and following clips, as well as the specified information of the duration of the preceding and following clips.

[0280] Step S12: Quality Inspection

[0281] Note that on the free-viewpoint image server 2 side, quality viewing by operator OP2 can be performed before or after the transmission in step S11. That is, the free-viewpoint image server 2 reproduces and displays the generated FV clip on the shooting style specified frame Gs, allowing operator OP2 to view the FV clip. In some cases, operator OP2 can also re-generate the FV clip without performing a transmission.

[0282] Step S13: Playlist Generation

[0283] Image creation controller 1 generates an output clip using the transmitted FV clip. In this case, one or both of the pre-clip and post-clip are coupled to the FV clip on the timeline to generate the output clip.

[0284] The output clip can be generated as streaming data, in which frames as the preceding clip, frames virtually generated as an FV clip, and frames as the following clip are actually connected in time sequence. However, in this processing example, the frames are virtually connected as a playlist.

[0285] That is, a playlist is generated such that the FV clip is reproduced by reproducing the frame intervals that are the preceding clips, and then the frame intervals that are the subsequent clips are reproduced, so that the output clip can be reproduced without generating actual streaming data that is connected as the output clip.

[0286] Step S14: Quality Inspection

[0287] The GUI on the image creation controller 1 side performs playback based on the playlist, and operator OP1 views the contents of the output clip.

[0288] Step S15: Reproduce the command

[0289] Operator OP1 issues a reproduction command based on quality confirmation and a pre-defined operation. Image creation controller 1 recognizes the input reproduction command.

[0290] Step S16: Reproduce

[0291] In response to a playback command, the image creation controller 1 supplies the output clip to the switcher 6. As a result, broadcasting of the output clip can be performed.

[0292] <6. Camera Change Detection>

[0293] In order to generate free-viewpoint images, since 3D models are generated using image data V1, V2, ..., V16, parameters including the position information of each imaging device 10 are important.

[0294] For example, if the position of an imaging device 10 moves in the middle of the broadcast or the imaging direction changes in the panning direction or tilt direction, the corresponding parameters need to be calibrated. Therefore, in Figure 1 In the image processing system, the utility server 8 detects changes in the camera. Here, a change in the camera means at least one change in the camera's position and imaging orientation.

[0295] refer to Figure 14 Describe the processing of image creation controller 1 and utility server 8 when detecting changes in the camera. Figure 14 With similar Figure 13 The format shows the processing procedure, but operator OP2 also performs operations on utility server 8.

[0296] Step S30: HD Output

[0297] The image creation controller 1 controls the image conversion unit 7 to output image data from video servers 4A, 4B, 4C, and 4D for camera motion detection. The images from video servers 4A, 4B, 4C, and 4D, i.e., the images from the sixteen imaging devices 10, undergo resolution conversion through the image conversion unit 7 and are then supplied to the utility server 8.

[0298] Step S31: Background Generation

[0299] The utility server 8 generates a background image based on the supplied image. Since the background image is an image that does not change unless there is a change in the camera, for example, a background image is generated for image data of the sixteen system (V1 to V16) that does not include subjects such as athletes.

[0300] Step S32: Check the differences

[0301] The background image is displayed as a GUI, allowing operator OP2 to view changes in the image.

[0302] Step S33: Automatic Change Detection

[0303] It can also automatically detect camera changes by performing comparative processing on the background image at various time points.

[0304] Step S34: Camera Change Detection

[0305] As a result of step S33 or step S32, a change in an imaging device 10 is detected.

[0306] Step S35: Image Acquisition

[0307] In response to the detection of changes in the imaging device 10, calibration is required. Therefore, the utility server 8 requests image data in the changed state from the image creation controller 1.

[0308] Step S36: Edit and cut out

[0309] Image creation controller 1 responds to image acquisition requests from utility server 8 by controlling video servers 4A, 4B, 4C, and 4D, and causes the video servers to perform clipping on image data V1 to V16.

[0310] Step S37: NAS Transfer

[0311] Image creation controller 1 controls video servers 4A, 4B, 4C, and 4D to transmit image data cut out as clips to NAS 5.

[0312] Step S38: Feature Point Correction

[0313] By transmitting the data to NAS 5, utility server 8 can reference and display images from the changed camera state. Operator OP2 performs the necessary calibration operations, such as feature point correction.

[0314] Step S39: Recalibrate

[0315] The utility server 8 re-executes the calibration used to create the 3D model by using image data (V1 to V16) under the changed camera conditions.

[0316] Step S40: Reacquire background

[0317] After calibration, in response to operator OP2's operation, utility server 8 requests to reacquire the image data of the background image.

[0318] Step S41: Edit and cut out

[0319] Image creation controller 1 responds to image acquisition requests from utility server 8 by controlling video servers 4A, 4B, 4C, and 4D, and causes the video servers to perform clipping on image data V1 to V16.

[0320] Step S42: NAS Transfer

[0321] Image creation controller 1 controls video servers 4A, 4B, 4C, and 4D to transmit image data cut out as clips to NAS 5.

[0322] Step S43: Background Generation

[0323] The utility server 8 generates a background image using image data transmitted to the NAS 5. This background image is used, for example, as a reference for subsequent camera change detection.

[0324] For example, by performing camera change detection and calibration as described above, parameters are corrected even if the position or imaging direction of the imaging device 10 changes during broadcasting, enabling the continuous generation of accurate FV clips.

[0325] <7. GUI for creating shooting styles>

[0326] The following is for reference. Figures 15-51 describe Figure 8 The screen showing the creation process (Gg) details, examples of the shooting style creation process, and various functions related to shooting style creation are displayed.

[0327] Figure 15 This is a diagram showing the initial screen of the creation operation screen Gg.

[0328] As described above, the preset list display unit 51, the shooting style list display unit 52, the shooting style window 53, the operation panel unit 54, and the preview window 55 are arranged on the creation operation screen Gg.

[0329] As shown in the figure, the preset list display unit 51 is equipped with a camera button B1, a target button B2, and a 3D model button B3. The camera button B1 is used to give instructions to display the preset list of the camera on the preset list display unit 51, and the target button B2 and the 3D model button B3 are used to give instructions to display the preset list of the target and the preset list of the 3D model as the background on the preset list display unit 51.

[0330] In the image, an underlined mark is shown in the camera button B1, indicating that the camera's preset list is selected.

[0331] The preset list display unit 51 has a folder reference button B4. By operating the folder reference button B4, the user can refer to the folder containing the data that they want to display as a list on the preset list display unit 51.

[0332] The "Create New" button B5 is located in the shooting style list display unit 52. Users can use the "Create New" button B5 to issue a command to add a new entry for a shooting style. The added shooting style entry is then displayed on the shooting style list display unit 52.

[0333] The shooting style window 53 includes an X-viewpoint button B6, a Y-viewpoint button B7, a Z-viewpoint button B8, a Ca-viewpoint button B9, and a Pe-viewpoint button B10. Each of these viewpoint buttons is used to indicate the observation viewpoint of the target to be displayed in the shooting style window 53. Specifically, the X-viewpoint button B6, Y-viewpoint button B7, and Z-viewpoint button B8 are used to indicate the X-axis, Y-axis, and Z-axis viewpoints, respectively, as viewpoints for observing the visualization information of the shooting style information displayed in the shooting style window 53. The Pe-viewpoint button B10 is used to indicate a mode that moves to change the observation viewpoint of the visualization information of the shooting style information to an arbitrary position. The Ca-viewpoint button B9 is used to give instructions to display an image obtained by observing the target's three-dimensional space through a movement trajectory defined as the shooting style information. Note that for images of the X-axis, Y-axis, Z-axis, Pe, and Ca viewpoints, refer to the description below. Figures 58-62 .

[0334] Here, in the creation operation screen Gg, for example, the displayed image in the shooting style window 53 or preview window 55 can be zoomed in or out according to predetermined operations such as mouse wheel operation. Furthermore, in the shooting style window 53 and preview window 55, the displayed image can be scrolled according to predetermined operations such as drag operation. Note that zooming in, zooming out, and scrolling of the displayed image can be performed according to the operation of buttons set on the screen.

[0335] The operation panel unit 54 is equipped with a replay button B11, a pause button B12, a stop button B13, a timeline operation unit 54a, a speed adjustment operation unit 56, and a trajectory shape adjustment operation unit 57.

[0336] Playback button B11, pause button B12, and stop button B13 are used to respectively indicate the playback, pause, and stop of the visualization image of the shooting style information displayed in the shooting style window 53 and the observation image from the viewpoint displayed in the preview window 55. Playback button B11, pause button B12, and stop button B13 are enabled at least when information about the movement trajectory of the viewpoint is determined to be shooting style information.

[0337] The timeline operation unit 54a is an area that receives operations related to the creation of a shooting style on the timeline representing the movement of the viewpoint in a free-viewpoint image. Examples of operations on the timeline operation unit 54a include dragging and dropping one of the cameras listed on the preset list display unit 51 to any position on the timeline (i.e., any point in time within the viewpoint movement period) (see [link to relevant documentation]). Figures 27-29 As described below, this operation is used to specify the timing of the viewpoint's position by dragging and dropping the camera during the viewpoint movement period.

[0338] Various operation buttons for adjusting the movement speed of the viewpoint are configured in the speed adjustment operation unit 56. Various operation buttons for adjusting the shape of the viewpoint's movement trajectory are configured in the trajectory shape adjustment operation unit 57.

[0339] The speed adjustment operation unit 56 and the trajectory shape adjustment operation unit 57 will be described later.

[0340] Describes the process of creating shooting styles and the various functions related to creating shooting styles.

[0341] First, the user (operator OP2 in this example) as follows: Figure 16 Perform the operation to obtain a preset list of cameras as shown. This operation is used to obtain a preset list of cameras that indicates the actual locations of the cameras installed on site.

[0342] Use the folder reference button B4 to specify the corresponding folder and obtain the camera's preset list.

[0343] When a specified folder is selected, the display processing unit 34a performs the process of displaying the camera's preset list on the preset list display unit 51 based on the data content of the specified folder.

[0344] Simultaneously, the display processing unit 34a performs processing to display visual instructions on the arrangement of each camera on the shooting style window 53 based on the acquired camera position information. Specifically, it performs processing to display camera position markers Mc indicating the positions of each camera.

[0345] Note that the display of camera location markers Mc can be implemented to identify each camera through color coding. For example, each camera can be color-coded in the camera preset list, and each camera location marker Mc can be displayed in the shooting style window 43 using the same color coding.

[0346] Furthermore, in the shooting style window 53, it is also conceivable to display the camera identification information (e.g., camera1, camera2, ..., cameraX, etc.) for the camera position marker Mc where the mouse is hovering.

[0347] In this example, the 3D model displayed as the background can be changed in the shooting style window 53.

[0348] When reference Figure 17 and Figure 18When describing changes to the background 3D model, if the user wishes to change the background 3D model, the user operates the 3D model button B3 to set the display state of the preset list display unit 51 to the preset list display state of the 3D model. In this display state, the default setting button B14, the grid setting button B15, and the N / A setting button B16, as shown in the figure, are displayed in the preset list display unit 51, and the user can switch the background 3D model by operating these buttons. Here, the default setting button B14 is used to indicate switching to a background 3D model that has been prepared as an initial setting (e.g., a 3D model indicating a stage or ground, etc.), and the grid setting button B15 is used to indicate switching to a background 3D model whose distance and angle can be visually identified (e.g., grid lines or squares, etc.). The N / A setting button B16 is used to indicate turning off the display of the background 3D model.

[0349] Figure 17 An example of the background 3D model of the shooting style window 53 is shown when the operation grid specification button B15 is pressed. Furthermore, Figure 18 This shows an example of the background 3D model of the shooting style window 53 when the default button B14 is operated.

[0350] When starting to create a shooting style, users such as Figure 19 Operate the newly created button B5 as shown.

[0351] In response to the operation of the newly created button B5, the display processing unit 34a displays a new entry for the shooting style on the shooting style list display unit 52. This entry displays an operation unit for specifying the inner camera as the starting point of the viewpoint movement and the outer camera as the ending point of the viewpoint movement.

[0352] In this example, during the generation of the free-viewpoint image, since the texture based on the image captured by the camera is pasted onto the 3D model of the subject, it is desirable to create as many movement trajectories as possible that traverse the camera position as possible as the viewpoint's movement trajectory. Specifically, since the start and end points of the viewpoint movement in the free-viewpoint image are the switching points with the preceding and following clipped images, the start and end points of the viewpoint movement should coincide with the camera position. Therefore, the start and end points of the viewpoint movement are specified as the camera positions of the inner and outer cameras, respectively.

[0353] Note that the starting and ending points of the viewpoint movement are not necessarily limited to the camera position, and can be any position other than the camera position.

[0354] As shown in the figure, the operation unit for specifying the internal and external cameras is, for example, an operation unit for specifying cameras in a drop-down format. When the user operates the drop-down command, information about each camera listed in the preset list of assignable cameras (in this example, camera number information) is displayed (see the description below). Figure 22 and Figure 24 ).

[0355] Furthermore, according to the operation of the newly created button B5, as described above, a new entry for the shooting style is displayed in the shooting style list display unit 52, and a marker indicating the position of the target Tg set by the user (hereinafter referred to as "target marker Mt") is displayed in the shooting style window 53.

[0356] Here, in the shooting style creation process, the position of the target Tg is set to an appropriate location assumed for the target scene. For example, in the case of wanting to generate an image of a shooting scene as a free-viewpoint image, this position is set to a location near the goal in the target's three-dimensional space (e.g., a football field in the case of a football). Here, it is assumed that the user can pre-set the position of the target Tg in the free-viewpoint image server 2.

[0357] As described above, a free-viewpoint image can be generated such that the line of sight from the viewpoint faces the target Tg. Specifically, the free-viewpoint image of this example can be generated such that the target Tg remains at a predetermined position (e.g., the center position) in the image frame for at least a portion of the time during the movement of the viewpoint.

[0358] Note that in the generation of free-viewpoint images, continuing to position the target Tg at a predetermined position in the image frame as described above is expressed as "following the target Tg". This "following the target Tg" is synonymous with the line of sight from the viewpoint continuing to face the target Tg as the viewpoint moves.

[0359] Here, with the camera's preset list displayed in the shooting style working window 53, the observation images from the field of view or viewpoint (images obtained by observing three-dimensional space from the viewpoint) are displayed in the shooting style window 53 and the preview window 55 respectively, based on the specified operation of the camera from the camera's preset list.

[0360] Specifically, Figure 20 and Figure 21 This shows an example of the creation operation screen Gg display when specifying a camera as camera1 and a camera as camera2 from the camera preset list.

[0361] In this scenario, the shooting style window 53 displays field-of-view information Fv, which is a visualization of the field of view from the camera's preset list, for the camera specified from the camera's preset list. As shown in the figure, in this example, information representing the field of view as a graphic is displayed as field-of-view information Fv.

[0362] Furthermore, in this shooting style window 53, the camera position marker Mc of the specified camera is highlighted compared to the camera position marker Mc of other cameras (an example with an enlarged size is shown in the figure), making it easy for the user to know where the specified camera is located.

[0363] In preview window 55, images obtained by observing three-dimensional space from a specified camera are displayed.

[0364] Here, in this example, it is assumed that the shooting style creation work is performed before generating the free viewpoint image. That is, it is assumed that the capture image for generating the free viewpoint image is performed in an unacquired state. Therefore, the image obtained by observing 3D space from the viewpoint mentioned here is not an image obtained by rendering a 3D model (hereinafter, for descriptive purposes, referred to as a "real 3D model") generated by performing object detection and other operations on images captured by each camera in the real space of the captured target as a 2D image, but rather an image obtained by rendering a virtual 3D model (referred to as a "virtual 3D model") that simulates the real space of the target as a 2D image. In the process of generating the observation image from the viewpoint in this case, since no capture image is acquired, the process of pasting the texture generated from the capture image onto the 3D model is not performed.

[0365] Figures 22-25 This is an illustration of the method for specifying the internal and external cameras.

[0366] like Figure 22 As shown, for the designation of the internal camera, the camera designation operation is performed from the drop-down list of internal cameras added to the shooting style list display unit 52.

[0367] Figure 23 The state of the creation operation screen Gg is shown when camera1 is designated as the internal camera. When camera1 is designated as the internal camera, as shown in the figure, the internal camera item is displayed as "1" in the entries added to the shooting style list display unit 52.

[0368] In addition, in the shooting style window 53, the camera position marker Mc of camera1 is highlighted, and the field of view information Fv of camera1 is displayed.

[0369] Note that, compared with the above Figure 20In contrast, the display modes of the camera position marker Mc and field of view information Fv in the shooting style window 53 may differ between cases where the camera is specified from the camera's preset list and cases where it is specified as an internal camera.

[0370] In preview window 55, an image obtained by observing three-dimensional space from camera 1 is displayed.

[0371] In the case of a designated external camera, such as Figure 24 As shown, camera-specified operations are performed from the drop-down list of external cameras added to the shooting style list display unit 52.

[0372] Figure 25 This shows the state of the creation operation screen Gg when camera9 is designated as an external camera. In this case, "9" is displayed in the external camera item in the entries added to the shooting style list display unit 52.

[0373] Here, the movement trajectory of the viewpoint is determined by specifying the inner and outer cameras. Therefore, in this case, the shooting style window 53 displays information indicating the movement trajectory of the viewpoint connecting the positions of the inner and outer cameras; this information is indicated as movement trajectory information Mm in the figure. Here, the movement trajectory information Mm is obtained by visualizing the movement trajectory of the viewpoint.

[0374] Specifically, in this case, in the shooting style window 53, the camera position marker Mc of camera9, designated as an external camera, is highlighted, and from... Figure 23 Additionally, the system displays linear movement trajectory information Mm for the positions of the connected internal and external cameras.

[0375] Although the illustrations are omitted, in the creation operation screen Gg, after specifying the inner and outer cameras as described above and forming the movement trajectory of the viewpoint, a preview of the shooting style can be displayed.

[0376] Specifically, the preview display can be started by operating the playback button B11 on the operation panel unit 54. In the shooting style window 53, as a preview display of the shooting style, an image showing the field of view information Fv changing constantly as the viewpoint moves is displayed. In addition, in conjunction with this preview display of the shooting style, the preview window 55 displays an observation image in three-dimensional space (an observation image from the viewpoint) that changes constantly as the viewpoint moves.

[0377] Furthermore, in this example, this preview display in shooting style and the preview display of the observed image in three-dimensional space can be performed not only by reproducing the operation of button B11, but also by dragging the search bar B17 in the timeline operation unit 54a.

[0378] Figure 26 This shows the state in which the search bar B17 is positioned at the desired location on the timeline by dragging the search bar B17 in the timeline operation unit 54a.

[0379] While the search bar B17 is being dragged, its position on the timeline (i.e., on the time axis from the start to the end of the free viewpoint image) changes constantly. Therefore, when the drag operation is performed, the field of view information Fv corresponding to the viewpoint position indicated by the search bar B17 is sequentially displayed in the shooting style window 53, and is perceived by the user as an image where the field of view information Fv changes constantly with the movement of the viewpoint. Similarly, in the preview window 55, an observation image of three-dimensional space from the constantly changing viewpoint is displayed according to the movement of the search bar B17.

[0380] Next, the points of reference for the viewpoint will be described.

[0381] On the creation operation screen Gg, you can specify the viewpoint's transit point and the timing of the viewpoint passing through the transit point.

[0382] Figures 27-30 This is an explanatory diagram specifying the viewpoint's transit point and transit time.

[0383] In this example, the timing of the viewpoint passing through the via point can be specified by dragging and dropping the operation of the camera that is expected to be designated as the via point on the timeline in the timeline operation unit 54a.

[0384] Figures 27-29 This is an illustrative diagram illustrating an example of operation when camera6 is specified as a via point.

[0385] First, such as Figure 27 As shown, a camera that is desired to be designated as a via point is selected from the camera preset list in the preset list display unit 51. In this example, the operation for selection is pressing the left mouse button.

[0386] Cameras selected in this way, such as Figure 28 Drag it on the screen as shown, and as... Figure 29 As shown, the camera is placed at the desired position on the timeline of the timeline operation unit 54a (in this example, the left-click press state is released). As a result, the timing of the camera as a via point and the viewpoint passing through the via point is completed.

[0387] In response to the specified completion, such as Figure 29As shown, the timeline of the timeline operation unit 54a displays via the dot marker Mv. The dot marker Mv is displayed at the position specified on the timeline by the aforementioned drop operation. That is, the free viewpoint image is displayed at the position of the specified timing within the time period from the start timing to the end timing of the free viewpoint image (within the time period of viewpoint movement).

[0388] In this example, the dot marker Mv is initially displayed as a square marker as shown in the figure.

[0389] Additionally, once the via point and via timing have been specified, the shooting style window 53 highlights the camera position marker Mc of the camera designated as the via point (camera 6 in this case) and displays field-of-view information Fv indicating the field of view from the camera. Furthermore, the preview window 55 displays an image obtained by observing three-dimensional space from the viewpoint of the camera position designated as the via point.

[0390] By selecting a camera from the preset list and dragging and dropping the selected camera as described above, you can specify the viewpoint's via point and the timing of the viewpoint passing through the via point.

[0391] Figure 30 The state of the creation operation screen Gg is shown when the via point and via timing are further specified for the two cameras in the same manner as described above.

[0392] Note that although an example of designating the camera position as the waypoint of the viewpoint is described here, any location other than the camera position can be designated as the waypoint.

[0393] In addition, the shape type of the viewpoint's movement trajectory can be specified on the creation operation screen Gg.

[0394] refer to Figures 31-33 Describe the specific operation process and the screen transitions based on the operation.

[0395] First, such as Figure 31 As shown, a target range for which a movement trajectory type is desired is specified on the timeline in the timeline operation unit 54a. This diagram illustrates... Figure 30 The example shown illustrates how to specify the range from the first to the third latitude point when setting three latitude points for the viewpoint.

[0396] The operation settings are configured in the operation buttons of the trajectory shape adjustment operation unit 57 to specify the shape type of the movement trajectory. For example, such as Figure 32 As shown, the operation is set in the curve interpolation button B18 in the trajectory shape adjustment operation unit 57.

[0397] In response to the operation of the curve interpolation button B18, the shooting style generation processing unit 34 targets the... Figure 31 The specified range of the viewpoint's movement trajectory is used to perform curve interpolation of the movement trajectory. Then, as... Figure 33 As shown, the display processing unit 34a performs the processing of displaying the motion trajectory information Mm generated by curve interpolation in the shooting style window 53.

[0398] By shaping the viewpoint's movement trajectory into a curved shape in this way, significant changes in the distance from the subject to the viewpoint can be prevented even as the viewpoint moves. In other words, it prevents large changes in the size of the subject in a free-viewpoint image.

[0399] In this example, when performing curve interpolation as described above, a process is executed to change the shape of the transit point marker Mv displayed on the timeline in the timeline operation unit 54a to a shape corresponding to the curve interpolation. Specifically, in this example, as shown in the figure, the shape of the transit point marker Mv changes from a square marker to a circular marker. As a result, the user can also be notified on the timeline that curve interpolation is being performed on the movement trajectory of the viewpoint connecting the transit points.

[0400] Note that in this example, an operation button for indicating that the shape of the movement trajectory is linearized is configured in the trajectory shape adjustment operation unit 57, and when the button is operated, the corresponding shape via the dot marker Mv becomes a square marker.

[0401] Here, the shape of the movement trajectory can be set to a shape other than a curve or a straight line. For example, such as Figure 33 The movement trajectory information Mm shows that a shape combining curves and straight lines can be used. Furthermore, the movement trajectory via curves is not limited to a constant curvature, and the curvature can be set to different values ​​within certain intervals.

[0402] Furthermore, in the case of changes in the shape of the movement trajectory as described above, the point marker Mv can be displayed not only in two types of display forms, namely straight lines and curves, as shown above, but also in different display forms corresponding to each change.

[0403] In addition, in the creation operation screen Gg of this example, the viewpoint movement speed can be specified.

[0404] In specifying movement speed, firstly, as Figure 31 As shown, a target range is specified on the timeline in the timeline operation unit 54a. Here, as... Figure 34 As shown, assume that the range only crosses the second transit point. Then, the operation button set in the speed adjustment operation unit 56 is operated. For example, as... Figure 34 As shown, the speed adjustment button B19 is located in the speed adjustment operation unit 56.

[0405] In response to the operation of the speed adjustment button B19, the shooting style generation processing unit 34 adjusts the speed of the viewpoint corresponding to the operation button for a portion of the specified viewpoint movement trajectory.

[0406] In this example, such as Figure 35 As shown, in response to this speed adjustment, the display processing unit 34a performs a process that changes the shape of the corresponding point marker Mv on the timeline to the shape of the pattern according to the executed speed adjustment. Note that the display pattern shown is merely an example. By performing this shape change, the user can be notified on the timeline that a speed adjustment has been performed within the corresponding range of the viewpoint movement trajectory.

[0407] Next, the adjustment of the target position will be described.

[0408] First, refer to Figure 36 and Figure 37 Describe the significance of the target Tg.

[0409] As described above, the target Tg is used to determine the line-of-sight direction from the viewpoint in the free viewpoint image. Figure 36 The field of view Rf (Rf1, Rf3, Rf6, and Rf9 for each camera position) and the line of sight Dg (Dg1, Dg3, Dg6, and Dg9 for each camera position) are shown in the movement trajectory from the viewpoint.

[0410] As described above, in the generation of the free viewpoint image in this example, a time period facing the target Tg can be specified during the viewpoint movement period. In other words, the time period during which the target Tg continues to be positioned at a predetermined position in the image frame can be specified as the time period following the target Tg. In this example, following the target Tg is performed such that the target Tg is, for example, as follows: Figure 37 As shown, it continues to be located in the center of the image frame.

[0411] Figure 36 The diagram illustrates the line-of-sight direction Dg and field of view Rf when following a target Tg during a viewpoint movement from camera1 to camera9. However, in this example, as shown, the line-of-sight direction Dg and field of view Rf are set to capture the target Tg at the center of the image frame.

[0412] In this example, the position of the target Tg can be adjusted on the creation operation screen Gg. For example, as an adjustment operation for the target Tg, one could envision performing an operation to adjust the position of the target marker Mt displayed in the shooting style window 53.

[0413] When the position of the target Tg is changed by adjusting, the shooting style generation processing unit 34 sets the viewing direction Dg and field of view Rf at each viewpoint position so as to maintain the changed target Tg at a predetermined position in the image frame.

[0414] Here, in the creation operation screen Gg of this example, it is possible to specify the position of the target Tg to move over time during the viewpoint movement period.

[0415] refer to Figures 38-45 Describes the specific operation process for moving the target Tg.

[0416] First, such as Figure 38 As shown, the operation of specifying a new point Ptn for the target Tg is performed. Specifically, in the shooting style window 53, the operation of setting a target position designation mark Mtn for specifying the new point Ptn at the desired location is performed. Although not shown, the target position designation mark Mtn is initially superimposed and displayed on the target mark Mt, and the user drags and drops the target position designation mark Mtn from the position of the target mark Mt to the desired location.

[0417] This operation specifies a new point Ptn for the target Tg.

[0418] Next, the user operates the target button B2 set on the preset list display unit 51 to put the preset list display unit 51 into the display state of the list of the target Tg. In the display state, as... Figure 39 As shown, the add button B20 for the target Tg is displayed on the preset list display unit 51, and the user can give the instruction to add the new target Tg to the position specified by the target position specification marker Mtn by operating the add button B20.

[0419] In response to the operation of the add button B20, as shown in the figure, the identification information (“Target0” in the figure) and location information (indicating the location information of the new point Ptn) of the added target Tg are displayed on the preset list display unit 51. Furthermore, in the shooting style window 53, an additional target marker Mtt, representing the added target Tg, is displayed at the location of the target location specification marker Mtn.

[0420] Next, as from Figures 40 to 42 As shown in the migration diagram, the user performs an operation to add a new target to the timeline. Specifically, this involves dragging and dropping the newly displayed target (here, "Target0") from the preset list display unit 51 to the desired position on the timeline of the timeline operation unit 54a.

[0421] like Figure 42The arrival target timing marker Mem shown is displayed at the position on the timeline specified by the drop operation. The arrival target timing marker Mem is a marker indicating the arrival target timing from the viewpoint's target Tg, when the position moves from the position indicated by the target marker Mt (i.e., the initial position of target Tg) to the position indicated by the additional target marker Mtt (i.e., the new point Ptn), to the new point Ptn of target Tg. In other words, the operation of adding a new target to the timeline as described above is the operation of specifying the target timing of the movement of the target Tg relative to target Tg to reach the new point Ptn.

[0422] After performing the operation described above to add a new target to the timeline, the user performs... Figures 43-45 The operation shown specifies the time period for the target Tg.

[0423] When specifying the time period for the target Tg, firstly, as Figure 43 As shown, the operation is set in the LookAt button B21 of the timeline operation unit 54a. Then, as shown, the time period specification column B22 for specifying a time period is displayed on the timeline. When the LookAt button B21 is operated, the time period specification column B22 is displayed in a mode that specifies the time period from the start time of the viewpoint movement to the time point indicated by the arrival target timing mark Mem, as shown. If the user wants to change the time period of the surface target Tg, the user performs the operation of extending or decreasing the time period specification column B22. Here, as... Figure 44 and Figure 45 As shown, assume that the time period specification column B22 is extended and the time period facing the target Tg is specified as the time period until the end of the viewpoint's movement (the time period from the start to the end of the viewpoint's movement).

[0424] Figures 46-49 It is used to describe the execution reference Figures 38-45 The diagram illustrates preview and reconstructed images of various specified operations on the movement of the target Tg, including shooting style preview and reconstructed images from the viewpoint. Specifically, it shows the preview and reconstructed images of the target Tg under various specified operations. Figures 38-45 The various specified operations for moving the target Tg described herein are followed by the operation of the reproduce button B11, resulting in the migration of the image displayed in the shooting style window 53 and the preview window 55.

[0425] In this case, the target Tg is moved to reach the new point Ptn at the time indicated by the arrival target timing marker Mem on the timeline during the period from the start to the end of the viewpoint movement. Therefore, from the start time of the viewpoint movement to the time indicated by the arrival target timing marker Mem, as... Figure 46 and Figure 47As shown, the target marker Mt gradually approaches the target location designated marker Mtn (the additional target marker Mtt in the shooting style window 53). Here, in Figure 46 and Figure 47 In the shooting style window 53 and the preview window 55, the target initial position marker Mst is shown, but the target initial position marker Mst is a marker indicating the position of the target Tg at the start of the viewpoint movement.

[0426] In this example, the target-oriented time period Tg is specified as the entire time period from the start to the end of the viewpoint movement. That is, the time period exceeding the timing indicated by the target arrival time marker Mem is specified as the target-oriented time period Tg.

[0427] In this way, when the time period for the target Tg is specified as a time period that exceeds the time period until the target time marker Mem is reached, in this example, during the time period that exceeds the time period until the target time marker Mem is reached, as the target Tg moves, the position of the target Tg gradually returns from the new point Ptn to the starting position of the movement.

[0428] Therefore, in the time period facing the target Tg, exceeding the time period until the target timing marker Mem is reached, such as Figure 48 and Figure 49 As shown, over time, the target marker Mt gradually approaches the target initial position marker Mst, which indicates the starting position of the movement.

[0429] By initiating the movement of the target Tg as described above, the degree of freedom in creating free viewpoint images can be increased compared to the case where the position of the target Tg is fixed.

[0430] In the above description, when specifying the location of the moving target Tg, it has been shown that a period exceeding the time until reaching the target timing mark Mem is specified as the period oriented towards the target Tg. However, as... Figure 50 As shown, the time period from the start of viewpoint movement to the arrival of the target timing mark Mem can also be specified as a target-oriented time period. In this case, during the viewpoint movement period after the arrival of the target timing mark Mem, a free viewpoint image that does not follow the specified target Tg is generated.

[0431] Furthermore, the above description illustrates the addition of a target Tg as an example of adding a new target Tg that specifies the marker Mtn using the target location, but multiple target Tgs can also be added.

[0432] Then, in this case, such as Figure 51 As shown, for example, the time period of each added target Tg can be specified individually. Figure 51This illustrates a state in which a marker Mtn is specified by using the target location (see [link]). Figure 38 and 39 When adding two targets Tg (hereinafter referred to as target Tg-1 and target Tg-2) as a new target Tg, the arrival time markers Mem-1 and Mem-2 are displayed separately on the timeline by adding targets Tg-1 and Tg-2 to different positions on the timeline. In this case, for target Tg-1, as shown in the time period designation column B22-1 in the figure, it is assumed that the time period from the start time of the viewpoint's movement to the arrival time marker Mem-1 is designated as the time period facing target Tg. Similarly, for target Tg-2, as shown in the time period designation column B22-2 in the figure, it is assumed that the time period from the time indicated by the arrival time marker Mem-1 after a predetermined time to the time indicated by the arrival time marker Mem-2 is designated as the time period facing target Tg.

[0433] In this case, during the time period indicated by the time period specification column B22-1 within the viewpoint movement period, a free viewpoint image is generated in which the position of target Tg gradually moves from the initial position (the position of target Tg at the start time of viewpoint movement) to the position of target Tg-1. And, during the time period indicated by the time period specification column B22-2, for example, a free viewpoint image is generated in which the position of target Tg gradually moves from the initial position to the position of target Tg-2.

[0434] Here, as from Figure 51 As described, the creation operation screen Gg can accept operations that specify the positions of multiple target Tg.

[0435] As a result, it is possible to generate free viewpoint images that follow target A during one period of viewpoint movement and target B during another period, and to freely improve the settings of the target to be followed.

[0436] Therefore, it can increase the degree of freedom in creating free viewpoint images.

[0437] Note that in the above description, the designation of the target Tg's position refers to an example where the target Tg's position moves during the viewpoint movement period, and the target Tg's position is designated as the destination point for movement. However, the designation of the target Tg's position can also be performed as a designation of the target Tg's position if it does not move during the viewpoint movement period.

[0438] refer to Figure 52 and Figure 53 The flowchart describes the processing related to generating and displaying the movement trajectory based on the operation input on the creation operation screen Gg.

[0439] Notice, Figure 52 and Figure 53 The processing shown is performed by the CPU 71 of the free viewpoint image server 2. This processing is used to implement some of the functions of the shooting style generation processing unit 34 described above.

[0440] Figure 52 The process associated with the generation and display of movement trajectories based on specified viewpoints from the internal and external cameras is shown.

[0441] First, in step S101, CPU 71 waits for the specified operation from the inner camera. In this example, as such... Figure 22 As shown, the operation of specifying the camera number described in the drop-down list of the inner camera in the shooting style entry displayed on the shooting style list display unit 52 is performed.

[0442] When performing a specified operation on the internal camera, CPU 71, as referred to Figure 23 As described, various types of display processing related to the internal camera are performed as internal camera display processing in step S102. For example, processing is performed to highlight the camera position mark Mc for the camera designated as the internal camera in the shooting style window 53 and display the field of view information Fv.

[0443] In step S103 following step S102, CPU 71 waits for the specified operation from the external camera (see...). Figure 24 (as described), and if the specified operation of the external camera is performed, the process is advanced to step S104.

[0444] In step S104, CPU 71 performs the process of generating the viewpoint movement trajectory connecting the inner and outer cameras.

[0445] Then, in the subsequent step S105, CPU 71 performs processing of the display external camera and viewpoint movement trajectory. That is, it performs reference... Figure 25 The description includes various display processing related to the external camera and the display processing of viewpoint movement trajectory information Mm.

[0446] CPU 71 terminates the execution of the display processing in step S105 in response to the execution of the display processing. Figure 52 The series of processes shown.

[0447] Figure 53 This illustrates the processing related to the generation and display of a movement trajectory based on a specified viewpoint via a point.

[0448] In step S110, CPU 71 waits for a specified operation via a point. In this example, the specified operation includes referencing... Figures 26-28 A series of operations on the timeline.

[0449] When a specified transit point is provided, in step S111, CPU 71 generates a viewpoint movement trajectory that passes through the specified point. That is, it generates a viewpoint movement trajectory that connects the inner camera, the specified point, and the outer camera.

[0450] Then, in the subsequent step S112, CPU 71 performs the processing of displaying the transit point and viewpoint movement trajectory. That is, for a specified transit point, it performs the following... Figure 29 The processing described herein is used to highlight the camera position marker in the shooting style window 53, display the field of view information Fv, and display the transit point marker Mv in the timeline.

[0451] CPU 71 terminates the execution of the display processing in step S112 in response to the execution of the display processing. Figure 53 The series of processes shown.

[0452] <8. GUI for creating free-viewpoint images>

[0453] Next, refer to Figures 54-72 ,describe Figure 7 The image shows details of the shooting style specification Gs, an example of the shooting style specification process for generating free viewpoint images, and various functions related to shooting style specification.

[0454] Figure 54 This is a diagram showing the initial frame of the shooting style specification frame Gs. Here, the processing related to the display of various types of information on the frame of the shooting style specification frame Gs, which will be described below, is handled by the aforementioned display processing unit 32a (see...). Figure 5 )implement.

[0455] As described above, the shooting style specification screen Gs includes a scene window 41, a scene list display unit 42, a shooting style window 43, a shooting style list display unit 44, a parameter display unit 45, and a transmission window 46.

[0456] In addition, on the shooting style specified screen Gs, the scene window 41 is equipped with a camera specified operation unit 47, a still image import button B31 and a moving image import button B32, and the play-back button B33, the pause button B34 and the stop button B35 are set at the bottom of the screen.

[0457] Furthermore, on the shooting style specification screen Gs, the shooting style window 43 is equipped with an X-axis viewpoint button B36, a Y-axis viewpoint button B37, a Z-axis viewpoint button B38, a Ca viewpoint button B39, a Pe viewpoint button B40, a display path restriction button B41, and a restriction release button B42. Additionally, the shooting style list display unit 44 is equipped with a filter operation unit 48. The filter operation unit 48 includes a drop-down button B43 and a reset button B44.

[0458] When generating a free-viewpoint image on the shooting style specified frame Gs, the user first performs an operation to import images (i.e., images of the scene that serve as the generation target for the free-viewpoint images) that are the image data V1 to V16 mentioned above. During this import, the user can arbitrarily operate either the still image import button B31 or the moving image import button B32 shown in the figure. The still image import button B31 is used to issue an instruction to import image data V1 to V16 using still images (FV clips) used to generate the aforementioned free-viewpoint images, and the moving image import button B32 is used to issue an instruction to import image data V1 to V16 using moving images (FV clips) used to generate the aforementioned free-viewpoint images.

[0459] In response to the operation of either the still image import button B31 or the moving image import button B32, the shooting style specification screen Gs is displayed as follows. Figure 55 The pop-up window W1 is shown, and the user can operate the "GET IN / OUT TC" button set in the pop-up window W1, such as... Figure 56 As shown, the pop-up window W1 displays information indicating the image data V1 to V16 to be imported. When importing the displayed data, the user clicks the OK button set in the pop-up window W1.

[0460] When the OK button is pressed, such as Figure 57 As shown, information about the imported scene is added to the scene list display unit 42 on the shooting style specified screen Gs, and the image of the imported scene is displayed in the scene window 41. As information about the scene added to the list via import, a thumbnail image of the scene is displayed, time information indicated by timecode (in this example, both the start and end times of the scene are displayed), and information indicating the scene's time period, etc. Note that the example shown is the case of importing image data V1 to V16 as still images, and the value indicating the scene's time period is "0".

[0461] The camera designation operation unit 47 is provided with a camera selection button for selecting which camera to display the images (i.e., image data V1 to V16) of each camera used to import the scene.

[0462] On the shooting style specification screen Gs, you can preview the shooting style in the shooting style window 43.

[0463] In the shooting style window 43, the viewing viewpoint of the shooting style in three-dimensional space can be switched using the X-axis viewpoint button B36, Y-axis viewpoint button B37, Z-axis viewpoint button B38, Ca viewpoint button B39, and Pe viewpoint button B40.

[0464] The X-axis viewpoint button B36, Y-axis viewpoint button B37, and Z-axis viewpoint button B38 are used to switch the viewpoint in 3D space to the X-axis, Y-axis, and Z-axis viewpoints, respectively. Here, the X-axis, Y-axis, and Z-axis are three axes that define 3D space. In this example, the X-axis defines the horizontal direction, the Y-axis defines the vertical direction, and the Z-axis defines a direction orthogonal to both the X-axis and Y-axis.

[0465] The Pe viewpoint button B40 is used to switch the viewing viewpoint in three-dimensional space to any viewpoint specified by the user.

[0466] The Ca viewpoint button B39 is used to switch the observation viewpoint in three-dimensional space to the viewpoint in the shooting style (a point on the viewpoint movement trajectory).

[0467] Figure 58 , Figure 59 , Figure 60 , Figure 61 and Figure 62 The images displayed in the shooting style window 43 are shown when the X-axis viewpoint button B36, Y-axis viewpoint button B37, Z-axis viewpoint button B38, Pe viewpoint button B40, and Ca viewpoint button B39 are operated respectively.

[0468] Here, the shooting style window 43 displays information indicating the shooting styles shown on the shooting style list display unit 44. The shooting styles displayed on the shooting style list display unit 44 are the shooting styles created through the aforementioned creation operation screen Gg, and are candidates for shooting styles used to generate free-viewpoint images. In other words, it is a shooting style that can be specified as a shooting style for generating free-viewpoint images.

[0469] In the shooting style window 43, the aforementioned movement trajectory information Mm, camera position marker Mc, and field of view information Fv are displayed as information indicating the shooting style (again, in this case, the field of view is information indicated graphically). Furthermore, as... Figure 61 As specifically shown, the camera position marker Mc indicating the positions of the inner and outer cameras is displayed larger than other camera position marker Mcs, and indicates the positions of the inner and outer cameras. Here, the camera position marker Mcs of the inner and outer cameras can be indicated in a different way than the camera position marker Mcs of other cameras besides the inner and outer cameras, that is, information indicating the positions of the camera used as the starting point of the viewpoint movement and the camera used as the ending point of the viewpoint movement (starting point arrangement position information, ending point arrangement position information), and the method is not limited to different display sizes.

[0470] Furthermore, in the shooting style window 43, the target marker Mt, which indicates the location of the aforementioned target Tg, is displayed as information indicating the shooting style.

[0471] Note that in Figure 62 In the case of (i.e., when the Ca viewpoint button B39 is operated), since the observed image comes from the viewpoint movement trajectory, the movement trajectory information Mm, camera position marker Mc, and field of view information Fv are not displayed, but the target marker Mt is displayed instead.

[0472] In the figure, the two target markers Mt are shown as target markers Mt (specifically, refer to...). Figure 60 and Figure 61 However, this means that shooting styles with different target Tg are mixed into multiple shooting styles as candidates displayed on the shooting style list display unit 44. That is, in this case, the shooting styles as candidates include, for example, those whose position is set by... Figure 61 The target marker Mt on the left side of the image indicates the target Tg. The shooting style and setting of its position are determined by... Figure 61 The target marker Mt on the right side indicates the shooting style of the target Tg.

[0473] In the shooting style window 43, a dynamic preview playback of the shooting style can be performed. This dynamic preview playback can be instructed by operating the playback button B33.

[0474] When the reproduce button B33 is pressed, Figures 58-61 The images shown depict the position and shape of the field of view information Fv changing constantly as the viewpoint moves along its trajectory, for viewpoints on the X, Y, and Z axes, and for any other viewpoint. Furthermore, in... Figure 62 In this case, images of the target in three-dimensional space, obtained by observing each point from the starting point to the ending point of the viewpoint's movement trajectory, are sequentially switched and displayed. That is, the observation images of three-dimensional space that change constantly as the viewpoint moves are displayed.

[0475] Note that the pause button B12 and the stop button B13 are used to indicate the pause and stop of the dynamic preview playback, respectively, as described above.

[0476] Here, in such Figure 62 In cases where a preview of the observed image from a viewpoint is reproduced as in the manner described above, a free-viewpoint image generated using an FV clip through the selected shooting style can be displayed as the display image. That is, the aforementioned realistic 3D model is generated based on the imported image data V1 to V16, and a 2D image rendered by pasting textures onto the realistic 3D model is displayed as the preview image.

[0477] However, since the generation of free-viewpoint images as FV clips requires a corresponding processing load and processing time, the time users have to wait until the preview playback begins is longer, and it may hinder the rapid generation of free-viewpoint images.

[0478] Therefore, in this example, in such Figure 62 In the case of previewing and reproducing the observed image from the viewpoint, instead of generating a real 3D model based on image data V1 to V16 (i.e., captured images in real space), the image obtained by rendering the aforementioned virtual 3D model (virtual 3D model in real space) is displayed as the display image.

[0479] As a result, the processing time required to display a preview of an image viewed from a viewpoint can be reduced, and the creation of free-viewpoint images can be performed quickly.

[0480] Here, as mentioned above, the shooting style window 43 can display the selected shooting style or multiple optional shooting styles.

[0481] Despite Figures 58-62 The image shows the case where only the selected shooting style is displayed, but the shooting style window 43 can also display the shooting style information of all the multiple shooting styles displayed on the shooting style list display unit 44.

[0482] In this way, the number of shooting styles to be displayed in the shooting style window 43 can be toggled by displaying the path restriction button B41 and the restriction release button B42. The path restriction button B41 is used to indicate that only the selected shooting style is displayed, and the restriction release button B42 is used to release the state of being limited to displaying only the selected shooting style, and also serves as an indicator button for displaying shooting style information for all the multiple shooting styles displayed on the shooting style list display unit 44.

[0483] Next, the shooting style list display unit 44 is described.

[0484] Candidate shooting styles that can be used to generate free-viewpoint images are displayed on the shooting style list display unit 44 (see, for example...). Figure 63 Examples of shooting style information displayed on the shooting style list display unit 44 include shooting style ID, identification information and label information of internal and external cameras, etc.

[0485] Furthermore, in this example, thumbnail images of movement trajectory information Mm are displayed for each shooting style in the shooting style list display unit 44. By displaying such thumbnail images, users can confirm what viewpoint movement trajectories exist for each shooting style, and even within the shooting style list.

[0486] Here, the label information is information that can be added to each created shooting style when creating a shooting style through the creation operation screen Gg described above, and in this example, it is text information. This information can be entered, for example, into the "Label" field set in the entry of the shooting style on the shooting style list display unit 52 of the creation operation screen Gg (see, for example...). Figure 19 Use this to set the tag information for the shooting style.

[0487] In the following text, this label information will be referred to as "Label Information I1".

[0488] The shooting style list display unit 44 is provided with a filtering operation unit 48 for filtering the shooting styles to be displayed in the list display (i.e., the shooting styles to be displayed on the shooting style list display unit 44).

[0489] refer to Figures 63-65 This describes the functions related to filtering the shooting style using the filtering operation unit 48.

[0490] First, when performing filtering, such as Figure 63 As shown, the user operates the drop-down button B43 on the filter operation unit 48. Then, as shown, the drop-down list 48a is displayed. A list of tag information I1 is displayed in the drop-down list 48a. The tag information I1 displayed in the drop-down list 48a is the tag information I1 set for each shooting style as a candidate. That is, as shown, if there are shooting styles with tag information I1 such as "CW, Cam9" and "CW, Right" set as candidate shooting styles, then tag information I1 such as "CW, Cam9" and "CW, Right" are displayed in the drop-down list 48a.

[0491] The user can give an instruction to display only the shooting styles whose tag information I1 is set on the shooting style list display unit 44 by performing an operation that specifies the tag information I1 to be displayed in the drop-down list 48a (e.g., a click operation).

[0492] Note that from this point, it can be understood that the display portion of each label information I1 in drop-down list 48a corresponds to the filter condition information that indicates the filter conditions used to filter and display shooting style information.

[0493] Figure 64 This shows the state of the shooting style specification screen Gs when "CW, Right" is specified as tag information I1. In this case, only the shooting style in which "CW, Right" is set as tag information I1 is displayed on the shooting style list display unit 44.

[0494] In this case, since only one shooting style, "CW, Right," exists, the shooting style information for "CW, Right" is displayed in the shooting style window 43. Note that setting the "CW, Right" shooting style is defined by the above... Figure 61 The shooting style window 43 shows the shooting style of the target Ta, whose position is indicated by the right target marker Mt, and the two target markers Mt indicate the position of the target Tg. Therefore, in this case, only one target marker Mt is displayed as shooting style information in the shooting style window 43.

[0495] Note that, for example, in the case of multiple shooting styles with specified label information I1, it is conceivable that the shooting style information displayed in the shooting style window 43 is displayed on a list at a predetermined position such as head position.

[0496] By filtering shooting styles based on the tag information I1 as described above, filtering based on any standard can be achieved according to the information content set as tag information I1. For example, if team information (e.g., Team A or Team B, etc.) is set as tag information I1, filtering based on criteria such as whether the shooting style is for a shooting scene of Team A or Team B can be achieved. Alternatively, by setting information indicating the direction of viewpoint movement (e.g., clockwise rotation and counterclockwise rotation, etc.) as tag information I1, filtering based on the direction of viewpoint movement can be achieved.

[0497] Furthermore, by setting the camera closest to the field of interest (such as the field of view closest to the subject to be the target Tg) as the tag information I1, it is possible to filter the shooting style based on the field of interest.

[0498] In the filter operation unit 48, the reset button B44 is a button used to indicate the reset of the filter. When the reset button B44 is operated, as from... Figures 64 to 65 As shown in the image transition, the filtering and display status of the shooting styles in the shooting style list display unit 44 is released, and the shooting styles are displayed in the list as candidate shooting styles for generating free viewpoint images.

[0499] Here, although the above has shown filtering based on label information I1 regarding shooting style, filtering of shooting style can also be performed based on information from the internal or external camera contained in the shooting style information.

[0500] Although an example of displaying information such as label information I1 indicating filter conditions in dropdown list 48a has been described above, however, as Figure 66 As shown, information indicating filter conditions (filter condition information) can also be displayed as buttons.

[0501] At this point, the information to be displayed as a button can be determined based on historical information about shooting styles used in the past to generate free-viewpoint images. For example, the label information I1 for a pre-defined advanced shooting style that was frequently used in the past can be displayed as a button.

[0502] exist Figure 66 The button layout shows the shooting style corresponding to the frequently used preset advanced shooting style, which is accompanied by label information such as "goal mouth", "Left" or "Right".

[0503] The button that displays filter information can be customized by the user.

[0504] Figure 67 An example of button display in this scenario is shown. In this case, the user can set arbitrary information for the display of each button. When the user sets the button display information, the image generation processing unit 32 manages the set information as information indicating the filtering conditions for the shooting style. For example, the information "Team A", "Left", and "Right" shown in the figure is managed as information indicating the filtering conditions for the shooting style. When a button is operated, the image generation processing unit 32 (specifically, the display processing unit 32a) performs the process of displaying the shooting style set on the shooting style list display unit 44, which matches the tag information I1 corresponding to the information managed by the button.

[0505] In addition, the information indicating the filtering conditions can be received by the user as input keyword information.

[0506] In this case, for example, Figure 66 and Figure 67 The keyword input unit 48b shown is set in the filter operation unit 48. In this case, the display processing unit 32a responds to the keyword input to the keyword input unit 48b by performing a process of displaying the shooting style setting tag information I1 that matches the input keyword information on the shooting style list display unit 44.

[0507] Note that in the above description, the shooting style specified for generating the free viewpoint image is executed as the shooting style specified on the shooting style list display unit 44. However, the shooting style can be specified as the shooting style specified on the shooting style window 43.

[0508] Here, in this embodiment, the shooting style information displayed in the shooting style window 43 is information obtained by the movement speed of the visual viewpoint.

[0509] Figure 68 and Figure 69 This is an example of a display showing the visualization of the viewpoint's movement speed. Note that... Figure 68 and Figure 69 An example of an observation image is shown in the shooting style window 43, displaying shooting style information at the viewpoint on the Y-axis as described above.

[0510] In this example, information indicating the period when the viewpoint's movement speed decreases is displayed as information about the visual viewpoint's movement speed. Figure 68 An example is shown where a camera position marker Mc, indicating a camera located in a region where the viewpoint's movement speed decreases, is displayed in a different display mode than other camera position markers Mc. For example, it is conceivable to display the corresponding camera position marker Mc in a different color or size than other camera position markers Mc.

[0511] Figure 69 This illustrates an example where the display mode of the corresponding interval in the motion trajectory information Mm during a period of decreased viewpoint movement speed differs from the display mode of other intervals. For example, as shown in the figure, it is conceivable to display the motion trajectory in the corresponding interval as a dotted line and the motion trajectory in other intervals as a solid line. Alternatively, the color, thickness, and linear shape (e.g., straight lines and wavy lines) of the motion trajectory could be displayed differently between the corresponding interval and other intervals.

[0512] Although not explicitly stated, when the movement trajectory is represented by a dotted line, the viewpoint's movement speed can also be represented by the density of the dots. For example, one could envision a display where the dot density increases with the movement speed.

[0513] Note that, although in Figure 68 and Figure 69 The image shown is an observation from a viewpoint on the Y-axis, but a similar display is performed for viewpoints on the X-axis, Z-axis, and viewpoint Pe (arbitrary viewpoint).

[0514] Furthermore, in this embodiment, the position information of target Tg in the shooting style information displayed in the shooting style window 43 is updated according to the operation of changing the position of target marker Mt.

[0515] The processing is Figure 5 The processing of the shooting style editing unit 32b shown.

[0516] refer to Figure 70 and Figure 71 This describes the target position editing process performed by the shooting style editing unit 32b.

[0517] Figure 70 The shooting style information is displayed in the shooting style window 43.

[0518] Suppose that an operation to change the position of the target marker Mt is performed in the shooting style window 43 as shown in the figure. The operation to change the position of the target marker Mt can be, for example, a drag-and-drop operation of the target marker Mt.

[0519] Here, the position of the target Tg after being changed by this change operation is called "position Pta", while the position of the target Tg before the change is called "Ptb".

[0520] In response to the operation of changing the position of the target marker Mt as described above, the shooting style editing processing unit 32b performs the processing of updating the position information of the target Tg from position Ptb to position Pta based on the shooting style information displayed in the shooting style window 43.

[0521] By updating the position information of the target Tg in the shooting style information in this way, a free viewpoint image is generated in the free viewpoint image generation process using the shooting style information, so that the line of sight Dg from each position on the viewpoint movement trajectory faces the updated position of the target Tg. Figure 71 This image shows the change in field of view Rf when the line of sight Dg from each position on the viewpoint movement trajectory faces the position Pta, which is the updated position.

[0522] Since the shooting style information can be edited on the screen Gs according to the shooting style as described above, it is not necessary to start the software for generating the shooting style information when you want to edit the shooting style information at the stage of specifying the shooting style used to generate the free viewpoint image.

[0523] Therefore, even when editing shooting style information is required, it is possible to quickly perform the work of creating free viewpoint images.

[0524] In addition, in this embodiment, the above-mentioned process of displaying information that the camera needs to be calibrated is performed on the shooting style specified screen Gs.

[0525] Specifically, the display processing unit 32a is based on the above reference. Figure 14 The described utility server 8, based on the results of change detection for each camera (e.g., automatic change detection in step S33), determines whether any camera has detected a change among the cameras whose camera position marker Mc is displayed in the shooting style window 43. If a camera has detected a change, the display processing unit 32a performs processing to display information in the shooting style window 43 notifying the corresponding camera (i.e., the camera that detected the change).

[0526] Note that a camera that detects a change can be restated as a camera that detects a change in the field of view.

[0527] Figure 72 This shows an example of a notification message displayed when a change is detected by the camera.

[0528] Regarding the display of notification information, as shown in the figure, the camera location marker Mc for the corresponding camera is displayed in a different display mode than other camera location markers Mc (even in this case, it is conceivable to change the color, size, shape, etc.). Furthermore, it is also conceivable to display attention-grabbing information near the corresponding camera location marker Mc, such as an exclamation mark as shown in the figure.

[0529] When generating free-viewpoint images, in order to accurately generate 3D information from images captured by multiple cameras, each camera must maintain an assumed position and orientation beforehand, and the parameters used to generate the 3D information need to be calibrated should any change in position or orientation occur in any camera. By notifying the user of cameras that detect changes in the field of view, as described above, the cameras requiring calibration can be notified.

[0530] Therefore, it is possible to generate free-viewpoint images based on accurate 3D information and improve the image quality of free-viewpoint images.

[0531] refer to Figure 73 and Figure 74 The flowchart describes the processing related to the filtering of the shooting style mentioned above. Note that... Figure 73 and Figure 74 The processing shown is performed by the CPU 71 of the free viewpoint image server 2 as the processing of the display processing unit 32a.

[0532] Figure 73 Showing the corresponding as Figure 63 As shown, the processing involves filtering the shooting style based on the label information I1 displayed on the shooting style specified screen Gs.

[0533] First, in step S201, CPU 71 performs the process of acquiring the tag information I1 from each candidate shooting style information. That is, it acquires each candidate shooting style information that can be used to generate a free viewpoint image. Here, the candidate shooting style information is stored in a readable storage device inside or outside the free viewpoint image server 2. In the process of step S201, the candidate shooting style information stored in this manner is acquired.

[0534] In step S202, following step S201, CPU 71 performs processing of the display label information I1. That is, in... Figure 63 When the drop-down list 48a is displayed as shown, in response to the operation of the drop-down button B43, the tag information I1 contained in the shooting style information obtained in step S201 is displayed.

[0535] In step S203 following step S202, CPU 71 waits for the specification operation of tag information I1, and if the specification operation of tag information I1 has already been executed, proceeds to step S204 and performs the process of filtering and displaying the shooting style to which the specified tag information I1 is attached. That is, it performs the process of displaying the shooting style information including the specified tag information I1 (where the specified tag information I1 is set) as candidate shooting style information on the shooting style list display unit 44. Figure 64 As shown, in this example, the shooting style information to be displayed includes, for example, shooting style identification information, information about the internal and external cameras, and label information I1, etc.

[0536] CPU 71 responds to the execution of step S204 and terminates the process. Figure 73 The series of processes shown.

[0537] Figure 74 This illustrates the processing related to filtering based on shooting style according to input keywords.

[0538] exist Figure 74 In step S210, CPU 71 waits for keyword input from the user, and when keyword input is available, it selects a shooting style that includes the input keyword in step S211. That is, among multiple candidate shooting style information, the shooting style information that includes the input keyword in the label information I1 is selected.

[0539] Then, in the subsequent step S212, the CPU 71 performs the process of displaying the selected shooting style. That is, it performs the process of displaying the selected shooting style information on the shooting style list display unit 44.

[0540] In response to the execution of step S212, CPU 71 terminates. Figure 74 The series of processes shown.

[0541] Figure 75 Is with Figure 72 The flowchart shown relates to the processing of a notification requiring camera calibration. Note the connection between... Figure 73 and Figure 74 The processing is similar to that in the past. Figure 75 The processing shown is also performed by the CPU 71 of the free viewpoint image server 2 as part of the processing of the display processing unit 32a.

[0542] In step S301, CPU 71 waits for camera change notification. That is, CPU 71 waits for the utility server 8 to detect the change via the aforementioned automatic change detection (…). Figure 14 In step S33), a change notification is transmitted when a change in the camera is detected. The change notification includes information specifying the camera from which the change was detected.

[0543] When a camera change notification is received, CPU 71 determines in step S302 whether the camera is the one currently displayed. That is, it determines whether the camera notified by the change notification is the one whose camera position marker Mc is displayed in the shooting style window 43. If the camera is not the one currently displayed, CPU 71 terminates. Figure 75 The series of processes shown.

[0544] On the other hand, if the camera is currently displayed, CPU 71 proceeds to step S303 and performs change notification processing. That is, for the corresponding camera position marker Mc displayed in the shooting style window 43, for example, it performs... Figure 72 The information displayed shows the change in the notification display mode.

[0545] In response to the execution of step S303, CPU 71 terminates. Figure 75 The series of processes shown.

[0546] <9. Change Example>

[0547] Note that the embodiments are not limited to the specific examples described above, and various configurations may be adopted as variations.

[0548] For example, in the above description, the device that performs the display processing of the creation operation screen Gg and accepts the operation input for generating the shooting style, and the device that performs the display processing of the shooting style specification screen Gs and accepts the operation input for generating the free viewpoint image, are examples of common devices for the free viewpoint image server 2. However, it is also possible to use a mode where these devices are independent devices.

[0549] Furthermore, in the above description regarding the filtering and display of shooting styles on the shooting style specification screen Gs, an example of filtering based on an operation section such as a button indicating filtering conditions was described. However, for example, the filtering and display of shooting styles can also be performed based on the specification of the target Tg (such as the specification of the target marker Mt displayed in the shooting style window 43). Specifically, only shooting styles whose target Tg is specified as candidate shooting styles are filtered and displayed.

[0550] Additionally, on the creation operation screen Gg and the shooting style specification screen Gs, information about the notification range can be displayed in cases where image quality cannot be guaranteed due to, for example, the actual camera being too far from the subject on the viewpoint's movement trajectory (e.g., a range where the resolution is equal to or less than a predetermined value).

[0551] <10. Overview of the Implementation Examples>

[0552] As described above, the first information processing device according to the embodiment includes a display processing unit (34a) that performs processing to display a screen as a shooting style information creation operation screen (Gg) that includes an operation receiving area (such as a shooting style list display unit 44 and an operation panel unit 54) that receives operation input for specifying shooting style information and a shooting style display area (shooting style window 53) that visualizes and displays the movement trajectory of a viewpoint based on the shooting style information that reflects the specified content through the operation input. The shooting style information creation operation screen (Gg) is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image.

[0553] As a result, users can perform the shooting style creation operation, while visually recognizing the movement trajectory of the viewpoint on the shooting style creation operation screen.

[0554] Therefore, it can improve the efficiency of creating shooting styles.

[0555] Furthermore, in the first information processing device according to the embodiment, the designated operation receiving area can receive designated operations for the start and end points of the movement trajectory (see [link]). Figures 22-25 and Figure 52 ).

[0556] As a result, it is possible to set any starting point and any ending point for the viewpoint's movement trajectory, instead of a fixed point.

[0557] Therefore, it can increase the degree of freedom in creating free viewpoint images.

[0558] Furthermore, in the first information processing device according to the embodiment, the designated operation receiving area can receive a designated operation of the viewpoint via a point (see [link]). Figures 27-30 and Figure 53 ).

[0559] Therefore, it is possible to set the trajectory passing through a specified point, rather than a linear trajectory connecting the start and end points, as the viewpoint's movement trajectory.

[0560] Therefore, it can increase the degree of freedom in creating free viewpoint images.

[0561] Furthermore, in the first information processing device according to this embodiment, the designated operation receiving area can receive a designated operation of the viewpoint via a point at a specified time (see [link]). Figures 27-30 ).

[0562] Therefore, it is possible not only to set the path of the viewpoint, but also to set the timing of the viewpoint passing through the path.

[0563] Therefore, it can increase the degree of freedom in setting the position through which the viewpoint passes and the degree of freedom in setting the timing of the viewpoint passing through the via point, and increase the degree of freedom in creating free viewpoint images.

[0564] Furthermore, in the first information processing device according to the embodiment, the designated operation receiving area can receive a designated operation of the shape type of the movement trajectory (see [link]). Figure 32 and Figure 33 ).

[0565] As a result, the shape of the viewpoint's movement trajectory can be variable, rather than fixed.

[0566] Therefore, it can increase the degree of freedom in creating free viewpoint images.

[0567] For example, if the movement trajectory is curved, it can prevent significant changes in the distance from the subject to the viewpoint even if the viewpoint moves. In other words, it can prevent significant changes in the size of the subject in a free-viewpoint image.

[0568] Furthermore, in the first information processing device according to the embodiment, a designated operation receiving area can receive a designated operation of the viewpoint's movement speed (see [link]). Figure 34 and Figure 35 ).

[0569] As a result, the viewpoint's movement speed can be variable, rather than fixed.

[0570] Therefore, it can increase the degree of freedom in creating free viewpoint images.

[0571] Furthermore, in the first information processing device according to the embodiment, the designated operation receiving area can receive a designated operation for changing the range of movement speed in the movement trajectory (see [link]). Figure 34 and Figure 35 ).

[0572] Therefore, it is possible to dynamically change the speed of the viewpoint in the movement trajectory.

[0573] Therefore, it can increase the degree of freedom in creating free viewpoint images.

[0574] Furthermore, in the first information processing device according to the embodiment, the designated operation receiving area can receive operation input for indicating a timeline of the period from the start time of movement of the viewpoint to the end time of movement (see timeline operation unit 54a).

[0575] By accepting input operations on the timeline, for example, one can simultaneously specify a via point and its timing by dragging and dropping the camera icon on the timeline, or specify an interval to provide a predetermined effect (such as specifying the interval for curve interpolation of the movement trajectory by dragging the camera icon on the timeline). Therefore, it is possible to facilitate the specification of various types of information related to shooting style.

[0576] Therefore, it can improve the efficiency of creating shooting styles.

[0577] Furthermore, in the first information processing device according to the embodiment, the display processing unit performs processing in the shooting style display area to display information (field information Fv) obtained by visualizing the field of view from the viewpoint (see [link]). Figure 20 wait).

[0578] Because the field of view is visually indicated, it makes it easier for users to grasp the shooting style.

[0579] Therefore, it allows users to easily grasp how shooting styles change through operational input, and improves the efficiency of shooting style creation.

[0580] Furthermore, in the first information processing device according to the embodiment, the display processing unit performs processing to display information representing the field of view from the viewpoint in a shooting style display area.

[0581] Because the field of view is displayed graphically, users can easily grasp the shooting style.

[0582] Therefore, it allows users to easily grasp how shooting styles change through operational input, and improves the efficiency of shooting style creation.

[0583] Furthermore, in the first information processing device according to the embodiment, the display processing unit performs processing to display an image obtained by observing three-dimensional space from a viewpoint on a creation operation screen (see preview window 55).

[0584] As a result, images similar to free-viewpoint images generated based on shooting style information can be displayed to users as previews, facilitating the understanding of shooting styles.

[0585] Therefore, it can improve the efficiency of creating shooting styles.

[0586] Furthermore, in the first information processing device according to the embodiment, the designated operation receiving area can receive a designated operation that defines the position of a target from the viewing direction of the viewpoint (see [link]). Figure 38 and Figure 39 wait).

[0587] As a result, an image that follows the target can be generated as a free viewpoint image. An image that follows the target means that the target remains positioned at a predetermined location (e.g., the center) within the image frame.

[0588] Therefore, it can increase the degree of freedom in creating free viewpoint images.

[0589] Furthermore, in the first information processing device according to the embodiment, the designated operation receiving area can receive a designated operation for a target time period (see [link]). Figures 41-52 ).

[0590] A target-oriented time period means the time during which the target remains positioned at a predetermined position within the image frame of a free viewpoint image. Because the target-oriented time period specification operation is enabled as described above, it is possible to generate images that follow the target position for a certain time period during the viewpoint movement and do not follow the target position for other time periods, as free viewpoint images, and to freely improve the setting of the time period following the target position.

[0591] Therefore, it can increase the degree of freedom in creating free viewpoint images.

[0592] Furthermore, in the first information processing device according to the embodiment, the designated operation receiving area can receive multiple designated operations for target locations as designated operations for target locations (see [link]). Figure 51 ).

[0593] As a result, it is possible to generate free viewpoint images that follow target A during one period of viewpoint movement and target B during another period, and to freely improve the settings of the target to be followed.

[0594] Therefore, it can increase the degree of freedom in creating free viewpoint images.

[0595] Furthermore, according to the first information processing method of the embodiment, the information processing device executes an information processing method for displaying a screen as a shooting style information creation operation screen, which includes a designated operation receiving area for receiving operation input for specifying shooting style information and a shooting style display area based on the shooting style information that reflects the specified content through the operation input and displays the movement trajectory of the viewpoint. The shooting style information creation operation screen is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image.

[0596] According to this first information processing method, the same operation and effect as the first information processing device described above can be obtained.

[0597] Furthermore, the second information processing device according to the embodiment includes a display processing unit (32a) that performs the processing of displaying a screen that displays a screen corresponding to the user input information among multiple shooting style information as a shooting style specified screen (Gs) for receiving the shooting style information and specifying the operation of the shooting style information, wherein the shooting style information is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image.

[0598] By filtering and displaying shooting style information based on user input, it is easy to find the shooting style information the user wants and shorten the time required to specify the shooting style information.

[0599] Therefore, the creation of free-viewpoint images can be performed quickly.

[0600] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing of filtering and displaying shooting style information corresponding to keywords as input information on the shooting style specifying screen (see [link]). Figure 66 , 67 , 74, etc.).

[0601] Therefore, it is possible to perform appropriate filtering of shooting style information that reflects the user's intent.

[0602] Therefore, it makes it easier for users to find the shooting style information they need and further reduces the time required to specify shooting style information.

[0603] Furthermore, in the second information processing device according to the embodiment, an operation section indicating filtering conditions for shooting style information is configured on a shooting style specifying screen, and a display processing unit performs processing of filtering and displaying shooting style information according to the filtering conditions indicated by the operation section based on the operation of the operation section (see [link]). Figure 63 , 64 66, 67 and 73).

[0604] As a result, the operations required to filter and display shooting style information can be reduced to simply selecting filter condition information.

[0605] Therefore, it can reduce the user's workload in filtering and displaying shooting style information.

[0606] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing to display information obtained from the movement trajectory of the visual viewpoint on a screen specified by the shooting style (see [link]). Figure 61 wait).

[0607] By displaying information about the movement trajectory of a visual viewpoint, users can easily image a shooting style.

[0608] Therefore, when specifying shooting style information for creating free-viewpoint images, it makes it easier for users to find the required shooting style information and reduces the time required to specify the shooting style information.

[0609] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing to display camera arrangement position information on a shooting style specified screen, indicating the arrangement positions of multiple cameras performing imaging for generating a free viewpoint image (see [link]). Figure 61 wait).

[0610] By displaying information indicating the placement of each camera, users can easily envision which type of image should be generated as a free-viewpoint image.

[0611] Therefore, the creation of free-viewpoint images can be performed quickly.

[0612] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing to display, on a shooting style specified screen, starting point arrangement position information and ending point arrangement position information indicating the respective positions of a camera among a plurality of cameras that serves as the starting point of the viewpoint movement and a camera that serves as the ending point of the viewpoint movement (see [link]). Figure 61 wait).

[0613] Therefore, it allows users to control the starting and ending camera positions of viewpoint movement in their shooting style.

[0614] Therefore, when specifying shooting style information for creating free-viewpoint images, it is easier to find the shooting style information desired by the user. In particular, in the case of generating an image where the preceding and following clips are connected to a free-viewpoint image as described above, it is desirable to match the camera that serves as the starting point of the viewpoint movement with the imaging camera of the preceding clip, and the camera that serves as the ending point of the viewpoint movement with the imaging camera of the following clip, making the connection between clips natural. However, by displaying the various camera positions as the starting point of movement and the camera as the ending point of movement as described above, it is easy to specify appropriate shooting styles corresponding to the imaging cameras of the preceding and following clips.

[0615] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing to display, in different modes, the starting point arrangement location information and the ending point arrangement location information, as well as the arrangement location information of the cameras other than the camera used as the starting point of movement and the camera used as the ending point of movement among a plurality of cameras.

[0616] Therefore, it allows users to intuitively grasp from which camera position the movement of the viewpoint begins and ends in the shooting style.

[0617] Therefore, when specifying shooting style information for creating free-viewpoint images, it is easier to find the shooting style information that the user wants.

[0618] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing to display information obtained by visualizing the movement speed of the viewpoint on a shooting style-specified screen (see [link]). Figure 68 and Figure 69 ).

[0619] The time period during which the speed of viewpoint movement changes is an important factor in the rendering of free viewpoint images.

[0620] Therefore, by displaying visual information about the speed of viewpoint movement as described above, it is easier to find the shooting style information that the user wants and the time required to specify the shooting style information can be reduced.

[0621] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing of information indicating a period of decreased movement speed as information obtained by visualizing the movement speed of the viewpoint.

[0622] The period during which the viewpoint's movement speed decreases is an important factor in the rendering of free-viewpoint images.

[0623] Therefore, by displaying information indicating the period when the viewpoint's movement speed decreases, as described above, it is easier to find the shooting style information the user wants and to shorten the time required to specify the shooting style information.

[0624] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing to display information obtained by visualizing the field of view from the viewpoint on a shooting style specified screen (see [link]). Figure 61 wait).

[0625] Because the field of view is visually indicated, it makes it easier for users to grasp the shooting style.

[0626] Therefore, it makes it easier for users to find the shooting style information they need and reduces the time required to specify shooting style information.

[0627] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing to display a target defining the direction of gaze from the viewpoint on a shooting style specified screen (see [link]). Figure 61 wait).

[0628] As a result, it allows users to easily determine the location of the subject in three-dimensional space as the target of the shooting style.

[0629] Therefore, it makes it easier for users to find the shooting style information they need and reduces the time required to specify shooting style information.

[0630] Furthermore, the second information processing device according to the embodiment includes a shooting style editing processing unit (32b) that updates the target position information in the shooting style information according to the change of the target position on the frame specified by the shooting style (see also...). Figure 70 and 71 ).

[0631] As a result, when you want to edit the shooting style information at a stage where you specify the shooting style information used to generate free-viewpoint images, it is not necessary to start the software used to generate the shooting style information.

[0632] Therefore, even when editing shooting style information is required, it is possible to quickly perform the work of creating free viewpoint images.

[0633] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing to display an image obtained by observing three-dimensional space from a viewpoint on a screen specified by the shooting style (see [link]). Figure 62 ).

[0634] As a result, images similar to free-viewpoint images generated based on shooting style information can be displayed to users as previews, facilitating the understanding of shooting styles.

[0635] Therefore, it makes it easier for users to find the shooting style information they need and reduces the time required to specify shooting style information.

[0636] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing to display an image obtained by rendering a virtual three-dimensional model of real space as an image obtained by viewing three-dimensional space from a viewpoint (see [link]). Figure 62 ).

[0637] As a result, when a preview display of the observed image from the viewpoint is achieved, it is not necessary to perform rendering processing using a 3D model generated from a captured image from the target's real space.

[0638] Therefore, the processing time required to display a preview of an image viewed from a viewpoint can be reduced, and the creation of free-viewpoint images can be performed quickly.

[0639] Furthermore, in the second information processing device according to the embodiment, the display processing unit performs processing to display information of the camera among a plurality of cameras that has detected a change in field of view (see [link]). Figure 72 ).

[0640] When generating free-viewpoint images, in order to accurately generate 3D information from images captured by multiple cameras, each camera must maintain an assumed position and orientation beforehand. Furthermore, if any camera experiences a change in position or orientation, it is necessary to calibrate the parameters used to generate the 3D information. By notifying the user of cameras that detect changes in the field of view, as described above, the cameras requiring calibration can be notified.

[0641] Therefore, it is possible to generate free-viewpoint images based on accurate 3D information and improve the image quality of free-viewpoint images.

[0642] Furthermore, the second information processing method according to the embodiment is an information processing method in which an information processing device performs a process of displaying a screen that shows a shooting style information corresponding to user input information among multiple shooting style information through filtering, as a shooting style specified screen for receiving a specified operation of the shooting style information, wherein the shooting style information is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image.

[0643] According to this second information processing method, the same operation and effect as the second information processing device described above can be obtained.

[0644] Here, as an example, it is possible to consider using a device such as a CPU or digital signal processor (DSP) or including a CPU or DSP to perform operations via... Figure 52 or Figure 53 The processing procedure of the display processing unit 34a described in the above.

[0645] That is, the first program in this embodiment is a program that can be read by a computer device, and is a program that enables the computer device to perform processing to display a screen as a shooting style information creation operation screen, which includes a designated operation receiving area for receiving operation input for specifying shooting style information and a shooting style display area based on the shooting style information reflecting the specified content through the operation input and displaying the movement trajectory of the viewpoint. The shooting style information creation operation screen is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image.

[0646] This procedure allows the aforementioned display processing unit 34a to be implemented in the device 70, which is an information processing device.

[0647] Furthermore, as an example, it is possible to consider using a CPU, DSP, or device including a CPU or DSP to perform operations via... Figure 73 or Figure 74 The processing procedure of the display processing unit 32a described in the above.

[0648] That is, the second program in this embodiment is a program that can be read by a computer device, and is a program that causes the computer device to perform the following processing, which includes: executing a shooting style specified screen that displays a screen that corresponds to the user input information among multiple shooting style information by filtering, as a specified operation for receiving the shooting style information, wherein the shooting style information is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image.

[0649] This procedure allows the aforementioned display processing unit 32a to be implemented in the device 70, which is an information processing device.

[0650] These programs can be pre-recorded in HDDs, which are recording media built into devices such as computer equipment, or in ROMs of microcomputers with CPUs.

[0651] Alternatively, the program can be temporarily or permanently stored (recorded) on removable recording media such as floppy disks, CD-ROMs, magneto-optical (MO) discs, digital versatile optical discs (DVDs), Blu-ray discs (registered trademark), magnetic disks, semiconductor memory, or memory cards. Such removable recording media can be provided as so-called packaged software.

[0652] Furthermore, this program can be installed on a personal computer from a removable recording medium, or downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0653] Furthermore, this program is suitable for providing display processing units 34a and 32a in embodiments across a wide range. For example, by downloading the program to a personal computer, portable information processing device, mobile phone, gaming device, video device, or personal digital assistant (PDA), the personal computer or the like can be used as a device to perform the same processing as the display processing unit 34a or display processing unit 32a of this disclosure.

[0654] Note that the effects described in this specification are merely illustrative and not limited, and other effects may be provided.

[0655] <11. This technology>

[0656] Note that this technology can also have the following configurations.

[0657] (1) An information processing device, comprising:

[0658] The display processing unit performs processing to display a screen that shows the shooting style information corresponding to the user's input information from multiple shooting style information that has been filtered, as a shooting style specified screen for receiving the specified operation of the shooting style information. The shooting style information is information that indicates the movement trajectory of at least one viewpoint in a free viewpoint image.

[0659] (2) The information processing device according to (1), wherein,

[0660] The display processing unit performs the process of filtering and displaying shooting style information corresponding to the keywords as input information on the specified shooting style screen.

[0661] (3) The information processing device according to (1) or (2), wherein,

[0662] On the screen specifying the shooting style, filter condition information indicating the filter conditions for shooting style information is displayed, as well as...

[0663] The display processing unit performs processing to filter and display shooting style information as input information according to the filtering conditions indicated by the selected filtering condition information.

[0664] (4) The information processing device according to any one of (1) to (3), wherein,

[0665] The display processing unit performs the processing of displaying information obtained by visualizing the movement trajectory of the viewpoint on the screen specified by the shooting style.

[0666] (5) The information processing apparatus according to any one of (1) to (4), wherein,

[0667] The display processing unit performs processing to display camera arrangement position information indicating the arrangement positions of multiple cameras on the shooting style specified screen, wherein the multiple cameras perform imaging for generating free viewpoint images.

[0668] (6) The information processing device according to (5), wherein,

[0669] The display processing unit performs processing to display, on the shooting style specified screen, starting point arrangement position information and ending point arrangement position information of the cameras among the plurality of cameras that serve as the starting point of movement of the viewpoint and the camera that serves as the ending point of movement of the viewpoint.

[0670] (7) The information processing device according to (6), wherein,

[0671] The display processing unit performs processing to display the starting point layout location information and the ending point layout location information in different modes, as well as the layout location information of cameras other than the camera used as the starting point of movement and the camera used as the ending point of movement among a plurality of cameras.

[0672] (8) The information processing apparatus according to any one of (4) to (7), wherein,

[0673] The display processing unit performs the processing of displaying information obtained by visualizing the movement speed of the viewpoint on the screen specified by the shooting style.

[0674] (9) The information processing device according to (8), wherein,

[0675] The display processing unit performs processing to display information indicating the period of reduced movement speed as information obtained by visualizing the movement speed of the viewpoint.

[0676] (10) The information processing apparatus according to any one of (4) to (9), wherein,

[0677] The display processing unit performs the process of displaying information obtained by visualizing the field of view from the viewpoint on the screen specified by the shooting style.

[0678] (11) The information processing apparatus according to any one of (4) to (10), wherein,

[0679] The display processing unit performs processing to display a target on the screen specified by the shooting style, defining the direction of the line of sight from the viewpoint.

[0680] (12) The information processing device according to (11) further includes:

[0681] The shooting style editing and processing unit updates the information about the target's position in the shooting style information according to the change in the target's position on the screen specified by the shooting style.

[0682] (13) The information processing apparatus according to any one of (1) to (12), wherein,

[0683] The display processing unit performs the processing of displaying an image obtained by observing three-dimensional space from the viewpoint on the screen specified by the shooting style.

[0684] (14) The information processing device according to (13), wherein,

[0685] The display processing unit performs the processing of displaying an image obtained by rendering a virtual 3D model of real space, rather than a 3D model generated from a captured image of real space, as an image obtained by observing 3D space from the viewpoint.

[0686] (15) The information processing device according to (5), wherein,

[0687] The display processing unit performs processing to display information about the camera among the plurality of cameras that detected a change in the field of view.

[0688] (16) An information processing method, wherein,

[0689] An information processing device performs processing to display a screen showing the shooting style information corresponding to user input information from multiple specified shooting style information that has been filtered and indicated, as a shooting style specified screen for receiving the specified operation of the shooting style information, wherein the shooting style information is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image.

[0690] (17) A computer device readable program,

[0691] The program causes the computer device to perform the following processing: displaying a screen showing the shooting style information corresponding to the user's input information from a plurality of specified shooting style information that has been filtered and indicated, as a shooting style specified screen for receiving the specified operation of the shooting style information, wherein the shooting style information is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image.

[0692] List of reference numerals

[0693] 2. Free Viewpoint Image Server

[0694] 8 Practical Servers

[0695] 10 Imaging equipment

[0696] 21 Interval Identification Processing Unit

[0697] 22 Target Image Transmission Control Unit

[0698] 23 Output Image Generation Unit

[0699] 31 Target Image Acquisition Unit

[0700] 32 Image Generation and Processing Units

[0701] 32a Display Processing Unit

[0702] 32b Shooting Style Editing Unit

[0703] 33. Conveyor Control Unit

[0704] Gs shooting style specified image

[0705] 41 Scene Window

[0706] 42 Scene list display units

[0707] 43 Shooting Style Window

[0708] 44 Shooting Style List Display Unit

[0709] 70 Information processing equipment

[0710] 71 CPU

[0711] 72 ROM

[0712] 73 RAM

[0713] 74 bus

[0714] 75 Input / Output Interfaces

[0715] 76 Input Units

[0716] 77 display units

[0717] 78 audio output units

[0718] 79 storage units

[0719] 80 communication units

[0720] 81 Removable Recording Media

[0721] 82 drives

[0722] Tg Target

[0723] Mc camera location marker

[0724] Fv Field of view information

[0725] Mt target marker

[0726] Mm movement trajectory information

[0727] Mv via dot marker

[0728] Mtn Target Location Specify Marker

[0729] Mtt Attach Target Marker

[0730] Mem arrival target timing mark

[0731] Mst Target Initial Position Marker

[0732] Rf field of view

[0733] Dg line of sight

[0734] 48 Filtering Operation Unit

[0735] 48a Drop-down list

[0736] 48b Keyword Input Unit

[0737] B33 Replay button

[0738] B34 Pause button

[0739] B35 Stop button

[0740] B36 X-axis viewpoint button

[0741] B37 Y-axis viewpoint button

[0742] B38 Z-axis viewpoint button

[0743] B39 Ca Viewpoint Button

[0744] B40 Pe Viewpoint Button

[0745] B43 Drop-down button

[0746] B44 Reset Button

Claims

1. An information processing device, comprising: The display processing unit performs processing to display a screen showing the shooting style information corresponding to the user's input information from multiple shooting style information entries that have been filtered, as a shooting style specified screen for the operation specified by the received shooting style information. The shooting style information is information indicating at least the movement trajectory of a viewpoint in a free viewpoint image. The display processing unit performs the process of displaying information obtained by visualizing the movement speed of the viewpoint on the frame specified by the shooting style. The display processing unit performs the processing of information indicating the period of reduced movement speed as information obtained by visualizing the movement speed of the viewpoint.

2. The information processing device according to claim 1, wherein, The display processing unit performs the process of filtering and displaying shooting style information corresponding to the keywords as input information on the specified shooting style screen.

3. The information processing device according to claim 1, wherein, On the screen specifying the shooting style, filter condition information indicating the filter conditions for shooting style information is displayed, as well as... The display processing unit performs processing to filter and display shooting style information as input information based on the filtering conditions indicated by the selected filtering condition information.

4. The information processing device according to claim 1, wherein, The display processing unit performs the processing of displaying information obtained by visualizing the movement trajectory of the viewpoint on the screen specified by the shooting style.

5. The information processing device according to claim 1, wherein, The display processing unit performs processing to display camera placement information indicating the placement positions of multiple cameras on the screen specified by the shooting style, wherein the multiple cameras perform imaging for generating free viewpoint images.

6. The information processing device according to claim 5, wherein, The display processing unit performs the processing of displaying, on the specified shooting style screen, starting point arrangement position information and ending point arrangement position information of the cameras among the plurality of cameras that serve as the starting point of movement of the viewpoint and the camera that serves as the ending point of movement of the viewpoint.

7. The information processing device according to claim 6, wherein, The display processing unit performs processing to display the starting point layout location information and the ending point layout location information in different modes, as well as the layout location information of cameras other than the camera used as the starting point of the movement and the camera used as the ending point of the movement.

8. The information processing device according to claim 4, wherein, The display processing unit performs the processing of displaying information obtained by visualizing the field of view from the viewpoint on the screen specified by the shooting style.

9. The information processing device according to claim 4, wherein, The display processing unit performs processing to display a target on the screen specified by the shooting style, defining the direction of the line of sight from the viewpoint.

10. The information processing device according to claim 9, further comprising: The shooting style editing and processing unit updates the information about the target's position in the shooting style information according to the change in the target's position on the screen specified by the shooting style.

11. The information processing device according to claim 1, wherein, The display processing unit performs the processing of displaying an image obtained by observing three-dimensional space from the viewpoint on the screen specified by the shooting style.

12. The information processing device according to claim 11, wherein, The display processing unit performs the processing of displaying an image obtained by rendering a virtual 3D model of real space as an image obtained by observing 3D space from the viewpoint.

13. The information processing device according to claim 5, wherein, The display processing unit performs the processing of displaying information to notify the camera among the plurality of cameras that detected a change in the field of view.

14. An information processing method, wherein, The information processing device performs the process of displaying a screen showing the shooting style information corresponding to the user's input information from multiple shooting style information entries, which are filtered and indicated, as a shooting style specified screen for receiving the specified operation of the shooting style information. The shooting style information is information indicating at least the movement trajectory of a viewpoint in a free viewpoint image. The information processing device performs the processing of displaying information obtained by visualizing the movement speed of the viewpoint on the screen specified by the shooting style. The information processing device performs the processing of information indicating the period of reduced movement speed as information obtained by visualizing the movement speed of the viewpoint.

15. A computer program product containing instructions, which, when executed by a processor, cause a computer device to perform the following processing: displaying a screen that indicates shooting style information corresponding to user input information from a plurality of filtered shooting style information as a shooting style specified screen for receiving a specified operation of the shooting style information, wherein the shooting style information is information indicating the movement trajectory of at least one viewpoint in a free viewpoint image. in, Information obtained by visualizing the movement speed of the viewpoint is displayed on the screen specified by the shooting style. The information indicating the period of reduced movement speed is obtained by visualizing the movement speed of the viewpoint.

Images