Server device

Abstract

The purpose of the present invention is to construct a mechanism for reproducing a free viewpoint video. Provided is a server device comprising one or multiple memories and one or multiple processors. The one or multiple memories hold one or more restoration models for generating time-series free viewpoint images, said reconstruction model(s) being trained in advance such that a scene from a first time to a second time can be reconstructed by using time-series photographed images of a plurality of viewpoints, these time-series photographed images having been obtained by continuously photographing a scene from each of the plurality of viewpoints over time. The one or multiple processors receive, from a client, a request including viewpoint information and time information pertaining to the scene, generate, by using the one or more reconstruction models, time-series images corresponding to the viewpoint information and the time information included in the request received from the client, and transmit, to the client, the time-series images in a transmission format capable of video reproduction.

Description

Server device

[0001] The present disclosure relates to a server device.

[0002] A technique called Neural Radiance Fields (NeRF) is known as an image generation technique that uses multiple image capture devices to capture a three-dimensional scene from different viewpoints and then reconstructs the three-dimensional scene using the captured two-dimensional images. This technique allows for the generation of free-viewpoint images of the three-dimensional scene.

[0003] On the other hand, currently, the free viewpoint images generated using this technology are targeted at still images, and in order to apply it to moving images and enable the playback of free viewpoint videos, a mechanism for moving images is required.

[0004] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, [online], [Retrieved March 31, 2023], Internet<URL:https: / / arxiv.org / pdf / 2003.08934.pdf>

[0005] The present disclosure establishes a mechanism for playing free viewpoint video.

[0006] A server device according to one aspect of the present disclosure has, for example, the following configuration: That is, the server device includes one or more memories and one or more processors, wherein the one or more memories hold one or more restoration models for generating time-series free-viewpoint images, the restoration models being pre-trained so as to be able to restore a scene from a first time to a second time using time-series captured images from a plurality of viewpoints obtained by capturing a scene successively over time from the plurality of viewpoints, and the one or more processors receive, from a client, a request including viewpoint information and time information for the scene, and use the one or more restoration models to generate time-series images according to the viewpoint information and the time information included in the request received from the client, and transmit the images in a transmission format that allows video playback on the client.

[0007] FIG. 1 is a first diagram illustrating an overview of a training process for a reconstruction model. FIG. 2 is a first diagram illustrating an overview of an image generation process using a trained reconstruction model. FIG. 3 is a first diagram illustrating an example of a trained reconstruction model applied to a server device. FIG. 4 is a first diagram illustrating an example of a system configuration of a free viewpoint video playback system. FIG. 5 is a diagram illustrating an example of a hardware configuration of a server device and a client terminal. FIG. 6 is a first diagram illustrating an example of a functional configuration of a server device. FIG. 7 is a diagram illustrating an example of a trained reconstruction model stored in a model storage unit of the server device according to the first embodiment. FIG. 8A is a first diagram illustrating a specific example of processing by the server device according to the first embodiment. FIG. 8B is a second diagram illustrating a specific example of processing by the server device according to the first embodiment. FIG. 8C is a third diagram illustrating a specific example of processing by the server device according to the first embodiment. FIG. 8D is a fourth diagram illustrating a specific example of processing by the server device according to the first embodiment. FIG. 9 is a first diagram illustrating an example of a functional configuration of a client terminal. FIG. 10 is a diagram illustrating an example of a video specification screen of a client terminal. FIG. 11 is a first diagram showing an example of a video playback screen of a client terminal. FIG. 12 is a second diagram showing an example of a video playback screen of a client terminal. FIG. 13 is a third diagram showing an example of a video playback screen of a client terminal. FIG. 14 is a first sequence diagram showing a flow of free viewpoint video playback processing by a free viewpoint video playback system. FIG. 15 is a second diagram for explaining an overview of a training process for a reconstruction model. FIG. 16 is a second diagram for explaining an overview of an image generation process using a trained reconstruction model. FIG. 17 is a second diagram showing an example of a trained reconstruction model applied to a server device. FIG. 18 is a diagram showing an example of a trained reconstruction model stored in a model storage unit of a server device according to the second embodiment. FIG. 19A is a first diagram showing a specific example of processing by a server device according to the second embodiment. FIG. 19B is a second diagram showing a specific example of processing by a server device according to the second embodiment. FIG. 20 is a third diagram for explaining an overview of a training process for a reconstruction model. FIG. 21 is a third diagram for explaining an overview of an image generation process using a trained reconstruction model.FIG. 22 is a third diagram showing an example of a trained reconstruction model applied to a server device. FIG. 23 is a diagram showing an example of a trained reconstruction model stored in a model storage unit of a server device according to the third embodiment. FIG. 24A is a first diagram showing a specific example of processing by a server device according to the third embodiment. FIG. 24B is a second diagram showing a specific example of processing by a server device according to the third embodiment. FIG. 25 is a fourth diagram for explaining an overview of training processing for a reconstruction model. FIG. 26 is a fourth diagram for explaining an overview of image generation processing using a trained reconstruction model. FIG. 27 is a fourth diagram showing an example of a trained reconstruction model applied to a server device. FIG. 28 is a diagram showing an example of a trained reconstruction model stored in a model storage unit of a server device according to the fourth embodiment. FIG. 29A is a first diagram showing a specific example of processing by a server device according to the fourth embodiment. FIG. 29B is a second diagram showing a specific example of processing by a server device according to the fourth embodiment. FIG. 30 is a fifth diagram for explaining an overview of training processing for a reconstruction model. FIG. 31 is a fifth diagram illustrating an overview of image generation processing using a trained reconstruction model. FIG. 32 is a fifth diagram illustrating an example of a trained reconstruction model applied to a server device. FIG. 33 is a diagram illustrating an example of a trained reconstruction model stored in a model storage unit of a server device according to the fifth embodiment. FIG. 34A is a first diagram illustrating a specific example of processing by a server device according to the fifth embodiment. FIG. 34B is a second diagram illustrating a specific example of processing by a server device according to the fifth embodiment. FIG. 35 is a second sequence diagram illustrating a flow of free viewpoint video playback processing by a free viewpoint video playback system. FIG. 36 is a second diagram illustrating an example of a system configuration of a free viewpoint video playback system. FIG. 37 is a second diagram illustrating an example of a functional configuration of a server device. FIG. 38 is a diagram illustrating an example of a trained reconstruction model stored in a model storage unit of a server device according to the sixth embodiment. FIG. 39A is a first diagram illustrating a specific example of processing by a server device according to the sixth embodiment. FIG. 39B is a second diagram illustrating a specific example of processing by a server device according to the sixth embodiment. Fig. 39C is a third diagram showing a specific example of processing by the server device according to the sixth embodiment. Fig. 40 is a second diagram showing an example of the functional configuration of a client terminal.FIG. 41 is a third sequence diagram showing the flow of free viewpoint video playback processing by a free viewpoint video playback system. FIG. 42 is a diagram showing an example of a trained reconstruction model stored in a model storage unit of a server device according to the seventh embodiment. FIG. 43A is a first diagram showing a specific example of processing by a server device according to the seventh embodiment. FIG. 43B is a second diagram showing a specific example of processing by a server device according to the seventh embodiment. FIG. 44 is a fourth sequence diagram showing the flow of free viewpoint video playback processing by a free viewpoint video playback system. FIG. 45 is a diagram showing an example of a trained reconstruction model stored in a model storage unit of a server device according to the eighth embodiment. FIG. 46A is a first diagram showing a specific example of processing by a server device according to the eighth embodiment. FIG. 46B is a second diagram showing a specific example of processing by a server device according to the eighth embodiment. FIG. 47 is a fifth sequence diagram showing the flow of free viewpoint video playback processing by a free viewpoint video playback system. FIG. 48 is a diagram showing an example of a trained reconstruction model stored in a model storage unit of a server device according to the ninth embodiment. FIG. 49A is a first diagram showing a specific example of processing by a server device according to the ninth embodiment. FIG. 49B is a second diagram showing a specific example of processing by a server device according to the ninth embodiment. FIG. 50 is a sixth sequence diagram showing a flow of free viewpoint video playback processing by a free viewpoint video playback system. FIG. 51 is a third diagram showing an example of the functional configuration of a server device. FIG. 52 is a third diagram showing an example of the functional configuration of a client terminal. FIG. 53 is a diagram showing an example of a trained restored model stored in a model storage unit of a server device according to a tenth embodiment. FIG. 54A is a first diagram showing a specific example of processing by a server device according to the tenth embodiment. FIG. 54B is a second diagram showing a specific example of processing by a server device according to the tenth embodiment. FIG. 55 is a seventh sequence diagram showing a flow of free viewpoint video playback processing by a free viewpoint video playback system.

[0008] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0009] [First embodiment] <Outline of training process of reconstruction model> First, an outline of the training process of a reconstruction model will be described using an example of a reconstruction model to which NeRF technology is applied as a reconstruction model that reconstructs a three-dimensional scene (hereinafter also referred to as a scene). Fig. 1 is a first diagram for explaining the outline of the training process of a reconstruction model.

[0010] In FIG. 1, a restoration model 110, which is an example of a restoration model for restoring a three-dimensional scene, is a neural network (NN) to which the NeRF technique is applied. θ In the training process 100, the reconstruction model 110 (F θ ) includes: coordinate information specifying the coordinates of a 3D point in the 3D scene 140 (e.g., (x １ , y １ , z １ )) and viewpoint information (e.g., viewpoint information (θ １ , φ １ )) and are input. This results in the restoration model 110 (F θ ) is a combination of coordinate information and viewpoint information of an input 3D point. １ , G １ , B １ )) and the opacity of the 3D point (e.g., σ １ , and the combination of the opacity specified by θ ) calculates the color and opacity of a certain 3D point at a certain viewpoint. Note that hereinafter, the coordinate information of the 3D point and the viewpoint information may be referred to as the 3D point and the viewpoint, respectively.

[0011] In the training process 100, a reconstruction model 110 (F θ ) similar processing is performed for multiple viewpoints. The example in FIG. 1 shows how similar processing is performed for two viewpoints (viewpoint 1, viewpoint 2).

[0012] Specifically, in the training process 100, a reconstruction model 110 (F θ) also includes: a 3D point in the 3D scene 140 (e.g., (x ２ , y ２ , z ２ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray2) from viewpoint 2 to the three-dimensional point (e.g., viewpoint information (θ ２ , φ ２ )) and are input. This results in the restoration model 110 (F θ ) is a combination of the input 3D point and viewpoint information. The color of the 3D point (for example, (R ２ , G ２ , B ２ )) and the opacity of the 3D point (e.g., σ ２ The output is a combination of (opacity specified by) and

[0013] In the training process 100 shown in FIG. 1, for each viewpoint (e.g., viewpoints 1 and 2), a reconstruction model 110 (F θ ) are subjected to a volume rendering process 120.

[0014] The volume rendering process 120 calculates the color of each pixel of an image seen from a certain viewpoint using a volume rendering method. Specifically, the volume rendering process 120 calculates the color of each pixel by using the restored model 110 (F θ ) and performs volume rendering using a predetermined product-sum operation based on the color and opacity output from the volume rendering processor 120. As a result, the volume rendering processor 120 generates a view image of a certain viewpoint. Note that the view image refers to an image of a scene seen from a specific viewpoint (i.e., an image based on specific viewpoint information) among free viewpoint images, which are images of a scene seen from various viewpoints (i.e., images based on various viewpoint information).

[0015] 1 , a loss calculation process 130 is performed on each of the generated view image from viewpoint 1 and the view image from viewpoint 2. For example, the view image from viewpoint 1 is compared with a captured image A captured by an imaging device at viewpoint 1 to calculate an error. Also, the view image from viewpoint 2 is compared with a captured image B captured by an imaging device at viewpoint 2 to calculate an error.

[0016] The error calculated in the loss calculation process 130 is used as the restoration model 110 (F θ ), the restoration model 110 (F θ ) is backpropagated. This results in the restored model 110 (F θ ) model parameters are updated. θ ) is trained to the reconstruction model 110 (F θ ) model parameters are updated, so that the trained reconstruction model (F θ ) is generated.

[0017] In order to simplify the explanation, we have omitted cases where training processing is performed using images captured by an imaging device at a viewpoint other than viewpoints 1 and 2, but images captured by an imaging device at a viewpoint other than viewpoints 1 and 2 may also be used in training processing.

[0018] <Outline of Image Generation Processing Using a Trained Reconstruction Model> Next, an outline of image generation processing using a trained reconstruction model will be described. Fig. 2 is a first diagram for explaining an outline of image generation processing using a trained reconstruction model.

[0019] As shown in FIG. 2, the image generation process for generating a view image of the viewpoint ij is performed by using a three-dimensional point (x ｎ , y ｎ , z ｎ ) and viewpoint information (θ ｉ , φ ｊ ) into the trained reconstruction model 210 (F θ), and calculates the color and opacity of each 3D point as its output. Then, the image generation process performs volume rendering process 120 based on the calculated color and opacity of each 3D point for each pixel of the view image, thereby generating a view image for viewpoint ij.

[0020] <Relationship between Captured Images and Trained Reconstruction Model> Next, a trained reconstruction model applied to the server device according to the first embodiment will be described. Fig. 3 is a first diagram showing an example of a trained reconstruction model applied to the server device. Note that, for the sake of simplicity, Fig. 3 also shows a case in which two viewpoints, viewpoint 1 and viewpoint 2, are used. However, as described above, in the training process, captured images captured by an imaging device at a viewpoint other than viewpoint 1 and viewpoint 2 may be used.

[0021] As shown in Fig. 3, a set of trained reconstruction models is applied to the server device. The set of trained reconstruction models is trained in advance to be able to reconstruct a scene from a first time to a second time using a time series of captured images obtained by capturing a scene successively in time from multiple viewpoints.

[0022] Specifically, the server device stores: time information T １ A photographed image A is photographed by an imaging device at viewpoint 1. １ and time information T １ Image B is taken by the imaging device at viewpoint 2 １ and a trained reconstruction model (F θ１ ) applies.

[0023] Similarly, the server device stores the following: time information T ２ A photographed image A is photographed by an imaging device at viewpoint 1. ２ and time information T ２ Image B is taken by the imaging device at viewpoint 2 ２ and a trained reconstruction model (F θ２ ) applies.

[0024] In the example of FIG. 3, due to space limitations, the time information T １１The trained reconstruction model (F θ１１ ), but the number of trained restoration models applied to the server device is not limited to 11. However, each trained restoration model is associated with time information and managed as a time-series trained restoration model.

[0025] In FIG. 3, the time information T １ , T ２ , T ３ , ... are the images A taken by the imaging device during the training process. １ , A ２ , ...or photographed image B １ , B ２ , ... (an example of a first time interval, for example, 30 fps). That is, a trained restoration model (an example of a first restoration model) of a time series of the frame period is applied to the server device to generate view images of a time series of the frame period.

[0026] <System Configuration of Free Viewpoint Video Playback System> Next, a description will be given of a system configuration of a free viewpoint video playback system including a server device according to the first embodiment. Fig. 4 is a first diagram showing an example of the system configuration of the free viewpoint video playback system.

[0027] 4 , the free viewpoint video playback system 400 includes a server device 410 according to the first embodiment and a client terminal 420. In the free viewpoint video playback system, the server device 410 and the client terminal 420 are communicably connected via a communication network 430.

[0028] A free viewpoint image generation program is installed in the server device 410 , and the server device 410 functions as a free viewpoint image generation unit 411 by executing the program.

[0029] The free viewpoint image generation unit 411 receives a request from the client terminal 420 via the communication network 430, and based on the time information and viewpoint information contained in the received request, reads out and executes a trained reconstruction model held in the model storage unit 606 described later.

[0030] As a result, the free viewpoint image generation unit 411 transmits the view image at each time information generated by executing the trained restoration model corresponding to each time information in a transmission format that allows video playback.

[0031] A playback program is installed in the client terminal 420, and by executing this program, the client terminal 420 functions as a playback unit 421. The playback program may be a dedicated application or a predetermined browser.

[0032] The playback unit 421 transmits a request including the time information and viewpoint information input by the user 440 to the server device 410 via the communication network 430 .

[0033] In addition, in response to sending a request to the server device 410, the playback unit 421 receives time-series view images sent from the server device 410, and plays back a free viewpoint video in which the received time-series view images are images of each frame of the video (frame images).

[0034] <Hardware Configuration of Server Device and Client Terminal> Next, a description will be given of the hardware configuration of the server device 410 and the client terminal 420. Fig. 5 is a diagram showing an example of the hardware configuration of the server device and the client terminal. Note that the server device 410 and the client terminal 420 have similar hardware configurations, so the hardware configuration of the server device 410 will be described here.

[0035] The server device 410 has, as its components, a processor 501, a main storage device 502 (memory), an auxiliary storage device 503 (memory), a network interface 504, and a device interface 505. The server device 410 may be realized as a computer in which these components are connected via a bus 506. Note that, although the example of Fig. 5 shows the server device 410 as including one of each component, the server device 410 may also include multiple of the same component.

[0036] Various computations of the server device 410 may be executed in parallel using one or more processors. Furthermore, various computations may be distributed to multiple computing cores in the processor 501 and executed in parallel. Furthermore, some or all of the processes, means, etc. disclosed herein may be executed by an external device 510 (at least one of a processor and a storage device) provided on a cloud that can communicate with the server device 410 via the network interface 504.

[0037] The processor 501 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.). The processor 501 may also be a semiconductor device including a dedicated processing circuit. The processor 501 is not limited to an electronic circuit using electronic logic elements, and may be realized by an optical circuit using optical logic elements. The processor 501 may also include an arithmetic function based on quantum computing.

[0038] The processor 501 performs various calculations based on various data and commands input from each device, etc., in the internal configuration of the server device 410, and outputs the calculation results and control signals to each device, etc. The processor 501 controls each component included in the server device 410 by executing an OS (Operating System), applications, etc.

[0039] Furthermore, processor 501 may refer to one or more electronic circuits arranged on a single chip, or may refer to one or more electronic circuits arranged on two or more chips or devices. When multiple electronic circuits are used, the respective electronic circuits may communicate with each other via wires or wirelessly.

[0040] The main memory device 502 is a memory device that stores instructions executed by the processor 501 and various data, and the various data stored in the main memory device 502 is read by the processor 501. The auxiliary memory device 503 is a memory device other than the main memory device 502, and for example, realizes a model storage unit 606 described below. Note that these memory devices refer to any electronic component that can store various data, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory device for saving various data in the server device 410 may be realized by the main memory device 502 or the auxiliary memory device 503, or may be realized by an internal memory built into the processor 501.

[0041] Furthermore, multiple processors 501 may be connected (coupled) to one main storage device 502, or a single processor 501 may be connected. Alternatively, multiple main storage devices 502 may be connected (coupled) to one processor 501. When the server device 410 is configured with at least one main storage device 502 and multiple processors 501 connected (coupled) to this at least one main storage device 502, the server device 410 may include a configuration in which at least one processor of the multiple processors 501 is connected (coupled) to at least one main storage device 502.

[0042] The network interface 504 is an interface for connecting to the communication network 430 wirelessly or via a wire.

[0043] The device interface 505 is an interface such as a USB that directly connects to an external device 520 .

[0044] The external device 520 may be, for example, an input device. In this embodiment, the input device is, for example, a keyboard, a mouse, a touch panel, or other device, and provides acquired information to the server device 410.

[0045] Furthermore, the external device 520 may be, for example, an output device. In this embodiment, the output device may be, for example, a display device such as an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), or an organic EL (Electro Luminescence) panel, or may be a speaker that outputs sound or the like.

[0046] The external device 520 may also be a storage device (memory). For example, the external device 520 may be a network storage or a storage such as an HDD.

[0047] Furthermore, the external device 520 may be a device having some of the functions of the components of the server device 410. In other words, the server device 410 may transmit and receive processing results to and from the external device 520.

[0048] <Functional Configuration of Server Device> Next, the functional configuration of the server device 410 will be described. Fig. 6 is a first diagram showing an example of the functional configuration of the server device. As described above, the server device 410 functions as a free viewpoint image generation unit 411. As shown in Fig. 6, the free viewpoint image generation unit 411 further includes a video designation receiving unit 601, a default video generation unit 602, a request receiving unit 603, a requested video generation unit 604, and a video transmission unit 605.

[0049] The video designation receiving unit 601 receives a designation of a free viewpoint video from the client terminal 420. In this embodiment, the server device 410 is configured to be able to play a plurality of free viewpoint videos on the client terminal 420, and here, the video designation receiving unit 601 receives a designation of one of the free viewpoint videos. The video designation receiving unit 601 notifies the default video generation unit 602 of identification information (e.g., a free viewpoint video identifier (ID)) for uniquely identifying the free viewpoint video whose designation has been received.

[0050] The default video generation unit 602 reads out from the model storage unit 606 a group of trained restoration models for generating view images included in the free viewpoint video identified by the identification information notified by the video specification receiving unit 601.

[0051] The default video generation unit 602 also inputs default viewpoint information into the group of trained restoration models that have been read out, and generates view images for each time (each time instant) according to the default viewpoint information. The view images according to the default viewpoint information generated by the default video generation unit 602 are notified to the video transmission unit 605.

[0052] The request receiving unit 603 receives a request from the client terminal 420. In this embodiment, the request transmitted from the client terminal 420 includes time information and viewpoint information. The request received by the request receiving unit 603 is notified to the requested video generating unit 604.

[0053] The requested video generation unit 604 performs processing according to the type of time information included in the request notified by the request receiving unit 603. For example, assume that the time information included in the request is time information based on a playback instruction in the client terminal 420. This time information may be, for example, the time at which the user 440 issued a playback instruction for the video, regardless of whether playback is in progress or stopped in the client terminal 420. In this case, the requested video generation unit 604 sequentially inputs the viewpoint information included in the request into the trained reconstruction models that have already been read out and that correspond to the time information notified by the request receiving unit 603. As a result, the requested video generation unit 604 sequentially generates view images according to the time information and viewpoint information included in the request and notifies the video transmission unit 605.

[0054] Also, for example, suppose the time information included in the request is time information based on a stop instruction in the client terminal 420 (an example of time information according to a termination condition). This time information may be, for example, the time at which the user 440 issues an instruction to stop playback of a video being played on the client terminal 420. In this case, the requested video generation unit 604 identifies the trained reconstruction model corresponding to the time information notified by the request reception unit 603 as the last trained reconstruction model being played, from among the trained reconstruction models already read, and inputs the viewpoint information included in the request. Then, the requested video generation unit 604 generates a last view image according to the time information and viewpoint information included in the request, notifies the video transmission unit 605, and then stops processing.

[0055] Also, for example, assume that the time information included in the request is time information based on an operation instruction during a pause in the client terminal 420. This time information may be, for example, a time based on an operation instruction (for example, an operation instruction on a seek bar indicator, which will be described later) given by the user 440 regarding a scene to be displayed while the video is paused in the client terminal 420. In this case, each time time information is notified by the request receiving unit 603, the requested video generating unit 604 inputs the viewpoint information included in the request into a trained restoration model corresponding to the time information, thereby generating a view image and notifying the video transmitting unit 605.

[0056] The video transmitting unit 605 transmits the view image according to the default viewpoint information notified by the default video generating unit 602 in a transmission format that allows the client terminal 420 to play the video.

[0057] Furthermore, the video transmitting unit 605 transmits a view image according to the time information and viewpoint information included in the request notified by the requested video generating unit 604 in a transmission format that allows video playback on the client terminal 420 .

[0058] Note that transmitting in a transmission format capable of video playback includes, for example, transmitting the view image as is to client terminal 420. Also, transmitting in a transmission format capable of video playback includes, for example, performing video encoding processing on the view image and then transmitting it to client terminal 420. Any encoding method may be used when performing video encoding processing on the view image and then transmitting it to client terminal 420, and for example, video encoding processing may be performed using H.264 / MPEG4. Also, when performing video encoding processing on the view image and then transmitting it to client terminal 420, the view image that has undergone video encoding processing is restored by client terminal 420. As a result, free viewpoint video is played back in client terminal 420 using the restored view image as frame images.

[0059] <Example of Trained Recovery Model> Next, a trained recovery model stored in the model storage unit 606 in the server device 410 according to the first embodiment will be described. Fig. 7 is a diagram showing an example of a trained recovery model stored in the model storage unit of the server device according to the first embodiment.

[0060] As shown in FIG. 7, the trained reconstruction model stored in the model storage unit 606 is associated with time information. Specifically, the time information T １ is a trained reconstruction model F θ１ is associated with the time information T ２ is a trained reconstruction model F θ２ Similarly, in the example of FIG. 7, the time information T ３ ~T １１ Then, the trained reconstruction model F θ３ ~F θ１１ The correspondence between the time information and the trained reconstruction model may be achieved by directly associating the time information with the trained reconstruction model, or by indirectly associating the time information with the trained reconstruction model via other data.

[0061] The server device 410 uses the trained restoration model stored in the model storage unit 606 to generate time-series view images according to the viewpoint information and time information included in the request received from the client terminal 420 .

[0062] In FIG. 7, the time information T １ , T ２ , T ３ , ... correspond to the frame period of the images captured by the imaging device during the training process, as described above. １ , T ２ , T ３ , . . . correspond to the frame period when playing back free viewpoint video in the free viewpoint video playback system 400.

[0063] 7, the trained restoration models associated with each piece of time information are different from each other. The different trained restoration models are configured by neural networks to which NeRF technology is applied, and are trained using different training data (captured images). The neural network architectures may be the same or may be partially different.

[0064] Each trained restoration model shown in FIG. 7 can generate a view image (free viewpoint image) from an arbitrary viewpoint for a scene at each time point.

[0065] 7, the model storage unit 606 stores at least a group of trained restoration models for generating view images for a series of scenes for one object. However, the group of trained restoration models stored in the model storage unit 606 is not limited to one, and the model storage unit 606 may store another group of trained restoration models for generating view images for a series of scenes for another object.

[0066] 7, the group of trained reconstruction models stored in the model storage unit 606 is shown as a set of trained reconstruction models with time information T １ ~T １１However, the number of trained reconstruction models included in the group of trained reconstruction models held by the model storage unit 606 is not limited to this.

[0067] <Specific Example of Processing by Server Device> Next, a specific example of processing by each unit (here, the default moving image generating unit 602 and the requested moving image generating unit 604) of the server device 410 according to the first embodiment will be described.

[0068] (1) Specific Example of Processing by Default Video Generation Unit First, a specific example of processing by the default video generation unit 602 will be described. Fig. 8A is a first diagram showing a specific example of processing by the server device according to the first embodiment. Fig. 8A shows a specific example of processing when the video designation acceptance unit 601 accepts designation of a free viewpoint video and the default video generation unit 602 receives notification of identification information of the designated free viewpoint video from the video designation acceptance unit 601.

[0069] As shown in FIG. 8A , the default video generation unit 602, which has received the notification of the identification information of the specified free viewpoint video, generates a trained restoration model F for generating a view image included in the specified free viewpoint video. θ１ ~F θ１１ is read from the model storage unit 606.

[0070] In addition, the default video generation unit 602 uses the read trained restoration model F θ１ ~F θ１１ For each, the default viewpoint information (θ ０ , φ ０ ) to obtain the trained reconstruction model F θ１ ~F θ１１ So, the default viewpoint information (θ ０ , φ ０ ) the view image X １ ~ View Image X １１ Generate.

[0071] In addition, the default video generating unit 602 generates the view image X １ ~ View Image X １１ time information T１ ~T １１ and notifies the video transmission unit 605. １ ~ View Image X １１ is transmitted in a transmission format that allows video playback on the client terminal 420.

[0072] (2) Specific example 1 of processing by the requested video generation unit Default viewpoint information (θ ０ , φ ０ ) the view image X １ ~ View Image X １１ It is assumed that a free viewpoint video having frame images of is played back by the client terminal 420. It is also assumed that a request including time information and viewpoint information is transmitted from the client terminal 420 in response to the free viewpoint video being played back by the client terminal 420. In this case, the request receiving unit 603 receives the request and notifies the requested video generating unit 604.

[0073] Here, a specific example of processing by the requested video generation unit 604 when a request (time information, viewpoint information) is notified from the request receiving unit 603 will be described. Fig. 8B is a second diagram showing a specific example of processing by the server device according to the first embodiment, and shows a specific example of processing by the requested video generation unit 604 when a request is notified from the request receiving unit 603.

[0074] As shown in FIG. 8B, the requested video generator 604 uses the trained reconstruction model F θ１ ~F θ１１ The time information included in the request (in the example of FIG. 8B, T ３ ) the corresponding trained reconstruction model F θ３ Identify.

[0075] In addition, the requested video generation unit 604 uses the identified trained reconstruction model F θ３ In contrast, the viewpoint information included in the request (in the example of FIG. 8B, (θ ｘ , φ ｘ )) to obtain the trained reconstruction model F θ３ Then, the viewpoint information (θ ｘ , φ ｘ) based on the time information T ３ View image X in ３ Generate.

[0076] Next, the requested video generation unit 604 selects a trained reconstruction model F corresponding to the next time information (next time) as the next trained reconstruction model. θ４ In addition, the requested video generation unit 604 identifies the identified trained reconstruction model F θ４ In contrast, the viewpoint information included in the request (in the example of FIG. 8B, (θ ｘ , φ ｘ )) to obtain the trained reconstruction model F θ４ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ４ View image X in ４ Generate.

[0077] Thereafter, the requested video generating unit 604 repeats the same process until an end condition is transmitted from the client terminal 420. In the example of FIG. 8B, the client terminal 420 transmits time information T １０ is shown to have been sent.

[0078] The client terminal 420 receives time information T １０ When the time information T １０ A trained reconstruction model F corresponding to θ１０ In addition, the requested video generation unit 604 identifies the identified trained reconstruction model F θ１０ In contrast, the viewpoint information included in the request (in the example of FIG. 8B, (θ ｘ , φ ｘ )) to obtain the trained reconstruction model F θ１０ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T １０ View image X in １０ Generate.

[0079] In this way, the requested video generation unit 604 generates a time series of view images according to the viewpoint information using a time series of trained restoration models ranging from a trained restoration model corresponding to the time information included in the request to a trained restoration model corresponding to a specified termination condition.

[0080] The termination condition here refers to time information based on a stop instruction to stop playback of a free viewpoint video in response to a request. When a stop button to stop a free viewpoint video being played is pressed, the client terminal 420 transmits time information corresponding to the timing of the button press to the server device 410 as the termination condition. However, the termination condition transmitted by the client terminal 420 is not limited to this. For example, when playing back a free viewpoint video, if a time range is specified and the client terminal 420 receives the specification, the client terminal 420 transmits time information corresponding to the end timing of the time range to the server device 410 as the termination condition.

[0081] Furthermore, the termination condition is not necessarily transmitted by the client terminal 420. For example, when the stop button is not pressed for the free viewpoint video being played, the trained restoration model corresponding to the last piece of time information among the trained restoration models in the time series becomes the trained restoration model corresponding to the predetermined termination condition.

[0082] The requested video generation unit 604 generates the view image X ３ ~ View Image X １０ time information T ３ ~T １０ The video transmitting unit 605 then notifies the video transmitting unit 605 of the view image X ３ ~ View Image X １０ can be transmitted in a transmission format that allows video playback on the client terminal 420.

[0083] (3) Specific example 2 of processing by the requested video generation unit Viewpoint information (θ ｘ , φ ｘ ) of the scene viewed from a viewpoint based on ３ ~ View Image X １０Assume that a request including time information is transmitted from the client terminal 420 in response to playback of a free viewpoint video having frame images of [frame image] on the client terminal 420. In this case, the request receiving unit 603 receives the request and notifies the requested video generating unit 604.

[0084] Here, a specific example of processing by the requested moving image generating unit 604 when a request (time information) is notified from the request receiving unit 603 will be described. Fig. 8C is a third diagram showing a specific example of processing by the server device according to the first embodiment, and shows a specific example of processing by the requested moving image generating unit 604 when a request is notified from the request receiving unit 603.

[0085] As shown in FIG. 8C, the requested video generator 604 uses the trained reconstruction model F θ１ ~F θ１１ The time information included in the request (in the example of FIG. 8C, T １ ) the corresponding trained reconstruction model F θ１ Identify.

[0086] In addition, the requested video generation unit 604 uses the identified trained reconstruction model F θ１ In the example of FIG. 8C, viewpoint information is not included in the request, so the requested video generation unit 604 inputs viewpoint information included in the most recent request (in the example of FIG. 8C, (θ ｘ , φ ｘ )) is used as input. This allows us to create a trained recovery model. θ１ Now, the current viewpoint information (θ ｘ , φ ｘ ) based on the time information T １ View image X in １ Generate.

[0087] Next, the requested video generation unit 604 selects a trained reconstruction model F corresponding to the next time information (next time) as the next trained reconstruction model. θ２ In addition, the requested video generation unit 604 identifies the identified trained reconstruction model F θ２ , the viewpoint information included in the most recent request (in the example of FIG. 8C, (θ ｘ , φ ｘ)) to create a trained recovery model. θ２ Now, the current viewpoint information (θ ｘ , φ ｘ ) based on the time information T ２ View image X in ２ Generate.

[0088] Thereafter, the requested video generating unit 604 repeats the same process until an end condition is transmitted from the client terminal 420. In the example of FIG. 8C, the client terminal 420 transmits time information T １０ is shown to have been sent.

[0089] The client terminal 420 receives time information T １０ When the time information T １０ A trained reconstruction model F corresponding to θ１０ In addition, the requested video generation unit 604 identifies the identified trained reconstruction model F θ１０ In contrast, the current viewpoint information (in the example of FIG. 8C, (θ ｘ , φ ｘ )) to create a trained recovery model. θ１０ Now, the current viewpoint information (θ ｘ , φ ｘ ) based on the time information T １０ View image X in １０ Generate.

[0090] The requested video generation unit 604 generates the view image X １ ~ View Image X １０ time information T １ ~T １０ The video transmitting unit 605 then notifies the video transmitting unit 605 of the time information and the current viewpoint information (θ ｘ , φ ｘ ) corresponding to the view image X １ ~ View Image X １０ can be transmitted in a transmission format that allows video playback on the client terminal 420.

[0091] (4) Specific Example 3 of Processing by Requested Video Generator Next, we will explain another specific example (a specific example different from Specific Example 2) of processing by the requested video generator 604 when a request (time information, viewpoint information) is notified from the request receiving unit 603. In Specific Example 2, the requested video generator 604 identified the next trained restoration model at a time interval corresponding to the frame period.

[0092] On the other hand, in the free viewpoint video playback system 400, not all view images generated by the identified trained restoration model can necessarily be played back as frame images on the client terminal 420. For example, in the following cases: - the frame period on the client terminal 420 is longer than the time interval at which view images are generated by the requested video generation unit 604; - the display mode on the client terminal 420 is double-speed mode or 10-second skip mode; - the communication load between the server device 410 and the client terminal 420 is high, resulting in a decrease in communication speed; - the processing load on the server device 410 or the client terminal 420 is high. Here, a specific example of processing (thinning processing) by the requested video generation unit 604 in a case where not all view images can be played back as frame images on the client terminal 420 will be described. FIG. 8D is a fourth diagram showing a specific example of processing by the server device according to the first embodiment.

[0093] As shown in FIG. 8D, the requested video generator 604 uses the trained reconstruction model F θ１ ~F θ１１ The time information included in the request (in the example of FIG. 8D, T ３ ) the corresponding trained reconstruction model F θ３ Identify.

[0094] In addition, the requested video generation unit 604 uses the identified trained reconstruction model F θ３ In contrast, the viewpoint information included in the request (in the example of FIG. 8D, (θ ｘ , φ ｘ )) to create a trained recovery model.θ３ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ３ View image X in ３ Generate.

[0095] Next, the requested video generation unit 604 determines the timing of generating a view image when identifying the next trained restoration model. The requested video generation unit 604 acquires information on: the frame period of the client terminal 420; the display mode of the client terminal 420; the communication load between the server device 410 and the client terminal 420; and the processing load of the server device 410 and the client terminal 420, and determines the timing of generating a view image based on the acquired information.

[0096] In the example of FIG. 8D, the requested video generation unit 604 determines the generation timing of the view image based on time information=T ６ and the trained reconstruction model F is used as the next trained reconstruction model. θ６ 8D shows how the requested video generator 604 identifies the trained restoration model F θ６ In contrast, the viewpoint information included in the request (in the example of FIG. 8D, (θ ｘ , φ ｘ )) is input. This allows the trained recovery model θ６ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ６ View image X in ６ Generate.

[0097] 8D, the requested video generating unit 604 repeats the same process (thinning-out process) until an end condition is transmitted from the client terminal 420. In the example of FIG. 8D, the client terminal 420 transmits time information T １０ is shown to have been sent.

[0098] The client terminal 420 receives time information T １０When the view image X is transmitted, the requested video generating unit 604 determines that it is not time to generate a view image, and １０ The process stops without generating a

[0099] The requested video generation unit 604 generates the view image X ３ , X ６ , X ９ time information T ３ , T ６ , T ９ The video transmitting unit 605 then notifies the video transmitting unit 605 of the view image X ３ , X ６ , X ９ can be transmitted in a transmission format that allows video playback on the client terminal 420.

[0100] <Functional Configuration of Client Terminal> Next, the functional configuration of the client terminal 420 will be described. Fig. 9 is a first diagram showing an example of the functional configuration of a client terminal. As described above, the client terminal 420 functions as a playback unit 421. As shown in Fig. 9, the playback unit 421 further includes a video designation transmission unit 901, a video reception unit 902, a video playback unit 903, a video display unit 904, and a request transmission unit 905.

[0101] The video designation transmission unit 901 receives a designation for a free viewpoint video from the user 440 via a video designation screen (details of which will be described later). The video designation transmission unit 901 transmits, to the server device 410, identification information for uniquely identifying the free viewpoint video for which the designation has been received.

[0102] The video receiving unit 902 receives a view image transmitted from the server device 410 and notifies the video playback unit 903. Alternatively, the video receiving unit 902 receives a view image that has been subjected to video encoding processing and is transmitted from the server device 410, restores the view image that has been subjected to video encoding processing, and notifies the video playback unit 903.

[0103] The video playback unit 903 notifies the video display unit 904 of the notified view image at a predetermined frame cycle.

[0104] The video display unit 904 plays free viewpoint video, in which the notified view images are used as frame images at a predetermined frame period, on a video playback screen (details of which will be described later). Furthermore, the video display unit 904 accepts requests (either one or both of time information and viewpoint information) from the user 440 on the played video playback screen, and notifies the request transmission unit 905 of the requests.

[0105] As described above, the time information included in the request notified to the request sending unit 905 includes: time information based on a playback instruction; time information based on a stop instruction; and time information based on various operations during stop.

[0106] The request sending unit 905 sends the request (time information, viewpoint information) notified by the video display unit 904 to the server device 410 .

[0107] <Display Screen of Client Terminal> Next, the display screen (video selection screen, video playback screen) of the client terminal 420 will be described.

[0108] (1) Video Designation Screen First, the video designation screen will be described. Fig. 10 is a diagram showing an example of the video designation screen of the client terminal.

[0109] 10 , by accessing the server device 410, a video selection screen 1000 of the client terminal 420 displays a list of free viewpoint videos that can be provided by the server device 410. The example of Fig. 10 shows four free viewpoint videos displayed as free viewpoint videos that can be provided by the server device 410.

[0110] The user 440 specifies a free viewpoint video to be played from among the free viewpoint videos displayed on the video specification screen 1000. As a result, the video specification transmission unit 901 transmits identification information for uniquely identifying the specified free viewpoint video to the server device 410. The example in Fig. 10 shows how "video I" has been specified as a free viewpoint video by the user 440.

[0111] (2) Video Playback Screen 1 Next, a specific example of a video playback screen will be described. Fig. 11 is a first diagram showing an example of a display screen of a client terminal.

[0112] When the user 440 selects "video I," the video playback screen of the client terminal 420 switches to a video playback screen 1110, and the free viewpoint video of "video I" is played back. As shown in Fig. 11 , the video playback screen includes a video display area 1117 and an operation instruction area 1111. The operation instruction area 1111 includes a seek bar 1112, a stop button 1113, a play button 1114, a 10-second skip button 1115, and the like.

[0113] The seek bar 1112 is a bar that indicates, by an indicator 1112', the current playback position of the free viewpoint video being played back in the video display area 1117. The indicator 1112' of the seek bar 1112 moves from the left to the right of the page in conjunction with the passage of time in the video during playback of the free viewpoint video. The user 440 can use the mouse pointer 1116 to move the indicator 1112' to the left or right of the page.

[0114] This allows the user 440 to move the indicator 1112' to a desired position and play back the free viewpoint video corresponding to the time information of that position. In other words, in Fig. 11, moving the indicator 1112' is equivalent to sending a request including the time information of the destination to the server device 410.

[0115] The stop button 1113 stops the playback of the free viewpoint video when pressed by the user 440 while the free viewpoint video is being played back in the video display area 1117. In other words, in FIG. 11 , pressing the stop button 1113 is equivalent to inputting an end condition to the server device 410.

[0116] The play button 1114 is a button that, when pressed by the user 440, causes the free viewpoint video to be played back from the current stop position (the position of the current indicator 1112′) when playing back a free viewpoint video that is stopped in the video display area 1117. In other words, in Fig. 11 , pressing the play button 1114 is equivalent to transmitting a request including time information of the current stop position to the server device 410.

[0117] The 10-second skip button 1115 is a button that, when pressed by the user 440 while a free viewpoint video is being played, moves the current playback position (the position of the current indicator 1112′) to a playback position 10 seconds ahead or 10 seconds back. In Fig. 11 , pressing the 10-second skip button is equivalent to transmitting to the server device 410 a request including time information of the playback position 10 seconds ahead or 10 seconds back from the current playback position.

[0118] 11 , video playback screen 1120 shows the display screen after a predetermined time has elapsed since video playback screen 1110 was displayed. As the predetermined time has elapsed, the movement of the subject contained in video display area 1117 of video playback screen 1120 has changed from the movement of the subject contained in video display area 1117 of video playback screen 1110. In addition, the position of indicator 1112′ in operation instruction area 1111 of video playback screen 1110 has moved to the right on the page in video playback screen 1120.

[0119] (3) Video Playback Screen 2 Next, another specific example of the video playback screen will be described. Here, after the free viewpoint video of "video I" has been played back, the user 440 presses the stop button 1113 to stop the playback of the free viewpoint video of "video 1." In this state, the user 440 further: moves the indicator 1112' of the seek bar 1112 in the operation instruction area 1111 to input time information; and drags the video display area 1117 with the mouse pointer 1116 to input viewpoint information. Fig. 12 is a second diagram showing an example of the video playback screen of the client terminal.

[0120] In Figure 12, video playback screen 1130 shows that after video playback screen 1120 is displayed, stop button 1113 is pressed to stop playback, and the position of indicator 1112' is moved to the left side of the page by mouse pointer 1116.

[0121] As shown in video playback screen 1130, by moving indicator 1112' to the left on the page, a frame image corresponding to the time information of the position of indicator 1112' is displayed in video display area 1117 of video playback screen 1130. In this case, since the viewpoint information has not changed, a frame image of the same subject's movement as that included in video display area 1117 of video playback screen 1110, viewed from the same viewpoint, is displayed.

[0122] On the other hand, in FIG. 12, a video playback screen 1140 shows a state in which, after the video playback screen 1130 is displayed, the video display area 1117 is dragged downward with the mouse pointer 1116, causing the viewpoint to rotate upward.

[0123] As shown in video playback screen 1140, by rotating the viewpoint upward, the viewpoint with respect to the subject included in video display area 1117 moves, and frame images of the scene viewed from above are displayed. Here, since the time information has not been changed, video display area 1117 on video playback screen 1140 displays frame images of the scene viewed from above, showing the same movements as those of the subject included in video display area 1117 on video playback screen 1130.

[0124] In the example of the video playback screen 1140, the video display area 1117 is shown being dragged downward by the mouse pointer 1116, but the direction in which the video display area 1117 is dragged is not limited to downward, and it can be dragged in any direction.

[0125] For example, suppose that video display area 1117 is dragged leftward on video playback screen 1130. In this case, video display area 1117 on video playback screen 1140 displays a frame image of a scene seen from the right side, showing the same movement as the subject included in video display area 1117 on video playback screen 1130.

[0126] Similarly, suppose that video display area 1117 is dragged rightward on video playback screen 1130. In this case, video display area 1117 on video playback screen 1140 displays a frame image of a scene seen from the left side, showing the same movement as the subject included in video display area 1117 on video playback screen 1130.

[0127] In response to the above operation on the client terminal 420, the server device 410 generates a view image according to the viewpoint information using a trained restoration model corresponding to the changed time information, for example, every time the time information is changed by the client terminal 420. Also, in the server device 410, every time the viewpoint information is changed by the client terminal 420, a view image according to the changed viewpoint information in the current time information is generated.

[0128] (4) Video Playback Screen 3 Next, another specific example of the video playback screen will be described. Here, the video playback screen will be described when the user 440 presses the play button 1114 in a state where viewpoint information is input by dragging the video display area 1117 downward with the mouse pointer 1116. Fig. 13 is a third diagram showing an example of the video playback screen of the client terminal.

[0129] 13 , a video playback screen 1150 shows a state in which the user 440 has pressed the play button 1114 after the video playback screen 1140 has been displayed. As shown in the video playback screen 1150, pressing the play button 1114 with the mouse pointer 1116 causes the free viewpoint video of "video 1" to be played from the current time information based on the input viewpoint information.

[0130] 13 , video playback screen 1160 shows a state in which a predetermined time has elapsed since play button 1114 was pressed on video playback screen 1150. As the predetermined time has elapsed, the movement of the subject contained in video display area 1117 on video playback screen 1160 has changed from the movement of the subject contained in video display area 1117 on video playback screen 1150. In addition, the position of indicator 1112′ in operation instruction area 1111 on video playback screen 1160 has moved further to the right on the page than the position of indicator 1112′ in operation instruction area 1111 on video playback screen 1150.

[0131] In addition, the video display area 1117 of the video playback screen 1160 displays frame images of a scene seen from above, showing the same movement as that of the subject included in the video display area 1117 of the video playback screen 1120.

[0132] <Flow of Free Viewpoint Video Playback Processing> Next, a description will be given of the flow of free viewpoint video playback processing by the free viewpoint video playback system 400. Fig. 14 is a first sequence diagram showing the flow of free viewpoint video playback processing by the free viewpoint video playback system.

[0133] In step S1420_1, the client terminal 420 receives a designation from the user 440 regarding a free viewpoint video to be displayed, and transmits, to the server device 410, identification information for uniquely identifying the designated free viewpoint video.

[0134] In step S1410_1, the server device 410 reads out a group of trained reconstruction models for generating view images included in the specified free viewpoint video. The server device 410 also adds default viewpoint information (θ ０ , φ ０ ) to view image X １ ~X １１ Generate.

[0135] In step S1410_2, the server device 410 sequentially transmits the generated view images to the client terminal 420.

[0136] In step S1420_2, the client terminal 420 plays back the free viewpoint video in which the view images transmitted from the server device 410 are used as frame images. The client terminal 420 also receives an instruction to stop the free viewpoint video being played back, and transmits the instruction to the server device 410. As a result, the server device 410 stops transmitting the view images.

[0137] In step S1420_3, the client terminal 420 accepts an instruction to move the indicator 1112' on the seek bar 1112. The client terminal 420 sequentially transmits to the server device 410 time information on each position of the indicator 1112' that is being moved.

[0138] In step S1410_3, each time the server device 410 receives time information for each position of the moving indicator 1112' from the client terminal 420, the server device 410 inputs default viewpoint information into the trained restoration model corresponding to the time information for each position. As a result, the server device 410 generates view images according to the time information for each position. Furthermore, the server device 410 sequentially transmits the generated view images to the client terminal 420. As a result, view images corresponding to the time information for each position of the moving indicator 1112' are displayed on the client terminal 420.

[0139] In step S1420_4, the client terminal 420 accepts the dragging of the moving image display area by the mouse pointer 1116. The client terminal 420 transmits to the server device 410 viewpoint information for each position of the moving mouse pointer 1116.

[0140] In step S1410_4, each time the server device 410 receives viewpoint information for each position of the moving mouse pointer 1116 from the client terminal 420, the server device 410 inputs the viewpoint information for each position into the trained restoration model corresponding to the current time information. The server device 410 then generates view images corresponding to the viewpoint information for each position. The server device 410 then sequentially transmits the generated view images to the client terminal 420. As a result, view images corresponding to the viewpoint information for each position of the moving mouse pointer 1116 are displayed on the client terminal 420.

[0141] In step S 1420 _ 5 , when the play button 1114 is pressed, the client terminal 420 transmits a play instruction to the server device 410 .

[0142] In step S1410_5, the server device 410 generates a view image by inputting the current viewpoint information into a trained reconstruction model corresponding to the current time information, and transmits the view image to the client terminal 420. Next, the server device 410 generates a view image by inputting the current viewpoint information into a trained reconstruction model corresponding to the next time information, and transmits the view image to the client terminal 420. Thereafter, the server device 410 repeats the same process until an end condition is transmitted from the client terminal 420.

[0143] In step S1420_6, the client terminal 420 plays back the free viewpoint video in which the view images transmitted from the server device 410 are used as frame images. The client terminal 420 also receives an instruction to stop the free viewpoint video being played back, and transmits it to the server device 410. As a result, the server device 410 stops generating and transmitting the view images.

[0144] <Summary> As is clear from the above description, the server device 410 according to the first embodiment includes one or more memories and one or more processors, and the one or more memories hold one or more trained restoration models (first restoration models) that have been trained in advance to be able to restore a scene from a first time to a second time using time-series captured images from multiple viewpoints obtained by capturing a scene continuously over time from each of the multiple viewpoints. The one or more trained restoration models (first restoration models) are time-series trained restoration models for a first time interval that generate time-series view images for a first time interval. More specifically, the one or more trained restoration models (first restoration models) are trained restoration models that respectively correspond one-to-one to different time information and are a plurality of trained restoration models that have been trained to output information of images at the corresponding time information.

[0145] Furthermore, the one or more processors: receive a request from a client terminal, the request including viewpoint information and time information for a scene; and use one or more trained restoration models to generate time-series view images according to the viewpoint information and time information included in the request received from the client terminal, and transmit the generated view images in a transmission format that enables video playback on the client terminal. More specifically, the one or more processors generate time-series view images for a first time interval according to the viewpoint information included in the request, using time-series trained restoration models (first restoration models) for a first time interval from a trained restoration model (first restoration model) corresponding to the time information included in the request to a trained restoration model (first restoration model) corresponding to a predetermined termination condition.

[0146] In this way, according to the first embodiment, a mechanism for playing back free viewpoint video can be constructed.

[0147] [Second Embodiment] In the above first embodiment, the model storage unit 606 has been described as holding one trained restoration model for each piece of time information, and one trained restoration model generates a view image at one piece of time information. However, the trained restoration model is not limited to this, and the model storage unit 606 may hold trained restoration models capable of generating view images at multiple pieces of continuous time information. Hereinafter, the second embodiment will be described, focusing on the differences from the above first embodiment.

[0148] <Outline of Training Process for Restoration Model> First, an outline of the training process for the restoration model applied to the server device 410 according to the second embodiment will be described. FIG. 15 is a second diagram for explaining the outline of the training process for the restoration model. The difference from the training process 100 described with reference to FIG. 1 in the first embodiment is that in the case of the training process 1500 shown in FIG. 15, the restoration model 110 (F θ ) includes: coordinate information specifying the coordinates of a 3D point in the 3D scene 140 (e.g., (x １ , y １ , z １ )) and viewpoint information (e.g., viewpoint information (θ １ , φ １ )), and time information (e.g., T=1) that specifies the time of the three-dimensional scene. θ ) is determined based on the combination of input coordinate information, viewpoint information, and time information. １１ , G １１ , B １１ )) and the opacity of the 3D point (e.g., σ １１ The reconstruction model 110 sequentially outputs a combination of the color and opacity of a certain 3D point at a certain viewpoint and a certain time. Note that hereinafter, the coordinate information, viewpoint information, and time information of the 3D point may be referred to as the 3D point, viewpoint, and time (or time instant), respectively.

[0149] Note that, similar to the training process 100, the training process 1500 also uses the reconstruction model 110 (F θ 15 shows how the same processing is performed for two viewpoints (viewpoint 1, viewpoint 2).

[0150] Specifically, in the training process 1500, the reconstruction model 110 (F θ ) includes: a 3D point in the 3D scene 140 (e.g., (x ２ , y ２ , z ２ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray2) from viewpoint 2 to the three-dimensional point (e.g., viewpoint information (θ ２ , φ ２ )), and time information specifying the time of the three-dimensional scene (for example, T=1), and θ ) is a combination of the input 3D point, viewpoint information, and time information. ２１ , G ２１ , B ２１ )) and the opacity of the 3D point (e.g., σ ２１ The combination of (opacity specified by) and is output sequentially.

[0151] Similarly to the training process 100, in the training process 1500, for each viewpoint (e.g., viewpoints 1 and 2), a reconstruction model 110 (F θ The combinations of color and opacity of the 3D points sequentially output from the 3D point processor 110 are subjected to a volume rendering process 120.

[0152] In this embodiment, the volume rendering process 120 calculates the color of each pixel of an image seen from a certain viewpoint at a certain time using a volume rendering method. Specifically, the volume rendering process 120 calculates the color of each pixel at a certain time by using the restored model 110 (F θ) is used to perform volume rendering using a predetermined product-sum operation based on the color and opacity output from the volume rendering process 120. As a result, the volume rendering process 120 generates a view image of a certain viewpoint at a certain time. The example in FIG. 15 shows how the volume rendering process 120 generates view images at each time information of viewpoint 1 (view images 11 to 13 of viewpoint 1) and view images at each time information of viewpoint 2 (view images 21 to 23 of viewpoint 2).

[0153] In addition, in the training process 1500 shown in FIG. 15, a loss calculation process 130 is performed on the view images at each time information of each generated viewpoint (view images 11 to 13 of viewpoint 1, and view images 21 to 23 of viewpoint 2).

[0154] For example, the view images at each time information of viewpoint 1 (view images 11 to 13 at viewpoint 1) are the captured images at each time information captured by the imaging device at viewpoint 1 (captured image A １ ~Photo A ３ ) and the error is calculated. Also, the view images at each time information of viewpoint 2 (view images 21 to 23 at viewpoint 2) are compared with the captured images at each time information captured by the imaging device at viewpoint 2 (captured images B １ ~Photo image B ３ ) to calculate the error.

[0155] The error calculated in the loss calculation process 130 is used as the restoration model 110 (F θ ), the restoration model 110 (F θ ) is backpropagated. This results in the restored model 110 (F θ ) model parameters are updated. θ ) training process updates the model parameters, so that the trained reconstruction model (F θ ) is generated.

[0156] In order to simplify the explanation, we have omitted cases where training processing is performed using images captured by an imaging device at a viewpoint other than viewpoints 1 and 2, but images captured by an imaging device at a viewpoint other than viewpoints 1 and 2 may also be used in training processing.

[0157] <Outline of Image Generation Processing Using a Trained Reconstruction Model> Next, an outline of image generation processing using a trained reconstruction model applied to the server device 410 according to the second embodiment will be described. Fig. 16 is a second diagram for explaining the outline of image generation processing using a trained reconstruction model.

[0158] As shown in FIG. 16, the image generation process for generating a view image of the viewpoint ij at time information T is performed by using a three-dimensional point (x ｎ , y ｎ , z ｎ ) and viewpoint information (θ ｉ , φ ｊ ) and time information T are used to train the reconstruction model 210 (F θ ), and calculates the color and opacity of each 3D point at the time information T as its output. Then, the image generation process performs a volume rendering process 120 based on the calculated color and opacity of each 3D point for each pixel of the view image to generate a view image of the viewpoint ij at the time information T. The image generation process uses a trained reconstruction model 210 (F θ ) sequentially inputting different time information T, view images (view image 1 to view image 3 of viewpoint ij) at different time information (for example, T=1, 2, 3) are sequentially generated.

[0159] <Relationship between Captured Images and Trained Reconstruction Model> Next, a trained reconstruction model applied to the server device 410 according to the second embodiment will be described. Fig. 17 is a second diagram showing an example of a trained reconstruction model applied to the server device. Note that, for the sake of simplicity, Fig. 17 also shows a case where two viewpoints, viewpoint 1 and viewpoint 2, are used. However, as described above, in the training process, captured images captured by an imaging device at a viewpoint other than viewpoint 1 and viewpoint 2 may be used.

[0160] As shown in FIG. 17 , the server device 410 is applied with: a group of trained restoration models that have been trained in advance to be able to restore a scene from a first time to a second time using a time series of captured images obtained by capturing the scene successively over time from multiple viewpoints.

[0161] Specifically, the server device 410 stores the following: time information T １ ~Time information T ３ A photographed image A is photographed by an imaging device at viewpoint 1. １ ~Photo A ３ and time information T １ ~Time information T ３ Image B is taken by the imaging device at viewpoint 2 １ ~Photo image B ３ and a trained reconstruction model F θ１_θ３ applies.

[0162] Similarly, the server device 410 stores the following: time information T ４ ~Time information T ６ A photographed image A is photographed by an imaging device at viewpoint 1. ４ ~Photo A ６ and time information T ４ ~Time information T ６ Image B is taken by the imaging device at viewpoint 2 ４ ~Photo image B ６ and a trained reconstruction model F θ４_θ６ applies.

[0163] In the example of FIG. 17, due to space limitations, the time information T １１ A trained reconstruction model F θ１０_θ１２ However, the number of trained restoration models applied to the server device 410 is not limited to four. However, each trained restoration model is associated with each piece of time information and managed as a time-series trained restoration model.

[0164] In FIG. 17, the time information T １ , T ４ , T ７, ... are the images A taken by the imaging device during the training process. １ , A ２ , ...or photographed image B １ , B ２ , ... corresponds to a second time interval that is longer than a frame period (an example of a first time interval) of 100 frames per second, 100 frames per second, .... In other words, a trained reconstruction model for generating time-series view images of the first time interval, which is a trained reconstruction model for a time series of the second time interval (an example of a second reconstruction model), is applied to the server device 410.

[0165] <Example of Trained Recovery Model> Next, a trained recovery model stored in the model storage unit 606 in the server device 410 according to the second embodiment will be described. Fig. 18 is a diagram showing an example of a trained recovery model stored in the model storage unit of the server device according to the second embodiment.

[0166] As shown in FIG. 18, the trained reconstruction model stored in the model storage unit 606 is associated with time information. Specifically, the time information T １ ~T ３ is a trained reconstruction model F θ１_θ３ is associated with the time information T ４ ~T ６ is a trained reconstruction model F θ４_θ６ Similarly, in the example of FIG. 18, the time information T ７ ~T ９ , T １０ ~T １２ Then, the trained reconstruction model F θ７_θ９ , F θ１０_θ１２ , respectively. That is, each model has its own corresponding (supporting) temporal information. The correspondence between the temporal information and the trained reconstruction model may be achieved by directly associating the temporal information with the trained reconstruction model, or by indirectly associating the temporal information with the trained reconstruction model via other data.

[0167] The server device 410 uses the trained restoration model stored in the model storage unit 606 to generate time-series view images according to the viewpoint information and time information included in the request received from the client terminal 420 .

[0168] In FIG. 18, the time information T １ , T ２ , T ３ , ... correspond to the frame period of the images captured by the imaging device during the training process, as described above. １ , T ２ , T ３ , . . . correspond to the frame period when playing back free viewpoint video in the free viewpoint video playback system 400.

[0169] 18, the trained restoration models associated with each piece of time information are different from each other. The different trained restoration models here are configured by neural networks to which the NeRF technology is applied, and are trained using different training data (captured images). The neural network architectures may be the same or may be partially different.

[0170] Each trained restoration model shown in FIG. 18 can generate a view image (free viewpoint image) from an arbitrary viewpoint for a scene at each time information.

[0171] 18 , the model storage unit 606 stores at least a group of trained restoration models for generating view images for a series of scenes for one target. However, the group of trained restoration models stored in the model storage unit 606 is not limited to one, and the model storage unit 606 may store another group of trained restoration models for generating view images for a series of scenes for another target.

[0172] 18, the group of trained reconstruction models stored in the model storage unit 606 is shown as a set of trained reconstruction models with time information T １ ~T １１ However, the number of trained reconstruction models included in the group of trained reconstruction models held by the model storage unit 606 is not limited to this.

[0173] <Specific Example of Processing by Server Device> Next, a specific example of processing by the default moving image generating unit 602 and the requested moving image generating unit 604 of the server device 410 according to the second embodiment will be described.

[0174] (1) Specific Example of Processing by Default Video Generation Unit First, a specific example of processing by the default video generation unit 602 will be described. Fig. 19A is a first diagram showing a specific example of processing by the server device 410 according to the second embodiment. Fig. 19A shows a specific example of processing when the video designation acceptance unit 601 accepts designation of a free viewpoint video and the default video generation unit 602 receives, from the video designation acceptance unit 601, notification of identification information of the designated free viewpoint video.

[0175] As shown in FIG. 19A , the default video generator 602 generates a trained reconstruction model F for generating view images included in a specified free viewpoint video. θ１_θ３ ~F θ１０_θ１２ is read from the model storage unit 606.

[0176] In addition, the default video generation unit 602 uses the read trained restoration model F θ１_θ３ ~F θ１０_θ１２ For each, the default viewpoint information (θ ０ , φ ０ ) and each time information. This creates a trained reconstruction model F θ１_θ３ ~F θ１０_θ１２ So, the default viewpoint information (θ ０ , φ ０ ) the view image X １ ~ View Image X １１ Generate.

[0177] In addition, the default video generating unit 602 generates the view image X １ ~ View Image X １１ time information T １ ~T １１ and notifies the video transmission unit 605. １ ~ View Image X １１ is transmitted in a transmission format that allows video playback on the client terminal 420.

[0178] (2) Specific example of processing by the requested video generation unit As described above, the default viewpoint information (θ ０ , φ ０ ) the view image X １ ~ View Image X １１ It is assumed that a free viewpoint video having frame images of is played back by the client terminal 420. In response to this, it is assumed that a request including time information and viewpoint information is transmitted from the client terminal 420. In this case, the request receiving unit 603 receives the request and notifies the requested video generating unit 604.

[0179] Here, a specific example of processing by the requested video generation unit 604 when a request (time information, viewpoint information) is notified from the request receiving unit 603 will be described. Fig. 19B is a second diagram showing a specific example of processing by the server device according to the second embodiment, and shows a specific example of processing by the requested video generation unit 604 when a request is notified from the request receiving unit 603.

[0180] As shown in FIG. 19B, the requested video generator 604 uses the trained reconstruction model F θ１_θ３ ~F θ１０_θ１２ The time information included in the request (in the example of FIG. 19B, T ３ ) the corresponding trained reconstruction model F θ１_θ３ Identify.

[0181] In addition, the requested video generation unit 604 uses the identified trained reconstruction model F θ１_θ３ In response to the request, the time information (in the example of FIG. 19B, T ３ ) and viewpoint information (in the example of FIG. 19B, (θ ｘ , φ ｘ )) to obtain the trained reconstruction model F θ１_θ３ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ３ View image X in ３ Generate.

[0182] Next, the requested video generation unit 604 selects the trained reconstruction model F as the next trained reconstruction model. θ４_θ６ In addition, the requested video generation unit 604 identifies the identified trained reconstruction model F θ４_θ６ For each time information (in the example of FIG. 19B, T ４ , T ５ , T ６ ) and viewpoint information included in the request (in the example of FIG. 19B, (θ ｘ , φ ｘ )) are input sequentially. This results in a trained reconstruction model F θ４_θ６ Then, the viewpoint information (θ ｘ , φ ｘ ) ４ ~T ６ View image X in ４ ~X ６ are generated sequentially.

[0183] Thereafter, the requested video generating unit 604 repeats the same process until an end condition is transmitted from the client terminal 420. In the example of FIG. 19B, the client terminal 420 transmits time information T １０ is shown to have been sent.

[0184] The client terminal 420 receives time information T １０ When the time information T １０ A trained reconstruction model F corresponding to θ１０_θ１２ In addition, the requested video generation unit 604 identifies the identified trained reconstruction model F θ１０_θ１２ For time information T １０ and viewpoint information included in the request (in the example of FIG. 19B, (θ ｘ , φ ｘ )) to obtain the trained reconstruction model F θ１０_θ１２ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T １０ View image X in １０ Generate.

[0185] In this way, the requested video generation unit 604 generates a time series of view images of a first time interval according to the viewpoint information using a time series of trained restoration models of a second time interval, from a trained restoration model corresponding to the time information included in the request to a trained restoration model corresponding to a predetermined termination condition.

[0186] The requested video generation unit 604 generates the view image X ３ ~ View Image X １０ time information T ３ ~T １０ The video transmitting unit 605 then notifies the video transmitting unit 605 of the view image X ３ ~ View Image X １０ can be transmitted in a transmission format that allows video playback on the client terminal 420.

[0187] <Summary> As is clear from the above description, the one or more memories included in the server device 410 according to the second embodiment hold a trained restoration model (second restoration model) for generating time-series view images at a first time interval, the trained restoration model (second restoration model) being a time-series trained restoration model at a second time interval that is longer than the first time interval. One or more trained restoration models (second restoration models) are held, and each of the one or more trained restoration models (second restoration models) is a trained restoration model that has been trained to output image information at input time information.

[0188] Furthermore, the one or more processors provided in the server device 410 according to the second embodiment: generate a time series of view images for a first time interval according to the viewpoint information included in the request, using a trained restoration model (second restoration model) of a time series for a second time interval from a trained restoration model (second restoration model) corresponding to the time information included in the request to a trained restoration model (second restoration model) corresponding to a predetermined termination condition.

[0189] As a result, according to the second embodiment, a mechanism for playing back free viewpoint video that is different from that of the first embodiment can be constructed.

[0190] [Third Embodiment] In the second embodiment, the model storage unit 606 stores a trained restoration model that generates view images at three consecutive pieces of time information as a trained restoration model that generates view images at multiple consecutive pieces of time information. However, the model storage unit 606 may store a trained restoration model that generates view images at a full time range as a trained restoration model that generates view images at multiple consecutive pieces of time information. Note that the full time range here refers to a finite time range captured by an imaging device, and in the third embodiment, this is described as, for example, three minutes. Note that if the frame rate is 30 fps, a three-minute free viewpoint video will include 5,400 frame images.

[0191] <Outline of Training Process of Restoration Model> First, an outline of the training process of the restoration model applied to the server device 410 according to the third embodiment will be described. FIG. 20 is a third diagram for explaining the outline of the training process of the restoration model. The difference from the training process 1500 described with reference to FIG. 15 in the second embodiment is that the training process 2000 shown in FIG. 20 does not use the restoration model 110 (F θ ) includes: a 3D point in the 3D scene 140 (e.g., (x １ , y １ , z １ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray1) from viewpoint 1 to the three-dimensional point (e.g., viewpoint information (θ １ , φ １ )), and time information (for example, T=1 to T=5400) corresponding to each three-dimensional point and viewpoint information, are input sequentially. θ ) is a combination of an input 3D point, viewpoint information, and time information. The color (for example, (R １_１ , G １_１ , B １_１ ) ~ (R １_５４００ , G １_５４００ , B １_５４００) and the opacity (e.g., σ １_１ , ~σ １_５４００ , and the combination of the opacity specified by F θ ) calculates the color and opacity of a 3D point at a certain viewpoint and time.

[0192] Note that, similar to the training process 1500, the training process 2000 also uses the reconstruction model 110 (F θ 20 shows how the same processing is performed for two viewpoints (viewpoint 1, viewpoint 2).

[0193] Specifically, in the training process 2000, the reconstruction model 110 (F θ ) also includes: a 3D point in the 3D scene 140 (e.g., (x ２_１ , y ２_１ , z ２_１ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray2) from viewpoint 2 to the three-dimensional point (e.g., viewpoint information (θ ２ , φ ２ )), and time information (for example, T=1 to T=5400) corresponding to each three-dimensional point and viewpoint information. θ ) is a combination of an input 3D point, viewpoint information, and time information. The color (for example, (R ２_１ , G ２_１ , B ２_１ ) ~ (R ２_５４００ , G ２_５４００ , B ２_５４００ ) and the opacity (e.g., σ ２_５４００ ~σ ２_５４００ The combination of (opacity specified by each) and is output sequentially.

[0194] Similarly to the training process 1500, in the training process 2000, for each viewpoint (e.g., viewpoints 1 and 2), a reconstruction model 110 (F θ The combinations of color and opacity of the 3D points sequentially output from the 3D point processor 110 are subjected to a volume rendering process 120.

[0195] In this embodiment, the volume rendering process 120 calculates the color of each pixel of an image seen from a certain viewpoint at a certain time using a volume rendering method. Specifically, the volume rendering process 120 calculates the color of each pixel at a certain time by using the restored model 110 (F θ ) is used to perform volume rendering using a predetermined product-sum operation based on the color and opacity output from the volume rendering process 120. As a result, the volume rendering process 120 generates a view image of a certain viewpoint at a certain time. The example in FIG. 20 shows how the volume rendering process 120 generates view images at each time information of viewpoint 1 (view image 1 to view image 5400 of viewpoint 1) and view images at each time information of viewpoint 2 (view image 1 to view image 5400 of viewpoint 2).

[0196] 20, a loss calculation process 130 is performed on the generated view images at each time information of viewpoint 1 (view image 1 to view image 5400 of viewpoint 1). Also, in the training process 2000 shown in FIG. 20, a loss calculation process 130 is performed on the generated view images at each time information of viewpoint 2 (view image 1 to view image 5400 of viewpoint 2).

[0197] Specifically, the view images at each time information of viewpoint 1 (view image 1 to view image 5400 at viewpoint 1) are captured images at each time information captured by the imaging device at viewpoint 1 (captured image A １ ~Photo A ５４００) and an error is calculated. Also, the view images at each time information of viewpoint 2 (view images 1 to 5400 at viewpoint 2) are compared with the captured images at each time information captured by the imaging device at viewpoint 2 (captured images B １ ~Photo image B ５４００ ) to calculate the error.

[0198] The error calculated in the loss calculation process 130 is used as the restoration model 110 (F θ ), the restoration model 110 (F θ ) is backpropagated. This results in the restored model 110 (F θ ) model parameters are updated. θ ) training process updates the model parameters, so that the trained reconstruction model (F θ ) is generated.

[0199] In order to simplify the explanation, we have omitted cases where training processing is performed using images captured by an imaging device at a viewpoint other than viewpoints 1 and 2, but images captured by an imaging device at a viewpoint other than viewpoints 1 and 2 may also be used in training processing.

[0200] <Outline of Image Generation Processing Using a Trained Reconstruction Model> Next, an outline of image generation processing using a trained reconstruction model applied to the server device 410 according to the third embodiment will be described. Fig. 21 is a third diagram for explaining the outline of image generation processing using a trained reconstruction model.

[0201] As shown in FIG. 21, the image generation process for generating a view image of the viewpoint ij at time information T is performed by using a three-dimensional point (x ｎ , y ｎ , z ｎ ) and viewpoint information (θ ｉ , φ ｊ ) and time information T are used to train the reconstruction model 210 (F θ), and calculates the color and opacity of each 3D point at the time information T as its output. Then, the image generation process performs a volume rendering process 120 based on the calculated color and opacity of each 3D point for each pixel of the view image to generate a view image of the viewpoint ij at the time information T. The image generation process uses a trained reconstruction model 210 (F θ ) sequentially inputting different time information T, view images (view image 1 to view image 5400 of viewpoint ij) at different time information (for example, T=1 to 5400) are sequentially generated.

[0202] <Relationship between Captured Images and Trained Reconstruction Model> Next, a trained reconstruction model applied to the server device 410 according to the third embodiment will be described. Fig. 22 is a third diagram showing an example of a trained reconstruction model applied to the server device. Note that, for the sake of simplicity, Fig. 22 also shows a case in which two viewpoints, viewpoint 1 and viewpoint 2, are used. However, as described above, in the training process, captured images captured by an imaging device at a viewpoint other than viewpoint 1 and viewpoint 2 may be used.

[0203] As shown in FIG. 22 , the server device 410 is applied with: a pre-trained restoration model that is pre-trained to be able to restore a scene from a first time to a second time using time-series captured images obtained by capturing the scene successively over time from multiple viewpoints.

[0204] Specifically, the server device 410 stores the following: time information T １ ~Time information T ５４００ A photographed image A is photographed by an imaging device at viewpoint 1. １ ~Photo A ５４００ and time information T １ ~Time information T ５４００ Image B is taken by the imaging device at viewpoint 2 １ ~Photo image B ５４００ and a trained reconstruction model F θ１_θ５４００ applies.

[0205] In FIG. 22, the time information T １ , T２ , T ３ , ... are the images A taken by the imaging device during the training process. １ , A ２ , ...or photographed image B １ , B ２ , ... (an example of a first time interval). That is, a trained restoration model (an example of a third restoration model) for generating time-series view images of the first time interval is applied to the server device 410.

[0206] <Example of Trained Recovery Model> Next, a trained recovery model stored in the model storage unit 606 in the server device 410 according to the third embodiment will be described. Fig. 23 is a diagram showing an example of a trained recovery model stored in the model storage unit of the server device according to the third embodiment.

[0207] As shown in FIG. 23, the trained reconstruction model stored in the model storage unit 606 is associated with time information. Specifically, the time information T １ ~T ５４００ is a trained reconstruction model F θ１_θ５４００ are associated.

[0208] The server device 410 uses the trained restoration model stored in the model storage unit 606 to generate time-series view images according to the viewpoint information and time information included in the request received from the client terminal 420 .

[0209] In FIG. 23, the time information T １ , T ２ , T ３ , ... correspond to the frame period of the images captured by the imaging device during the training process, as described above. １ , T ２ , T ３ , . . . correspond to the frame period when playing back free viewpoint video in the free viewpoint video playback system 400.

[0210] Furthermore, the trained restoration model shown in FIG. 23 can generate view images (free viewpoint images) from any viewpoint for a scene at each time point.

[0211] 23 , the model storage unit 606 stores at least one trained restoration model for generating view images for a series of scenes for one target. However, the number of trained restoration models stored in the model storage unit 606 is not limited to one, and the model storage unit 606 may store another trained restoration model for generating view images for a series of scenes for another target.

[0212] <Specific Example of Processing by Server Device> Next, a specific example of processing by the default moving image generating unit 602 and the requested moving image generating unit 604 of the server device 410 according to the third embodiment will be described.

[0213] (1) Specific example of processing by default video generation unit First, a description will be given of a specific example of processing by the default video generation unit 602. Fig. 24A is a first diagram showing a specific example of processing by the server device 410 according to the third embodiment. Fig. 24A shows a specific example of processing when the video designation acceptance unit 601 accepts designation of a free viewpoint video and the default video generation unit 602 receives notification of identification information of the designated free viewpoint video from the video designation acceptance unit 601.

[0214] As shown in FIG. 24A , the default video generator 602 generates a trained reconstruction model F θ１_θ５４００ is read from the model storage unit 606.

[0215] In addition, the default video generation unit 602 uses the read trained restoration model F θ１_θ５４００ For the default viewpoint information (θ ０ , φ ０ ) and each time information are input sequentially. This results in a trained reconstruction model F θ１_θ５４００ So, the default viewpoint information (θ ０ , φ ０ ) the view image X １ ~ View Image X ５４００ are generated sequentially.

[0216] In addition, the default video generating unit 602 generates the view image X １ ~ View Image X ５４００ time information T １ ~T ５４００ and notifies the video transmission unit 605. １ ~ View Image X ５４００ is transmitted in a transmission format that allows video playback on the client terminal 420.

[0217] (2) Specific example of processing by the requested video generation unit As described above, the default viewpoint information (θ ０ , φ ０ ) the view image X １ ~ View Image X ５４００ It is assumed that a free viewpoint video having frame images of is played back by the client terminal 420. In response to this, it is assumed that a request including time information and viewpoint information is transmitted from the client terminal 420. In this case, the request receiving unit 603 receives the request and notifies the requested video generating unit 604.

[0218] Here, a specific example of processing by the requested video generating unit 604 when a request (time information, viewpoint information) is notified from the request receiving unit 603 will be described. Fig. 24B is a second diagram showing a specific example of processing by the server device according to the third embodiment, and shows a specific example of processing by the requested video generating unit 604 when a request is notified from the request receiving unit 603.

[0219] As shown in FIG. 24B, the requested video generator 604 uses the trained reconstruction model F θ１_θ５４００ Identify.

[0220] In addition, the requested video generation unit 604 uses the identified trained reconstruction model F θ１_θ５４００ In response to the request, the time information (in the example of FIG. 24B, T ３ ) and viewpoint information (in the example of FIG. 24B, (θ ｘ , φ ｘ )) to obtain the trained reconstruction model F θ１_θ５４００ Then, the viewpoint information (θ ｘ , φ ｘ) based on the time information T ３ View image X in ３ Generate.

[0221] Next, the requested video generator 604 uses the identified trained reconstruction model F θ１_θ５４００ For the next time information T ４ and viewpoint information included in the request (in the example of FIG. 24B, (θ ｘ , φ ｘ )) and input the trained reconstruction model F θ１_θ５４００ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ４ View image X in ４ Generate.

[0222] Thereafter, the requested video generating unit 604 repeats the same process until an end condition is transmitted from the client terminal 420. In the example of FIG. 24B, the client terminal 420 transmits time information T １０ is shown to have been sent.

[0223] The client terminal 420 receives time information T １０ When the requested video generator 604 receives the trained reconstruction model F θ１_θ５４００ For the time information T as the termination condition １０ and viewpoint information included in the request (in the example of FIG. 24B, (θ ｘ , φ ｘ )) and . This will generate a trained reconstruction model F θ１_θ５４００ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T １０ View image X in １０ Generate.

[0224] In this way, the requested video generator 604 uses the trained restoration model to generate a time series of view images with a frame period according to viewpoint information from the time information included in the request to a predetermined end condition.

[0225] The requested video generation unit 604 generates the view image X ３ ~ View Image X １０ time information T ３ ~T １０ The video transmitting unit 605 then notifies the video transmitting unit 605 of the view image X ３ ~ View Image X １０ can be transmitted in a transmission format that allows video playback on the client terminal 420.

[0226] <Summary> As is clear from the above description, the one or more memories included in the server device 410 according to the third embodiment hold a trained reconstruction model (third reconstruction model) for generating time-series view images at a first time interval. The trained reconstruction model (third reconstruction model) is a single trained reconstruction model that is trained to receive input time information and output image information corresponding to the input time information.

[0227] Furthermore, the one or more processors included in the server device 410 according to the third embodiment: - Use a trained restoration model (third restoration model) to generate view images in a time series of a first time interval according to viewpoint information included in the request, which are view images from the time information included in the request to a predetermined termination condition.

[0228] As a result, according to the third embodiment, a mechanism for playing back free viewpoint video that is different from those of the first and second embodiments can be constructed.

[0229] [Fourth Embodiment] In the above-described first embodiment, the model storage unit 606 has been described as holding one trained restoration model for each piece of time information, and one trained restoration model generating a view image at one piece of time information. However, the trained restoration model held by the model storage unit 606 for each piece of time information is not limited to this. For example, the model storage unit 606 may hold a trained differential restoration model that generates a difference image from a view image generated by a trained restoration model of the previous piece of time information. Hereinafter, the fourth embodiment will be described, focusing on differences from the above-described first embodiment.

[0230] <Outline of Training Process for Restoration Model> First, an outline of the training process for the restoration model applied to the server device 410 according to the fourth embodiment will be described. Fig. 25 is a fourth diagram for explaining the outline of the training process for the restoration model. The training process 2500 shown in Fig. 25 differs from the training process 100 described with reference to Fig. 1 in the first embodiment in that the following restoration models are used in the training process 2500: a key restoration model 110 (F θ ), and a differential restoration model 2501 (ΔF θ１ ), and a differential restoration model 2502 (ΔF θ２ ) and .

[0231] As shown in FIG. 25, in the training process 2500, the key recovery model 110 (F θ ) includes: a 3D point in the 3D scene 140 (e.g., (x １ , y １ , z １ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray1) from viewpoint 1 to the three-dimensional point (e.g., viewpoint information (θ １ , φ １ )) and are input. This results in the key recovery model 110 (F θ ) is the color (for example, (R １１ , G １１ , B １１ )), and the opacity of the three-dimensional point at time T=1 (for example, σ １１ That is, the key recovery model 110 calculates the color and opacity of a certain 3D point at a certain viewpoint.

[0232] In the training process 2500, the key recovery model 110 (F θ 25 shows how the same processing is performed for two viewpoints (viewpoint 1, viewpoint 2).

[0233] Specifically, in the training process 2500, the key recovery model 110 (F θ) also includes: a 3D point in the 3D scene 140 (e.g., (x ２ , y ２ , z ２ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray2) from viewpoint 2 to the three-dimensional point (e.g., viewpoint information (θ ２ , φ ２ )) and are input. This results in the key recovery model 110 (F θ ) is the color (for example, (R ２１ , G ２１ , B ２１ )), and the opacity of the three-dimensional point at time T=1 (for example, σ ２１ That is, the key recovery model 110 calculates the color and opacity of a certain 3D point at a certain viewpoint.

[0234] Also, as shown in FIG. 25, in the training process 2500, a differential restoration model 2501 (ΔF θ１ ) includes: a 3D point in the 3D scene 140 (e.g., (x １ , y １ , z １ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray1) from viewpoint 1 to the three-dimensional point (e.g., viewpoint information (θ １ , φ １ )) and are input. This results in a differential restoration model 2501 (ΔF θ１ ) is the key recovery model 110 (F θ ) is a difference between the color and opacity of a 3D point generated one frame period before the color and opacity of a 3D point generated one frame period after the color and opacity of a 3D point generated one frame period before ... １２ , ΔG １２ , ΔB １２ )), and the differential opacity (for example, Δσ １２) and the differential restoration model 2501 (ΔF θ１ ) calculates the differential color and opacity of a 3D point at a viewpoint.

[0235] In the training process 2500, the differential restoration model 2501 (ΔF θ１ 25 shows how the same processing is performed for two viewpoints (viewpoint 1, viewpoint 2).

[0236] Specifically, in the training process 2500, a differential restoration model 2501 (ΔF θ１ ) also includes: a 3D point in the 3D scene 140 (e.g., (x ２ , y ２ , z ２ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray2) from viewpoint 2 to the three-dimensional point (e.g., viewpoint information (θ ２ , φ ２ )) and are input. This results in a differential restoration model 2501 (ΔF θ１ ) is the key recovery model 110 (F θ ) is a difference between the color and opacity of a 3D point generated one frame period before the color and opacity of a 3D point generated one frame period after the color and opacity of a 3D point generated one frame period before ... ２２ , ΔG ２２ , ΔB ２２ )), and the differential opacity (for example, Δσ ２２ The output is a combination of (differential opacity specified by) and

[0237] Also, as shown in FIG. 25, in the training process 2500, a differential restoration model 2502 (ΔF θ２ ) includes: a 3D point in the 3D scene 140 (e.g., (x １ , y １ , z １)) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray1) from viewpoint 1 to the three-dimensional point (e.g., viewpoint information (θ １ , φ １ )) and are input. This results in a differential restoration model 2502 (ΔF θ２ ) is the key recovery model 110 (F θ ) is a difference between the color and opacity of a 3D point generated one frame period before the color and opacity of a 3D point generated two frame periods after the color and opacity of a 3D point generated two frame periods before ... １３ , ΔG １３ , ΔB １３ )), and the differential opacity (for example, Δσ １３ That is, the differential restoration model 2502 (ΔF θ２ ) calculates the differential color and opacity of a 3D point at a viewpoint.

[0238] In the training process 2500, the differential restoration model 2502 (ΔF θ２ 25 shows how the same processing is performed for two viewpoints (viewpoint 1, viewpoint 2).

[0239] Specifically, in the training process 2500, a differential restoration model 2502 (ΔF θ２ ) also includes: a 3D point in the 3D scene 140 (e.g., (x ２ , y ２ , z ２ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray2) from viewpoint 2 to the three-dimensional point (e.g., viewpoint information (θ ２ , φ ２ )) and are input. This results in a differential restoration model 2502 (ΔF θ２ ) is the key recovery model 110 (F θ) is a difference between the color and opacity of a 3D point generated one frame period before the color and opacity of a 3D point generated two frame periods after the color and opacity of a 3D point generated two frame periods before ... ２３ , ΔG ２３ , ΔB ２３ )), and the differential opacity (for example, Δσ ２３ The output is a combination of (differential opacity specified by) and

[0240] Similarly to the training process 100, in the training process 2500 shown in FIG. 25, for each viewpoint (e.g., viewpoints 1 and 2), a key recovery model 110 (F θ The color and opacity combinations of the 3D points output from the 3D point processor 110 are subjected to a volume rendering process 120.

[0241] In this embodiment, the volume rendering process 120 calculates the color of each pixel of an image seen from a certain viewpoint using a volume rendering method. Specifically, the volume rendering process 120 calculates the color of each pixel by using the key restoration model 110 (F θ 25 shows how view image 1 from viewpoint 1 and view image 1 from viewpoint 2 are generated by the volume rendering process 120.

[0242] In the training process 2500 shown in FIG. 25, for each viewpoint (e.g., viewpoints 1 and 2), a differential restoration model 2501 (ΔF θ１ ), differential restoration model 2502 (ΔF θ２ ) is subjected to a volume rendering process 120 on the combination of differential color and differential opacity of the 3D points output from the 3D point rendering process 120.

[0243] In this embodiment, the volume rendering process 120 calculates the differential color of each pixel, which indicates the difference between an image seen from a certain viewpoint and an image seen from the same viewpoint at the previous time, using a volume rendering method. The differential color of each pixel, which indicates the difference, is calculated by using a differential restoration model 2501 (ΔF θ１ ), differential restoration model 2502 (ΔF θ２ 25 , the volume rendering process 120 generates a differential view image at a certain viewpoint from the previous time. The example in FIG. 25 shows how the volume rendering process 120 generates: differential view images at each time information of viewpoint 1 (differential view image 1 and differential view image 2 at viewpoint 1); and differential view images at each time information of viewpoint 2 (differential view image 1 and differential view image 2 at viewpoint 2).

[0244] 25, a difference image generation process 2510 generates a difference image to be used in the loss calculation process 130. Specifically, the difference image generation process 2510 generates a difference image to be used in the loss calculation process 130. １ The captured image A corresponding to time information T=2 is acquired. ２ and capture image A １ By calculating the difference between １ -A ２ ) is generated. ３ and capture image A ２ By calculating the difference between ２ -A ３ ) is generated. １ The captured image B corresponding to time information T=2 is acquired. ２ and capture image B １ By calculating the difference between １ -B ２ ) is generated. ３and capture image B ２ By calculating the difference between ２ -B ３ ) to generate the

[0245] In the training process 2500 shown in Fig. 25, the view images of each generated viewpoint (view image 1 of viewpoint 1 and view image 1 of viewpoint 2) are subjected to a loss calculation process 130. Specifically, the view image 1 of viewpoint 1 is calculated by multiplying the captured image A captured by the imaging device of viewpoint 1 by the loss calculation process 130. １ The view image 1 at the viewpoint 2 is compared with the captured image B captured by the imaging device at the viewpoint 2, and the error is calculated. １ is compared with the above to calculate the error.

[0246] The error calculated in the loss calculation process 130 is calculated using the key recovery model 110 (F θ ), the key recovery model 110 (F θ ) is backpropagated. This results in the key recovery model 110 (F θ ) model parameters are updated. θ ) training process updates the model parameters, so that the trained key recovery model F θ is generated.

[0247] Similarly, in the training process 2500 shown in FIG. 25 , the loss calculation process 130 is performed on the generated differential view images of each viewpoint (differential view image 1 of viewpoint 1 and differential view image 1 of viewpoint 2). Specifically, the differential view image 1 of viewpoint 1 is calculated based on the differential image (A １ -A ２ ) to calculate the error. The difference view image 1 of the viewpoint 2 is compared with the difference image (B １ -B ２ ) to calculate the error.

[0248] The error calculated in the loss calculation process 130 is used as the differential restoration model 2501 (ΔF θ１), the differential restoration model 2501 (ΔF θ１ ) is backpropagated. This results in a differential restoration model 2501 (ΔF θ１ ) model parameters are updated. θ１ ) training process, the model parameters are updated, and the trained differential restoration model ΔF θ１ is generated.

[0249] Similarly, in the training process 2500 shown in FIG. 25 , the loss calculation process 130 is performed on the generated differential view images of each viewpoint (differential view image 2 of viewpoint 1 and differential view image 2 of viewpoint 2). Specifically, the differential view image 2 of viewpoint 1 is calculated based on the differential image (A ２ -A ３ ) to calculate the error. The difference view image 2 of the viewpoint 2 is compared with the difference image (B ２ -B ３ ) to calculate the error.

[0250] The error calculated in the loss calculation process 130 is used as the differential restoration model 2502 (ΔF θ２ ), the differential restoration model 2502 (ΔF θ２ ) is backpropagated. This results in a differential restoration model 2502 (ΔF θ２ ) model parameters are updated. θ２ ) training process, the model parameters are updated, and the trained differential restoration model ΔF θ２ is generated.

[0251] In order to simplify the explanation, we have omitted cases where training processing is performed using images captured by an imaging device at a viewpoint other than viewpoints 1 and 2, but images captured by an imaging device at a viewpoint other than viewpoints 1 and 2 may also be used in training processing.

[0252] <Outline of Image Generation Processing Using a Trained Reconstruction Model> Next, an outline of image generation processing using a trained reconstruction model applied to the server device 410 according to the fourth embodiment will be described. Fig. 26 is a fourth diagram for explaining the outline of image generation processing using a trained reconstruction model.

[0253] As shown in FIG. 26, the image generation process for generating a view image of the viewpoint ij at time information T is performed by using a three-dimensional point (x ｎ , y ｎ , z ｎ ) and viewpoint information (θ ｉ , φ ｊ ) and the trained key recovery model 210 (F θ ), and calculates the color and opacity of each 3D point at the time information T as its output. Then, the image generation process performs volume rendering process 120 based on the calculated color and opacity of each 3D point for each pixel of the view image, thereby generating a view image of the viewpoint ij at the time information T.

[0254] As shown in FIG. 26, the image generation process for generating a view image of the viewpoint ij at time information one step after the time information T is performed using a three-dimensional point (x ｎ , y ｎ , z ｎ ) and viewpoint information (θ ｉ , φ ｊ ) and the trained differential restoration model 2601 (ΔF θ１ ), and calculates, as its output, the differential color and differential opacity of each 3D point with respect to time information T at time information one time after time information T. Then, the image generation process performs volume rendering process 120 based on the calculated differential color and differential opacity of each 3D point for each pixel of the view image, thereby generating a differential view image 1 of viewpoint ij with time information T at time information one time after time information T. Furthermore, the image generation process performs addition process 2611 that adds differential view image 1 of viewpoint ij to view image 1 of viewpoint ij at time information T, thereby generating view image 2 of viewpoint ij.

[0255] As shown in FIG. 26, the image generation process for generating a view image of the viewpoint ij at time information two times after the time information T is performed by using the three-dimensional point (x ｎ , y ｎ , z ｎ ) and viewpoint information (θ ｉ , φ ｊ ) and the trained differential restoration model 2602 (ΔF θ２ ), and calculates, as its output, the differential color and differential opacity of each 3D point at time information two times later than time information T and at time information one time later than time information T. Then, the image generation process performs volume rendering process 120 based on the calculated differential color and differential opacity of each 3D point for each pixel of the view image, thereby generating a differential view image 2 of viewpoint ij at time information two times later than time information T and at time information one time later than time information T. Furthermore, the image generation process performs addition process 2612 that adds differential view image 2 of viewpoint ij to view image 2 of viewpoint ij one time later than time information T.

[0256] <Relationship between Captured Images and Trained Reconstruction Model> Next, a trained reconstruction model applied to the server device 410 according to the fourth embodiment will be described. Fig. 27 is a fourth diagram showing an example of a trained reconstruction model applied to the server device. Note that, for the sake of simplicity, Fig. 27 also shows a case in which two viewpoints, viewpoint 1 and viewpoint 2, are used. However, as described above, in the training process, captured images captured by an imaging device at a viewpoint other than viewpoint 1 and viewpoint 2 may be used.

[0257] As shown in Figure 27, a pre-trained restoration model is applied to the server device 410 so that it can restore a scene from a first time to a second time using a time series of captured images obtained by capturing the scene continuously over time from multiple viewpoints.

[0258] Specifically, the server device 410 stores the following: time information T １ A photographed image A is photographed by an imaging device at viewpoint 1. １ and time information T １Image B is taken by the imaging device at viewpoint 2 １ and a trained key recovery model F that has been trained using θ１ applies.

[0259] As shown in FIG. 27, the server device 410 stores the following information: １ A photographed image A is photographed by an imaging device at viewpoint 1. １ and time information T ２ A photographed image A is photographed by an imaging device at viewpoint 1. ２ The difference image (A １ -A ２ ) and time information T １ Image B is taken by the imaging device at viewpoint 2 １ and time information T ２ Image B is taken by the imaging device at viewpoint 2 ２ The difference image (B １ -B ２ ) and a trained differential restoration model ΔF θ１ applies.

[0260] As shown in FIG. 27, the server device 410 stores the following information: ２ A photographed image A is photographed by an imaging device at viewpoint 1. ２ and time information T ３ A photographed image A is photographed by an imaging device at viewpoint 1. ３ The difference image (A ２ -A ３ ) and time information T ２ Image B is taken by the imaging device at viewpoint 2 ２ and time information T ３ Image B is taken by the imaging device at viewpoint 2 ３ The difference image (B ２ -B ３ ) and a trained differential restoration model ΔF θ２ applies.

[0261] Similarly, the server device 410 stores the following: time information T ４ A photographed image A is photographed by an imaging device at viewpoint 1.４ and time information T ４ Image B is taken by the imaging device at viewpoint 2 ４ and a trained key recovery model F that has been trained using θ４ applies.

[0262] As shown in FIG. 27, the server device 410 stores the following information: ４ A photographed image A is photographed by an imaging device at viewpoint 1. ４ and time information T ５ A photographed image A is photographed by an imaging device at viewpoint 1. ５ The difference image (A ４ -A ５ ) and time information T ４ Image B is taken by the imaging device at viewpoint 2 ４ and time information T ５ Image B is taken by the imaging device at viewpoint 2 ５ The difference image (B ４ -B ５ ) and a trained differential restoration model ΔF θ１ applies.

[0263] As shown in FIG. 27, the server device 410 stores the following information: ５ A photographed image A is photographed by an imaging device at viewpoint 1. ５ and time information T ６ A photographed image A is photographed by an imaging device at viewpoint 1. ６ The difference image (A ５ -A ６ ) and time information T ５ Image B is taken by the imaging device at viewpoint 2 ５ and time information T ６ Image B is taken by the imaging device at viewpoint 2 ６ The difference image (B ５ -B ６ ) and a trained differential restoration model ΔF θ２ applies.

[0264] In the example of FIG. 27, due to space limitations, the time information T １１The trained differential restoration model ΔF θ１ 27, the number of trained key recovery models and the number of trained differential recovery models applied to the free viewpoint video playback system 400 are not limited to the example in Fig. 27. However, it is assumed that all trained key recovery models and trained differential recovery models are associated with each piece of time information and managed as time-series trained recovery models.

[0265] In FIG. 27, the time information T １ , T ４ , T ７ , ... are the images A taken by the imaging device during the training process. １ , A ２ , ...or photographed image B １ , B ２ , ... (an example of a first time interval). That is, the server device 410 is applied with: a trained key recovery model for generating time series view images of a third time interval longer than the first time interval, the trained key recovery model for a time series of the third time interval (an example of a fourth recovery model), and a trained differential recovery model for generating time series view images of the first time interval, the trained differential recovery model for a time series of the first time interval (an example of a fifth recovery model), which generates a difference image representing the difference between a view image generated the first time interval before.

[0266] <Example of Trained Recovery Model> Next, a trained recovery model stored in the model storage unit 606 in the server device 410 according to the fourth embodiment will be described. Fig. 28 is a diagram showing an example of a trained recovery model of the server device according to the fourth embodiment.

[0267] As shown in Fig. 28, the trained key recovery model and the trained differential recovery model stored in the model storage unit 606 are associated with time information. １ is a trained key recovery model F θ１ is associated with the time information T ２ ~T ３ is a trained differential restoration model ΔF θ１, ΔF θ２ Similarly, in the example of FIG. 28, the time information T ４ ~T ６ is a trained key recovery model F θ４ , the trained differential restoration model ΔF θ１ , ΔF θ２ In addition, the time information T ７ ~T ９ is a trained key recovery model F θ７ , the trained differential restoration model ΔF θ１ , ΔF θ２ is associated with the time information T １０ ~T １１ is a trained key recovery model F θ１０ , the trained differential restoration model ΔF θ１ The association between the time information and the trained key recovery model (or the trained differential recovery model) may be achieved by directly associating the time information with the trained key recovery model (or the trained differential recovery model) or by indirectly associating the time information with the trained key recovery model (or the trained differential recovery model) via other data.

[0268] The server device 410 uses the trained key recovery model and trained differential recovery model stored in the model storage unit 606 to generate time-series view images according to the viewpoint information and time information included in the request received from the client terminal 420.

[0269] In FIG. 28, the time information T １ , T ２ , T ３ , ... correspond to the frame period of the images captured by the imaging device during the training process, as described above. １ , T ２ , T ３ , . . . correspond to the frame period when playing back free viewpoint video in the free viewpoint video playback system 400.

[0270] 28, the trained key recovery models or trained differential recovery models associated with each piece of time information are different from each other. The different trained key recovery models or trained differential recovery models here are configured by neural networks to which NeRF technology is applied, and are trained using different training data (captured images). The neural network architectures may be the same or may be partially different.

[0271] In addition, by using each trained key recovery model or each trained differential recovery model shown in Figure 28, the server device 410 generates a view image (free viewpoint image) from an arbitrary viewpoint for the scene at each time information.

[0272] 28 , the model storage unit 606 stores at least a group of trained key recovery models and trained differential recovery models for generating view images for a series of scenes for one target. However, the group of trained key recovery models and trained differential recovery models stored in the model storage unit 606 is not limited to one, and the model storage unit 606 may store another group of trained key recovery models and trained differential recovery models for generating view images for a series of scenes for another target.

[0273] 28, due to space limitations, the group of trained key recovery models and trained differential recovery models stored in the model storage unit 606 are stored in a format similar to that shown in FIG. １ ~T １１ However, the number of trained key recovery models and the number of trained differential recovery models stored in the model storage unit 606 are not limited to this.

[0274] <Specific Example of Processing by Server Device> Next, a specific example of processing by the default moving image generating unit 602 and the requested moving image generating unit 604 of the server device 410 according to the fourth embodiment will be described.

[0275] (1) Specific example of processing by default video generation unit First, a description will be given of a specific example of processing by the default video generation unit 602. Fig. 29A is a first diagram showing a specific example of processing by the server device 410 according to the fourth embodiment. Fig. 29A shows a specific example of processing when the video designation acceptance unit 601 accepts designation of a free viewpoint video and the default video generation unit 602 receives notification of identification information of the designated free viewpoint video from the video designation acceptance unit 601.

[0276] As shown in FIG. 29A , the default video generation unit 602 selects the following trained restoration models for generating view images included in the specified free viewpoint video: θ１ , F θ４ , F θ７ , F θ１０ , and time information T ２ , T ３ , T ５ , T ６ , T ８ , T ９ , T １１ The trained differential restoration model ΔF θ１ , ΔF θ２ , are read from the model storage unit 606.

[0277] The default video generator 602 also reads out the trained key recovery model F θ１ , F θ４ , F θ７ , F θ１０ , and time information T ２ , T ３ , T ５ , T ６ , T ８ , T ９ , T １１ The trained differential restoration model ΔF θ１ , ΔF θ２ , for each, the default viewpoint information (θ ０ , φ ０ )

[0278] This allows the trained key recovery model F θ１ , F θ４ , F θ７ , Fθ１０ So, the default viewpoint information (θ ０ , φ ０ ) the view image X １ , X ４ , X ７ , X １０ Generate.

[0279] In addition, the time information T ２ , T ３ , T ５ , T ６ , T ８ , T ９ , T １１ The trained differential restoration model ΔF θ１ , ΔF θ２ Then, the difference image ΔX １ , ΔX ２ , ΔX ４ , ΔX ５ , ΔX ７ , ΔX ８ , ΔX １０ Generate.

[0280] Also, view image X １ For the difference image ΔX １ By adding the default viewpoint information (θ ０ , φ ０ ) based on the time information T ２ View image X in ２ is generated.

[0281] Also, view image X ２ For the difference image ΔX ２ By adding the default viewpoint information (θ ０ , φ ０ ) based on the time information T ３ View image X in ３ is generated.

[0282] Also, view image X ４ For the difference image ΔX ４ By adding the default viewpoint information (θ ０ , φ ０ ) based on the time information T ５View image X in ５ is generated.

[0283] Also, view image X ５ For the difference image ΔX ５ By adding the default viewpoint information (θ ０ , φ ０ ) based on the time information T ６ View image X in ６ is generated.

[0284] Also, view image X ７ For the difference image ΔX ７ By adding the default viewpoint information (θ ０ , φ ０ ) based on the time information T ８ View image X in ８ is generated.

[0285] Also, view image X ８ For the difference image ΔX ８ By adding the default viewpoint information (θ ０ , φ ０ ) based on the time information T ９ View image X in ９ is generated.

[0286] Also, view image X １０ For the difference image ΔX １０ By adding the default viewpoint information (θ ０ , φ ０ ) based on the time information T １１ View image X in １１ is generated.

[0287] In addition, the default video generating unit 602 generates the view image X １ ~ View Image X １１ time information T １ ~T １１ and notifies the video transmission unit 605. １ ~ View Image X １１is transmitted in a transmission format that allows video playback on the client terminal 420.

[0288] In the above description, the default video generating unit 602 generates the differential image ΔX １ , ΔX ２ , ΔX ４ , ΔX ５ , ΔX ７ , ΔX ８ , ΔX １０ , and the view image X ２ , X ３ , X ５ , X ６ , X ８ , X ９ , X １１ In the above description, the default video generator 602 generates the generated view image X ２ , X ３ , X ５ , X ６ , X ８ , X ９ , X １１ is notified to the video transmitting unit 605.

[0289] However, the processing performed by the default video generator 602 is not limited to this. For example, the default video generator 602 may: １ , X ４ , X ７ , X １０ and the differential image ΔX generated by the trained differential reconstruction model. １ , ΔX ２ , ΔX ４ , ΔX ５ , ΔX ７ , ΔX ８ , ΔX １０ The video transmitting unit 605 may be notified of the above.

[0290] In this case, the client terminal 420 receives the view image X from the server device 410. １ , X ４ , X ７ , X １０ The client terminal 420 also receives the differential image ΔX １ , ΔX ２ , ΔX４ , ΔX ５ , ΔX ７ , ΔX ８ , ΔX １０ Then, the client terminal 420 receives the received view image X １ , X ４ , X ７ , X １０ and the received differential image ΔX １ , ΔX ２ , ΔX ４ , ΔX ５ , ΔX ７ , ΔX ８ , ΔX １０ and the view image X ２ , X ３ , X ５ , X ６ , X ８ , X ９ , X １１ Generate.

[0291] In this way, part of the processing performed by the default video generating unit 602 may be executed by the client terminal 420 .

[0292] (2) Specific example of processing by the requested video generation unit As described above, the default viewpoint information (θ ０ , φ ０ ) the view image X １ ~ View Image X １１ It is assumed that a free viewpoint video having frame images of is played back by the client terminal 420. In response to this, it is assumed that a request including time information and viewpoint information is transmitted from the client terminal 420. In this case, the request receiving unit 603 receives the request and notifies the requested video generating unit 604.

[0293] Here, a specific example of processing by the requested video generating unit 604 when a request (time information, viewpoint information) is notified from the request receiving unit 603 will be described. Fig. 29B is a second diagram showing a specific example of processing by the server device according to the fourth embodiment, and shows a specific example of processing by the requested video generating unit 604 when a request is notified from the request receiving unit 603.

[0294] As shown in FIG. 29B , the requested video generator 604 selects the trained reconstruction model from the trained reconstruction models that have already been read, and then selects the trained reconstruction model from the trained reconstruction model. ３ ) the trained differential restoration model ΔF θ２ In addition, the requested video generation unit 604 identifies the trained differential restoration model ΔF θ２ Based on the view image X ３ The trained key recovery model F required to generate θ１ , the trained differential restoration model ΔF θ１ Identify.

[0295] In addition, the requested video generator 604 uses the identified trained key recovery model F θ１ , the trained differential restoration model ΔF θ１ , ΔF θ２ In contrast, the viewpoint information included in the request (in the example of FIG. 29B, (θ ｘ , φ ｘ )) to generate a trained key recovery model F θ１ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T １ View image X in １ Also, a trained differential restoration model ΔF θ１ , ΔF θ２ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ２ , T ３ The difference image ΔX in １ , ΔX ２ Furthermore, the requested video generation unit 604 generates the generated view image X １ and the difference image ΔX １ , ΔX ２ Using this, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ３ View image X in ３ Generate.

[0296] Next, the requested video generation unit 604 selects the trained key recovery model F corresponding to the next time information (next time) as the next trained recovery model. θ４ In addition, the requested video generation unit 604 identifies the identified trained key recovery model F θ４ The viewpoint information included in the request (θ ｘ , φ ｘ ) to generate a trained key recovery model F θ４ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ４ View image X in ４ Generate.

[0297] Next, the requested video generation unit 604 selects a trained differential restoration model ΔF corresponding to the next time information (next time) as the next trained restoration model. θ１ In addition, the requested video generation unit 604 identifies the identified trained differential restoration model ΔF θ１ The viewpoint information included in the request (θ ｘ , φ ｘ ) to obtain the trained differential restoration model ΔF θ１ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ５ The difference image ΔX in ４ Furthermore, the requested video generation unit 604 generates the generated view image X ４ and the difference image ΔX ４ Using this, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ５ View image X in ５ Generate.

[0298] Next, the requested video generation unit 604 selects a trained differential restoration model ΔF corresponding to the next time information (next time) as the next trained restoration model. θ２ In addition, the requested video generation unit 604 identifies the identified trained differential restoration model ΔF θ２ The viewpoint information included in the request (θｘ , φ ｘ ) to obtain the trained differential restoration model ΔF θ２ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ６ The difference image ΔX in ５ Furthermore, the requested video generation unit 604 generates the generated view image X ５ and the difference image ΔX ５ Using this, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ６ View image X in ６ Generate.

[0299] Thereafter, the requested video generating unit 604 repeats the same process until an end condition is transmitted from the client terminal 420. In the example of FIG. 29B, the client terminal 420 transmits time information T １０ is shown to have been sent.

[0300] The client terminal 420 receives time information T １０ When the time information T １０ A trained key recovery model F corresponding to θ１０ In addition, the requested video generation unit 604 identifies the identified trained reconstruction model F θ１０ In contrast, the viewpoint information included in the request (in the example of FIG. 29B, (θ ｘ , φ ｘ )) to generate a trained key recovery model F θ１０ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T １０ View image X in １０ Generate.

[0301] In this way, the requested video generation unit 604: generates time-series view images of a third time interval according to viewpoint information using a time-series trained key recovery model of a third time interval from a trained key recovery model corresponding to the time information included in the request to a trained key recovery model corresponding to a predetermined termination condition; generates time-series difference images of a first time interval according to viewpoint information using a time-series trained differential recovery model of a first time interval from a trained differential recovery model corresponding to the time information included in the request to a trained differential recovery model corresponding to the predetermined termination condition. The difference images correspond to time information excluding time information at which view images are generated using the time-series trained key recovery model; and generates time-series view images of the first time interval, excluding view images generated using the trained key recovery model, by adding the difference images to view images of the previous first time interval.

[0302] The requested video generation unit 604 generates the view image X ３ ~ View Image X １０ time information T ３ ~T １０ The video transmitting unit 605 then notifies the video transmitting unit 605 of the view image X ３ ~ View Image X １０ is transmitted in a transmission format that allows video playback on the client terminal 420.

[0303] <Summary> As is clear from the above description, the one or more memories included in the server device 410 according to the fourth embodiment: hold a trained key recovery model (fourth recovery model) of a time series of a third time interval for generating time series view images of a third time interval that is longer than the first time interval; hold a trained differential recovery model (fourth differential recovery model) used when generating view images of the time series of the first time interval, excluding view images generated using the trained key recovery model (fourth recovery model). The trained differential recovery model (fourth differential recovery model) is a trained differential recovery model of a time series of a first time interval for generating difference images representing the difference from view images generated the first time interval earlier.

[0304] Furthermore, one or more processors included in the server device 410 according to the fourth embodiment: generate time-series view images of a third time interval corresponding to viewpoint information using a time-series trained key recovery model (fourth recovery model) of a third time interval from a trained key recovery model (fourth recovery model) corresponding to the time information included in the request to a trained key recovery model (fourth recovery model) corresponding to a predetermined termination condition; generate time-series difference images of a first time interval corresponding to viewpoint information using a time-series trained differential recovery model (fourth differential recovery model) of a first time interval from a trained differential recovery model (fourth differential recovery model) corresponding to the time information included in the request to a trained differential recovery model (fourth differential recovery model) corresponding to the predetermined termination condition. The difference images are time-series difference images corresponding to time information excluding the time information at which the view images were generated using the time-series trained key recovery model (fourth recovery model). - By adding the difference image to the view image from the first time interval earlier, a time series of view images for the first time interval is generated, excluding view images generated using the trained key recovery model (fourth recovery model).

[0305] As a result, according to the fourth embodiment, a mechanism for playing back free viewpoint video that is different from the first to third embodiments can be constructed.

[0306] Fifth Embodiment In the first embodiment, a case has been described in which one imaging device captures the three-dimensional scene 140 from the same viewpoint. However, for example, two imaging devices may capture the three-dimensional scene 140 from the same viewpoint. This makes it possible to divide the three-dimensional scene 140 into two spaces and generate trained restoration models that generate view images in each space. The fifth embodiment will be described below, focusing on the differences from the first embodiment.

[0307] <Outline of Training Process for Restoration Model> First, an outline of the training process for the restoration model applied to the server device 410 according to the fifth embodiment will be described. FIG. 30 is a fifth diagram for explaining the outline of the training process for the restoration model. In the case of the training process 3000 shown in FIG. 30, the following restoration models are used: Space 1 restoration model 110_1 (F θ ) and, Space 2 restoration model 110_2 (F θ ), and the space 1 restoration model 110_1 (F θ ) includes: a 3D point (e.g., (x １_１ , y １_１ , z １_１ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray1) from viewpoint 1 to the three-dimensional point (e.g., viewpoint information (θ １ , φ １ )) and are input. As a result, the space 1 restoration model 110_1 (F θ ) is the color (for example, (R １_１ , G １_１ , B １_１ )) and the opacity (e.g., σ) of the 3D point in the upper half space (Space 1) in the 3D scene 140. １_１ ) and the space 1 restoration model 110_1 (F θ ) calculates the color and opacity of a 3D point in space 1 at a certain viewpoint.

[0308] As in the training process 100, in the training process 3000, the space 1 reconstruction model 110_1 (F θ 30 shows how the same processing is performed for two viewpoints (viewpoint 1, viewpoint 2).

[0309] Specifically, in the training process 3000, the space 1 reconstruction model 110_1 (F θ ) also includes: a 3D point (e.g., (x ２_１ , y ２_１ , z ２_１ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray2) from viewpoint 2 to the three-dimensional point (e.g., viewpoint information (θ ２ , φ ２ )) and are input. As a result, the space 1 restoration model 110_2 (F θ ) is the color (for example, (R ２_１ , G ２_１ , B ２_１ )) and the opacity (e.g., σ) of the 3D point in the upper half space (Space 1) in the 3D scene 140. ２_１ The output is a combination of (opacity specified by) and

[0310] On the other hand, the space 2 reconstruction model 110_1 (F θ ) includes: a 3D point (e.g., (x １_２ , y １_２ , z １_２ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray1) from viewpoint 1 to the three-dimensional point (e.g., viewpoint information (θ １ , φ １ )) and are input. As a result, the space 2 restoration model 110_1 (F θ) is the color (for example, (R １_２ , G １_２ , B １_２ )) and the opacity (e.g., σ) of the 3D point in the lower half space (Space 2) in the 3D scene 140. １_２ That is, the spatial 2 restoration model 110_1 (F θ ) calculates the color and opacity of a 3D point in space 2 at a certain viewpoint.

[0311] As in the training process 100, in the training process 3000, the space 2 reconstruction model 110_2 (F θ 30 shows how the same processing is performed for two viewpoints (viewpoint 1, viewpoint 2).

[0312] Specifically, in the training process 3000, the space 2 reconstruction model 110_2 (F θ ) further includes: a 3D point in the lower half space (space 2) in the 3D scene 140 (e.g., (x ２_２ , y ２_２ , z ２_２ )) and viewpoint information specifying a direction vector representing a line of sight (e.g., Ray2) from viewpoint 2 to the three-dimensional point (e.g., viewpoint information (θ ２ , φ ２ )) and are input. As a result, the space 2 restoration model 110_1 (F θ ) is the color (for example, (R ２_２ , G ２_２ , B ２_２ )) and the opacity (e.g., σ) of the 3D point in the lower half space (Space 2) in the 3D scene 140. ２_２ The output is a combination of (opacity specified by) and

[0313] Similarly to the training process 100, in the training process 3000, for each viewpoint (e.g., viewpoints 1 and 2), a space 1 reconstruction model 110_1 (F θ The color and opacity combinations of the 3D points output from the 3D point processor 110 are subjected to a volume rendering process 120.

[0314] In this embodiment, the volume rendering process 120 calculates the color of each pixel of an image seen from a certain viewpoint using a volume rendering method. Specifically, the volume rendering process 120 calculates the color of each pixel in the space 1 by using the space 1 restoration model 110_1 (F θ 30 shows how a view image (space 1) from viewpoint 1 and a view image (space 1) from viewpoint 2 are generated by the volume rendering process 120.

[0315] Similarly, the volume rendering process 120 calculates the color of each pixel in the space 2 by using the space 2 restoration model 110_2 (F θ 30 shows how a view image (space 2) from viewpoint 1 and a view image (space 2) from viewpoint 2 are generated by the volume rendering process 120.

[0316] In the training process 3000 shown in Fig. 30, the generated view image (space 1) from viewpoint 1 and the view image (space 1) from viewpoint 2 are subjected to a loss calculation process 130. For example, the view image (space 1) from viewpoint 1 is generated by using the captured image A captured by the imaging device at viewpoint 1. １＿１The view image (space 1) at viewpoint 2 is compared with the captured image B captured by the imaging device at viewpoint 2, and the error is calculated. １＿１ is compared with the above to calculate the error.

[0317] Similarly, a loss calculation process 130 is performed on the generated view image (space 2) of viewpoint 1 and the view image (space 2) of viewpoint 2. For example, the view image (space 2) of viewpoint 1 is calculated by multiplying the captured image A captured by the imaging device at viewpoint 1. １＿２ The view image (space 2) at viewpoint 2 is compared with the captured image B １＿２ is compared with the above to calculate the error.

[0318] The errors calculated in the loss calculation process 130 are respectively the space 1 restoration model 110_1 (F θ ) and the space 2 reconstruction model 110_2 (F θ ), the space 1 restoration model 110_1 (F θ ) and the space 2 reconstruction model 110_2 (F θ ) is backpropagated. As a result, the space 1 reconstruction model 110_1 (F θ ) model parameters and the space 2 restoration model 110_2 (F θ ) model parameters are updated. θ ) training process, the model parameters are updated, and according to the training process shown in FIG. 30, the trained space 1 reconstruction model (F θ ) is generated. Also, the space 2 restoration model 110_2 (F θ ) training process, the model parameters are updated, and according to the training process shown in FIG. 30, the trained space 2 reconstruction model (F θ ) is generated.

[0319] In order to simplify the explanation, we have omitted cases where training processing is performed using images captured by an imaging device at a viewpoint other than viewpoints 1 and 2, but images captured by an imaging device at a viewpoint other than viewpoints 1 and 2 may also be used in training processing.

[0320] <Outline of Image Generation Processing Using a Trained Reconstruction Model> Next, an outline of image generation processing using a trained reconstruction model applied to the server device 410 according to the fifth embodiment will be described. Fig. 31 is a fifth diagram for explaining the outline of image generation processing using a trained reconstruction model.

[0321] As shown in FIG. 31, the image generation process for generating a view image of a viewpoint ij in a space 1 is performed by using a three-dimensional point (x ｎ , y ｎ , z ｎ ) and viewpoint information (θ ｉ , φ ｊ ) is the trained space 1 reconstruction model 3110_1 (F θ ), and calculates the color and opacity of each 3D point as its output. Then, the image generation process performs volume rendering process 120 based on the calculated color and opacity of each 3D point for each pixel of the view image of space 1, thereby generating a view image of viewpoint ij in space 1.

[0322] As shown in FIG. 31, the image generation process for generating a view image of the viewpoint ij in the space 2 is performed by using a three-dimensional point (x ｎ , y ｎ , z ｎ ) and viewpoint information (θ ｉ , φ ｊ ) is the trained space 2 reconstruction model 3110_2 (F θ ), and calculates the color and opacity of each 3D point as its output. Then, the image generation process performs volume rendering process 120 based on the calculated color and opacity of each 3D point for each pixel of the view image of space 2, thereby generating a view image of viewpoint ij in space 2.

[0323] <Relationship between Captured Images and Trained Reconstruction Model> Next, a trained reconstruction model applied to the server device 410 according to the fifth embodiment will be described. Fig. 32 is a fifth diagram showing an example of a trained reconstruction model applied to the server device. Note that, for the sake of simplicity, Fig. 32 also shows a case in which two viewpoints, viewpoint 1 and viewpoint 2, are used. However, as described above, in the training process, captured images captured by an imaging device at a viewpoint other than viewpoint 1 and viewpoint 2 may be used.

[0324] 32, a group of trained restoration models corresponding to each space is applied to the server device 410. The group of trained restoration models corresponding to each space is trained in advance so as to be able to restore a scene from a first time to a second time using time-series captured images obtained by capturing each space of the scene successively in time from multiple viewpoints.

[0325] Specifically, the server device 410 stores the following: time information T １ A photographed image A of space 1 photographed by an imaging device at viewpoint 1 １_１ and time information T １ Image B of space 1 captured by an imaging device at viewpoint 2 １_１ and a trained space 1 reconstruction model F θ１ and time information T １ A photographed image A of space 2 photographed by an imaging device at viewpoint 1 １_２ and time information T １ A photographed image B of space 2 photographed by an imaging device at viewpoint 2 １_２ and a trained spatial 2 reconstruction model F θ１ The following applies.

[0326] Similarly, the server device 410 stores the following: time information T ２ A photographed image A of space 1 photographed by an imaging device at viewpoint 1 ２_１ and time information T ２ Image B of space 1 captured by an imaging device at viewpoint 2 ２_１ and a trained space 1 reconstruction model Fθ２ and time information T ２ A photographed image A of space 2 photographed by an imaging device at viewpoint 1 ２_２ and time information T ２ A photographed image B of space 2 photographed by an imaging device at viewpoint 2 ２_２ and a trained spatial 2 reconstruction model F θ２ The following applies.

[0327] In the example of FIG. 32, due to space limitations, the time information T １１ The trained space-1 reconstruction model F θ１１ and the trained spatial 2 reconstruction model F θ１１ However, the number of trained restoration models applied to the server device 410 is not limited to 22. However, it is assumed that each trained restoration model is managed as a time-series trained restoration model associated with temporal information and spatial information.

[0328] In FIG. 32, the time information T １ , T ２ , T ３ , ... are images taken by an imaging device during training processing. １_１ (or A １_２ ), A ２_１ (or A ２_２ ), ...or, photographed image B １_１ (or B １_２ ), B ２_１ (or B ２_２ ), ..., ..., corresponds to a frame period (an example of a first time interval). That is, a trained restoration model (another example of a first restoration model) of a time series of first time intervals corresponding to each space is applied to the server device 410 to generate time series view images of the first time intervals of each space.

[0329] <Example of Trained Recovery Model> Next, a trained recovery model stored in the model storage unit 606 in the server device 410 according to the fifth embodiment will be described. Fig. 33 is a diagram showing an example of a trained recovery model of the server device according to the fifth embodiment.

[0330] As shown in FIG. 33, the trained reconstruction model stored in the model storage unit 606 is associated with time information. Specifically, the time information T １ The trained space-reconstruction model F θ１ and the trained spatial 2 reconstruction model F θ１ is associated with the time information T ２ The trained space-reconstruction model F θ２ and the trained spatial 2 reconstruction model F θ２ Similarly, in the example of FIG. 33, the time information T ３ ~T １１ Then, the trained space 1 reconstruction model F θ３ and the trained spatial 2 reconstruction model F θ３ ~Trained space 1 reconstruction model F θ１１ and the trained spatial 2 reconstruction model F θ１１ The correspondence between the time information and the trained space 1 reconstruction model and the trained space 2 reconstruction model may be performed by directly associating the time information with the trained space 1 reconstruction model and the trained space 2 reconstruction model, or may be performed by indirectly associating the time information with the trained space 1 reconstruction model and the trained space 2 reconstruction model via other data.

[0331] The server device 410 uses a trained restoration model corresponding to each space stored in the model storage unit 606 to generate time-series view images according to the viewpoint information, time information, and space information included in the request received from the client terminal 420.

[0332] In FIG. 33, the time information T １ , T ２ , T ３ , ... correspond to the frame period of the images captured by the imaging device during the training process, as described above. １ , T ２ , T ３ , . . . correspond to the frame period when playing back free viewpoint video in the free viewpoint video playback system 400.

[0333] 33, the trained space 1 restoration model or the trained space 2 restoration model associated with each piece of time information is a different trained space 1 restoration model or a different trained space 2 restoration model. The different trained space 1 restoration model or the trained space 2 restoration model referred to here is configured by a neural network to which NeRF technology is applied, and is trained using different training data (captured images). The neural network architecture may be the same or may have some different architectures.

[0334] In addition, each trained space 1 restoration model or each trained space 2 restoration model shown in Figure 33 can generate a view image (free viewpoint image) from any viewpoint for the corresponding space in a scene of each temporal information.

[0335] 33 , the model storage unit 606 stores at least a set of trained space 1 restoration models and a set of trained space 2 restoration models for generating view images for a series of scenes for one object, for each space. However, the set of trained space 1 restoration models and the set of trained space 2 restoration models stored in the model storage unit 606 are not limited to one, and the model storage unit 606 may store another set of trained space 1 restoration models and trained space 2 restoration models for generating view images for a series of scenes for another one object, for each space.

[0336] 33, the group of trained space 1 reconstruction models and trained space 2 reconstruction models stored in the model storage unit 606 are stored in a table with time information T １ ~T １１ However, the number of trained space 1 reconstruction models and trained space 2 reconstruction models stored in the model storage unit 606 is not limited to this.

[0337] <Specific Example of Processing by Server Device> Next, a specific example of processing by the default moving image generating unit 602 and the requested moving image generating unit 604 of the server device 410 according to the fifth embodiment will be described.

[0338] (1) Specific example of processing by default video generation unit First, a specific example of processing by the default video generation unit 602 will be described. Fig. 34A is a first diagram showing a specific example of processing by the server device 410 according to the fifth embodiment. Fig. 34A shows a specific example of processing when the video designation acceptance unit 601 accepts designation of a free viewpoint video and the default video generation unit 602 receives notification of identification information of the designated free viewpoint video from the video designation acceptance unit 601.

[0339] As shown in FIG. 34A , the default video generation unit 602 selects the following trained restoration models for generating view images included in the specified free viewpoint video: θ１ ~F θ１１ , and a trained spatial 2 reconstruction model F θ１ ~F θ１１ , are read from the model storage unit 606.

[0340] In addition, the default video generation unit 602 reads out the trained space 1 reconstruction model F θ１ ~F θ１１ and the trained spatial 2 reconstruction model F θ１ ~F θ１１ For each, the default viewpoint information (θ ０ , φ ０ ) is input. This results in a trained space 1 reconstruction model F θ１ ~F θ１１ So, the default viewpoint information (θ ０ , φ ０ ) the view image X at each time point of the space 1 of the scene seen from the viewpoint based on １_１ ~X １１_１ Also, a trained spatial 2 reconstruction model F θ１ ~F θ１１ So, the default viewpoint information (θ ０ , φ ０ ) of the scene space 2 viewed from the viewpoint based on １_２ ~X １１_２ Generate.

[0341] In addition, the default video generating unit 602 generates the view image X １_１ and view image X１_２ ~ View Image X １１_１ and view image X １１_２ time information T １ ~T １１ and notifies the video transmission unit 605. １_１ and view image X １_２ ~ View Image X １１_１ and view image X １１_２ is transmitted in a transmission format that allows video playback on the client terminal 420.

[0342] (2) Specific example of processing by the requested video generation unit As described above, the default viewpoint information (θ ０ , φ ０ ) a view image X of the scene seen from a viewpoint based on １_１ ~X １１_１ and view image X １_２ ~X １１_２ It is assumed that a free viewpoint video having frame images of is played back by the client terminal 420. In response to this, it is assumed that a request including time information, viewpoint information, and spatial information is transmitted from the client terminal 420. In this case, the request receiving unit 603 receives the request and notifies the requested video generating unit 604.

[0343] Here, a specific example of processing by the requested moving image generating unit 604 when a request (time information, viewpoint information, spatial information) is notified from the request receiving unit 603 will be described. Fig. 34B is a second diagram showing a specific example of processing by the server device according to the fifth embodiment, and shows a specific example of processing by the requested moving image generating unit 604 when a request is notified from the request receiving unit 603.

[0344] As shown in FIG. 34B , the requested video generation unit 604 selects the requested (in the example of FIG. 34B , T ３ , space 1) corresponding to the trained space 1 reconstruction model F θ３ Identify.

[0345] In addition, the requested video generation unit 604 uses the specified trained space 1 restoration model F θ３ In contrast, the viewpoint information included in the request (in the example of FIG. 34B, (θｘ , φ ｘ )) is input. This results in a trained spatial 1 reconstruction model F θ３ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ３ View image X of space 1 in ３_１ Generate.

[0346] Next, the requested video generation unit 604 selects the trained space 1 reconstruction model F corresponding to the next time information (next time) as the next trained reconstruction model. θ４ In addition, the requested video generation unit 604 identifies the trained space 1 restoration model F θ４ In contrast, the viewpoint information included in the request (in the example of FIG. 34B, (θ ｘ , φ ｘ )) is input. This results in a trained spatial 1 reconstruction model F θ４ Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T ４ View image X of space 1 in ４_１ Generate.

[0347] Thereafter, the requested video generating unit 604 repeats the same process until an end condition is transmitted from the client terminal 420. In the example of FIG. 34B, the client terminal 420 transmits time information T １０ is shown to have been sent.

[0348] The client terminal 420 receives time information T １０ When the time information T １０ A trained space-1 reconstruction model F corresponding to θ１０ In addition, the requested video generation unit 604 identifies the trained space 1 restoration model F θ１０ In contrast, the viewpoint information included in the request (in the example of FIG. 34B, (θ ｘ , φ ｘ )) is input. This results in a trained spatial 1 reconstruction model F θ１０Then, the viewpoint information (θ ｘ , φ ｘ ) based on the time information T １０ View image X of space 1 in １０ Generate.

[0349] In this way, the request video generation unit 604 generates a time series of view images for the first time interval according to the viewpoint information using: a trained reconstruction model of a time series for a first time interval from a trained reconstruction model corresponding to the temporal information included in the request to a trained reconstruction model corresponding to a predetermined termination condition; and a trained reconstruction model of a time series corresponding to the spatial information included in the request.

[0350] The requested video generator 604 generates the view image X ３_１ ~ View Image X １０_１ time information T ３ ~T １０ The video transmitting unit 605 then notifies the video transmitting unit 605 of the view image X ３_１ ~ View Image X １０_１ can be transmitted in a transmission format that allows video playback on the client terminal 420.

[0351] <Flow of Free Viewpoint Video Playback Processing> Next, we will explain the flow of free viewpoint video playback processing by the free viewpoint video playback system 400. Fig. 35 is a second sequence diagram showing the flow of free viewpoint video playback processing by the free viewpoint video playback system.

[0352] In step S3520_1, the client terminal 420 receives a designation from the user 440 regarding the free viewpoint video to be displayed, and transmits, to the server device 410, identification information for uniquely identifying the designated free viewpoint video.

[0353] In step S3510_1, the server device 410 reads out a group of trained space 1 restoration models and a group of trained space 2 restoration models for the space 1 and the space 2 for generating view images included in the specified free viewpoint video. The server device 410 also adds default viewpoint information (θ０ , φ ０ ) to obtain the view image X of space 1. １_１ ~X １１_１ and the view image X of space 2 １_２ ~X １１_２ Generate.

[0354] In step S3510_2, the server device 410 sequentially transmits the generated view images of the space 1 and the space 2 to the client terminal 420.

[0355] In step S3520_2, the client terminal 420 plays back the free viewpoint video in which the view images of Space 1 and Space 2 transmitted from the server device 410 are used as frame images. The client terminal 420 also receives an instruction to stop the free viewpoint video being played back and transmits it to the server device 410. As a result, the server device 410 stops transmitting the view images of Space 1 and Space 2.

[0356] In step S3520_3, the client terminal 420 accepts an instruction to move the indicator 1112' on the seek bar 1112. The client terminal 420 sequentially transmits to the server device 410 time information on each position of the indicator 1112' that is being moved.

[0357] In step S3510_3, each time the server device 410 receives time information for each position of the moving indicator 1112' from the client terminal 420, the server device 410 inputs default viewpoint information into the trained space 1 restoration model and the trained space 2 restoration model for space 1 and space 2 corresponding to the time information for each position. As a result, the server device 410 generates view images for space 1 and space 2. Furthermore, the server device 410 sequentially transmits the generated view images for space 1 and space 2 to the client terminal 420. As a result, the client terminal 420 displays view images for space 1 and space 2 corresponding to the time information for each position of the moving indicator 1112'.

[0358] In step S3520_4, the client terminal 420 accepts the dragging of the moving image display area by the mouse pointer 1116. The client terminal 420 transmits to the server device 410 viewpoint information for each position of the moving mouse pointer 1116.

[0359] In step S3510_4, each time the server device 410 receives viewpoint information for each position of the moving mouse pointer 1116 from the client terminal 420, the server device 410 inputs the viewpoint information for each position into the trained space 1 restoration model and the trained space 2 restoration model for space 1 and space 2 that correspond to the current time information. As a result, the server device 410 generates view images for space 1 and space 2. Furthermore, the server device 410 sequentially transmits the generated view images for space 1 and space 2 to the client terminal 420. As a result, the client terminal 420 displays view images for space 1 and space 2 that correspond to the viewpoint information for each position of the moving mouse pointer 1116.

[0360] In step S3520_5, the client terminal 420 accepts input of space information (for example, space 1) and transmits it to the server device 410.

[0361] In step S3520_6, when the play button 1114 is pressed, the client terminal 420 transmits a play instruction to the server device 410.

[0362] In step S3510_5, the server device 410 generates a view image of space 1 by inputting the current viewpoint information into a trained space 1 restoration model corresponding to the current time information and the input spatial information (space 1), and transmits the view image to the client terminal 420. Next, the server device 410 generates a view image of space 1 by inputting the current viewpoint information into a trained space 1 restoration model corresponding to the next time information and the input spatial information (space 1), and transmits the view image to the client terminal 420. Thereafter, the server device 410 repeats the same process until an end condition is transmitted from the client terminal 420.

[0363] In step S3520_7, the client terminal 420 plays back the free viewpoint video in which the view image of space 1 transmitted from the server device 410 is used as a frame image. The client terminal 420 also receives an instruction to stop the free viewpoint video being played back and transmits it to the server device 410. As a result, the server device 410 stops generating and transmitting the view image of space 1.

[0364] <Summary> As is clear from the above description, the server device 410 according to the fifth embodiment includes one or more memories and one or more processors, and the one or more memories hold: a trained space 1 restoration model or a trained space 2 restoration model (first restoration model) for generating a time series of view images of a first time interval for a specific space, the trained space 1 restoration model or the trained space 2 restoration model (first restoration model) of the time series of the first time interval.

[0365] Furthermore, one or more processors included in the server device 410 according to the fifth embodiment generate a time series of view images for a first time interval according to viewpoint information, using a time series of trained space 1 restoration model or trained space 2 restoration model (first restoration model) for a first time interval from a trained space 1 restoration model or trained space 2 restoration model (first restoration model) corresponding to the time information included in the request to a trained space 1 restoration model or trained space 2 restoration model (first restoration model) corresponding to a predetermined termination condition. The time series of trained space 1 restoration model or trained space 2 restoration model (first restoration model) for the first time interval is a trained restoration model corresponding to the spatial information included in the request.

[0366] In this way, according to the fifth embodiment, it is possible to construct a mechanism for playing back a free viewpoint video about a specific space.

[0367] Sixth Embodiment In the first to fifth embodiments described above, the user 440 inputs time information, viewpoint information, and (spatial information) to the client terminal 420, and the server device 410 generates a view image according to the input time information, viewpoint information, and (spatial information).

[0368] However, the mechanism for playing back free viewpoint video on the client terminal 420 is not limited to this. For example, a configuration may be adopted in which the user 440 inputs time information and (spatial information), and the server device 410 transmits a trained reconstruction model corresponding to the input time information and (spatial information) to the client terminal 420. In this case, the client terminal 420 executes the received time-series trained reconstruction model based on the viewpoint information input by the user 440, thereby generating a view image corresponding to the viewpoint information and playing back free viewpoint video. This makes it possible to play back free viewpoint video using a mechanism different from the first to fifth embodiments. The sixth embodiment will be described below, focusing on the differences from the first embodiment.

[0369] <System configuration of free viewpoint video playback system> First, a description will be given of the system configuration of a free viewpoint video playback system including a server device according to embodiment 6. Fig. 36 is a second diagram showing an example of the system configuration of the free viewpoint video playback system.

[0370] 36 , a free viewpoint video playback system 3600 includes a server device 3610 according to the sixth embodiment and a client terminal 3620. In the free viewpoint video playback system 3600, the server device 3610 and the client terminal 3620 are communicatively connected via a communication network 430.

[0371] A restored model providing program is installed in the server device 3610, and by executing this program, the server device 3610 functions as a restored model providing unit 3611.

[0372] The restoration model providing unit 3611 receives a request from the client terminal 3620 via the communication network 430. Furthermore, the restoration model providing unit 3611 transmits a time-series trained restoration model read from the model storage unit 606 to the client terminal 3620 based on time information included in the received request.

[0373] A free viewpoint video playback program is installed in the client terminal 3620, and by executing this program, the client terminal 3620 functions as a free viewpoint video playback unit 3621. Note that the free viewpoint video playback program may be a dedicated application or a predetermined browser.

[0374] The free viewpoint video playback unit 3621 transmits a request including the time information input by the user 440 to the server device 3610 via the communication network 430 .

[0375] Furthermore, in response to transmitting a request to the server device 3610, the free viewpoint video playback unit 3621 receives a time-series trained restoration model transmitted from the server device 3610. Furthermore, the free viewpoint video playback unit 3621 executes the received time-series trained restoration model based on the viewpoint information input by the user 440, thereby generating time-series view images according to the viewpoint information at each piece of time information, and plays back the free viewpoint video by using the generated view images as frame images of the video.

[0376] <Functional Configuration of Server Device> Next, the functional configuration of the server device 3610 according to the sixth embodiment will be described. Fig. 37 is a second diagram showing an example of the functional configuration of the server device. As described above, the server device 3610 functions as a restoration model providing unit 3611. As shown in Fig. 37, the restoration model providing unit 3611 further includes a video designation receiving unit 3701, a request receiving unit 3702, a selection unit 3703, and a model sending unit 3704.

[0377] The video designation receiving unit 3701 receives designation of a free viewpoint video from the client terminal 3620. The server device 3610 according to the sixth embodiment is configured to be able to provide the client terminal 3620 with a group of trained restoration models for generating view images included in the free viewpoint video. The video designation receiving unit 3701 receives designation of one of the free viewpoint videos. The video designation receiving unit 3701 notifies the selection unit 3703 of identification information (e.g., an identifier (ID) of the free viewpoint video) for uniquely identifying the free viewpoint video for which designation has been received.

[0378] The request receiving unit 3702 receives a request transmitted from the client terminal 3620. In this embodiment, the request transmitted from the client terminal 3620 includes time information input by the user 440. The request received by the request receiving unit 3702 is notified to the selection unit 3703.

[0379] The selection unit 3703 notifies the model transmission unit 3704 of trained reconstruction models for generating view images included in the free viewpoint video identified by the identification information notified by the video designation receiving unit 3701, which trained reconstruction models correspond to the time information notified by the request receiving unit 3702. Specifically, the selection unit 3703 reads out, from a plurality of groups of trained reconstruction models held in the model storage unit 606, a group of trained reconstruction models for generating view images for each piece of time information (each time) included in the free viewpoint video notified by the video designation receiving unit 3701. Furthermore, the selection unit 3703 notifies the model transmission unit 3704 of at least some of the trained reconstruction models from the read group of trained reconstruction models that correspond to the time information notified by the request receiving unit 3702.

[0380] The selection unit 3703 performs processing according to the type of time information notified by the request reception unit 3702. For example, assume that the time information included in the request is time information based on a playback instruction in the client terminal 3620. This time information may be, for example, the time at which the user 440 issued a playback instruction for the video, regardless of whether playback is in progress or stopped in the client terminal 3620. In this case, the selection unit 3703 sequentially notifies the model transmission unit 3704 of the trained restoration models that have already been read out and that correspond to the time information notified by the request reception unit 3702.

[0381] Also, for example, assume that the time information included in the request is time information based on a stop instruction in the client terminal 3620 (an example of time information according to a termination condition). This time information may be, for example, the time at which the user 440 issues an instruction to stop playback of a video being played on the client terminal 3620. In this case, the selection unit 3703 identifies, from among the trained reconstruction models that have already been read out, the trained reconstruction model that corresponds to the time information notified by the request reception unit 3702 as the last trained reconstruction model being played, and notifies the model transmission unit 3704. Then, the selection unit 3703 notifies the model transmission unit 3704 of the identified last trained reconstruction model, and then stops processing.

[0382] Also, for example, assume that the time information included in the request is time information based on an operational instruction during a pause in the client terminal 3620. This time information may be, for example, a time based on an operational instruction (for example, an operational instruction on a seek bar indicator) given by the user 440 regarding a scene to be displayed while the video is paused in the client terminal 3620. In this case, each time time information is notified by the request receiving unit 3702, the selecting unit 3703 notifies the model sending unit 3704 of the trained restoration model corresponding to the time information.

[0383] The model sending unit 3704 sends the trained reconstruction model notified by the selection unit 3703 to the client terminal 3620. Note that the trained reconstruction model sent by the model sending unit 3704 to the client terminal 3620 may be the trained reconstruction model itself (program), or model parameters (including, for example, NN weight parameters) and / or hyperparameters (including, for example, the number of NN layers and the number of nodes in each layer) of the trained reconstruction model. Alternatively, in the case of a trained reconstruction model that has already been sent by the model sending unit 3704 to the client terminal 3620, the trained reconstruction model may be information for identifying the sent trained reconstruction model.

[0384] In other words, the trained reconstruction model transmitted by the model transmission unit 3704 refers to information for enabling the trained reconstruction model to be executed to be executed on the client terminal 3620 .

[0385] In this way, the model sending unit 3704 sends the trained restoration model notified by the selection unit 3703 to the client terminal 3620 in a sending format that can be executed in the client terminal 3620.

[0386] In the following description, it is assumed that the model transmitting unit 3704 transmits the trained reconstruction model itself (program).

[0387] <Example of Trained Recovery Model> Next, a trained recovery model stored in the model storage unit 606 in the server device 3610 according to the sixth embodiment will be described. Fig. 38 is a diagram showing an example of a trained recovery model stored in the model storage unit of the server device according to the sixth embodiment.

[0388] As shown in FIG. 38, the trained reconstruction model stored in the model storage unit 606 is associated with time information. Specifically, the time information T １ is a trained reconstruction model F θ１ is associated with the time information T ２ is a trained reconstruction model F θ２ Similarly, in the example of FIG. 38, the time information T ３ ~T １１ is a trained reconstruction model F θ３ ~F θ１１ The correspondence between the time information and the trained reconstruction model may be achieved by directly associating the time information with the trained reconstruction model, or by indirectly associating the time information with the trained reconstruction model via other data.

[0389] Note that the trained reconstruction model F shown in FIG. θ１ ~F θ１１ is the trained reconstruction model F shown in FIG. θ１ ~F θ１１ is the same as

[0390] <Specific Example of Processing by Server Apparatus> Next, a specific example of processing by each unit (here, the selection unit 3703) of the server apparatus 3610 will be described.

[0391] (1) Specific Example 1 of Processing by the Selector Fig. 39A is a first diagram illustrating a specific example of processing by the server device according to the sixth embodiment. Fig. 39A illustrates a specific example of processing when the selector 3703 receives a notification of identification information of a specified free viewpoint video from the video specification accepting unit 3701 and a notification of time information included in the request from the request accepting unit 3702.

[0392] As shown in FIG. 39A , the selection unit 3703, which has received a notification about the identification information of the specified free viewpoint video, selects a trained restoration model F θ１ ~F θ１１ is read from the model storage unit 606.

[0393] The selection unit 3703 also selects the trained reconstruction model F θ１ ~F θ１１ Among the time information included in the request (in the example of FIG. 39A, T ３ ) the corresponding trained reconstruction model F θ３ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained reconstruction model F θ３ to the client terminal 3620. As a result, the client terminal 3620 receives a trained reconstruction model F based on the default viewpoint information. θ３ is executed, and the default viewpoint information (θ ０ , φ ０ ) based on the time information T ３ view image at (for example, view image X ３ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ３ A free viewpoint video with the frame images is played.

[0394] Next, the selection unit 3703 selects a trained reconstruction model F corresponding to the next time information (next time) as the next trained reconstruction model. θ４and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained reconstruction model F θ４ As a result, the client terminal 3620 receives the default viewpoint information (θ ０ , φ ０ ) based on the trained reconstruction model F θ４ is executed, and the default viewpoint information (θ ０ , φ ０ ) based on the time information T ４ view image at (for example, view image X ４ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ４ A free viewpoint video with the frame images is played.

[0395] Thereafter, the selection unit 3703 repeats the same process until an end condition is transmitted from the client terminal 3620. In the example of FIG. 39A, the client terminal 3620 transmits time information T １０ is shown to have been sent.

[0396] The termination condition here refers to time information based on a stop instruction to stop playback of a free viewpoint video in response to a request. When a stop button to stop a free viewpoint video being played is pressed, the client terminal 3620 transmits time information corresponding to the timing of the button press as the termination condition to the server device 3610. Alternatively, when the client terminal 3620 receives, for example, a time range specification when playing back a free viewpoint video, it transmits time information corresponding to the end timing of the time range as the termination condition to the server device 3610.

[0397] The client terminal 3620 receives time information T １０ is transmitted, the selection unit 3703 selects the time information T １０ A trained reconstruction model F corresponding to θ１０ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained reconstruction model Fθ１０ As a result, the client terminal 3620 receives the default viewpoint information (θ ０ , φ ０ ) based on the trained reconstruction model F θ１０ In addition, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) based on the time information T １０ view image at (for example, view image X １０ ) is generated. Furthermore, in the client terminal 3620, the generated view image X １０ A free viewpoint video with the frame images is played.

[0398] (2) Specific Example 2 of Processing by the Selector As described above, the time information T ３ From the time information T corresponding to the termination condition １０ A trained reconstruction model F θ３ ~ Trained reconstruction model F θ１０ is transmitted to the client terminal 3620. As a result, the view image X ３ ~ View Image X １０ It is assumed that a free viewpoint video having frame images of is played back on the client terminal 3620. Furthermore, it is assumed that a request including time information is transmitted from the client terminal 3620, and viewpoint information is input by the user 440 at the client terminal 3620. In this case, the request receiving unit 3702 receives the request and notifies the selecting unit 3703.

[0399] Here, a specific example of processing by the selection unit 3703 when the request (time information) is notified from the request reception unit 3702 will be described. Fig. 39B is a second diagram showing a specific example of processing by the server device according to the sixth embodiment, and shows a specific example of processing by the selection unit 3703 when the request is notified from the request reception unit 3702.

[0400] As shown in FIG. 39B, the selection unit 3703 selects a trained reconstruction model F that has already been read from the model storage unit 606. θ１ ~F θ１１The time information included in the request (in the example of FIG. 39B, T １ ) the corresponding trained reconstruction model F θ１ Identify.

[0401] The selection unit 3703 also selects the identified trained reconstruction model F θ１ to the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the trained reconstruction model F θ１ As a result, the client terminal 3620 receives the viewpoint information (θ ｘ , φ ｘ ) based on the trained reconstruction model F θ１ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T １ view image at (for example, view image X １ ) is generated. Furthermore, in the client terminal 3620, the generated view image X １ A free viewpoint video with the frame images is played.

[0402] Next, the selection unit 3703 selects a trained reconstruction model F corresponding to the next time information (next time) as the next trained reconstruction model. θ２ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained reconstruction model F θ２ As a result, the client terminal 3620 receives the viewpoint information (θ ｘ , φ ｘ ) based on the trained reconstruction model F θ２ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T ２ view image at (for example, view image X ２ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ２ A free viewpoint video with the frame images is played.

[0403] Next, the selection unit 3703 selects a trained reconstruction model F corresponding to the next time information (next time) as the next trained reconstruction model. θ３ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained reconstruction model F θ３ As a result, the client terminal 3620 receives the viewpoint information (θ ｘ , φ ｘ ) based on the trained reconstruction model F θ３ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T ３ view image at (for example, view image X ３ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ３ A free viewpoint video with the frame images is played.

[0404] Thereafter, the selection unit 3703 repeats the same process until an end condition is transmitted from the client terminal 3620. In the example of FIG. 39B, the client terminal 3620 transmits time information T １１ is shown to have been sent.

[0405] The client terminal 3620 receives time information T １１ is transmitted, the selection unit 3703 selects the time information T １１ A trained reconstruction model F corresponding to θ１１ The selection unit 3703 also selects the identified trained reconstruction model F θ１１ to the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the trained reconstruction model F θ１１ As a result, the client terminal 3620 receives the viewpoint information (θ ｘ , φ ｘ ) based on the trained reconstruction model F θ１１In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T １１ view image at (for example, view image X １１ ) is generated. Furthermore, in the client terminal 3620, the generated view image X １１ A free viewpoint video with the frame images is played.

[0406] In the above description, the selection unit 3703 is configured to notify all identified trained restoration models to the model transmission unit 3704. However, the processing by the selection unit 3703 is not limited to this. For example, if the selection unit 3703 recognizes that the identified trained restoration model has already been transmitted to the client terminal 3620, the selection unit 3703 may be configured not to notify the model transmission unit 3704.

[0407] Specifically, in the case of FIG. 39B, the selection unit 3703 selects the trained reconstruction model F θ３ ~F θ１０ The model sending unit 3704 may not be notified of the above.

[0408] (3) Specific Example 3 of Processing by the Selection Unit Next, a description will be given of another specific example (a specific example different from Specific Example 1) of processing by the selection unit 3703 when a request (time information) is notified from the request receiving unit 3702. In the above-described Specific Example 1, when identifying the next trained reconstruction model, the selection unit 3703 identifies the next trained reconstruction model at a time interval corresponding to the frame period.

[0409] On the other hand, in the free viewpoint video playback system 3600, even if a specified trained restoration model is transmitted, it is not always possible to play back all view images as frame images on the client terminal 3620. For example, in the following cases, it is not always possible to play back all view images as frame images on the client terminal 3620: when the frame period on the client terminal 3620 is longer than the time interval of the transmitted time-series trained restoration model; when the display mode on the client terminal 3620 is double-speed mode or 10-second skip mode; when the communication load between the server device 3610 and the client terminal 3620 is high and the communication speed is reduced; or when the processing load on the server device 3610 or the client terminal 3620 is increased. Here, a specific example of processing (thinning processing) by the selection unit 3703 in a case where it is not possible to play back all view images as frame images on the client terminal 3620 will be described. FIG. 39C is a third diagram showing a specific example of processing by the server device according to the sixth embodiment.

[0410] As shown in FIG. 39C, the selection unit 3703 selects a trained reconstruction model F that has already been read from the model storage unit 606. θ１ ~F θ１１ Among the time information included in the request (in the example of FIG. 39C, ３ ) the corresponding trained reconstruction model F θ３ Identify.

[0411] The selection unit 3703 also selects the identified trained reconstruction model F θ３ to the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the trained reconstruction model F θ３ As a result, the client terminal 3620 receives the default viewpoint information (θ ０ , φ ０ ) based on the trained reconstruction model F θ３ is executed, and the default viewpoint information (θ ０ , φ ０ ) based on the time information T ３view image at (for example, view image X ３ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ３ A free viewpoint video with the frame images is played.

[0412] Next, the selection unit 3703 determines the generation timing of a view image when identifying the next trained restoration model. The selection unit 3703 acquires information on: the frame period of the client terminal 3620; the display mode of the client terminal 3620; the communication load between the server device 3610 and the client terminal 3620; and the processing load of the server device 3610 and the client terminal 3620, and determines the generation timing of the view image based on the acquired information.

[0413] In the example of FIG. 39C, the selection unit 3703 selects the generation timing of the view image based on the time information T ６ and the trained reconstruction model F is used as the next trained reconstruction model. θ６ 39C shows a state in which the selection unit 3703 selects the identified trained restoration model F θ６ is notified to the model sending unit 3704, and the notified trained recovery model F θ６ is transmitted to the client terminal 3620. In addition, the example of FIG. 39C shows a state in which the default viewpoint information (θ ０ , φ ０ ) based on the trained reconstruction model F θ６ Furthermore, the example of FIG. 39C shows the state where the default viewpoint information (θ ０ , φ ０ ) based on the time information T ６ view image at (for example, view image X ６ ) is generated.

[0414] 39C, the selection unit 3703 repeats the same process (thinning-out process) until the end condition is transmitted from the client terminal 3620. In the example of FIG. 39C, the client terminal 3620 transmits time information T １０is shown to have been sent.

[0415] The client terminal 3620 receives time information T １０ is transmitted, the selection unit 3703 determines that it is not time to generate a view image, and selects the trained restoration model F θ６ The process stops without specifying the

[0416] <Functional Configuration of Client Terminal> Next, the functional configuration of the client terminal 3620 according to the sixth embodiment will be described. Fig. 40 is a second diagram showing an example of the functional configuration of the client terminal. As described above, the client terminal 3620 functions as a free viewpoint video playback unit 3621. As shown in Fig. 40 , the free viewpoint video playback unit 3621 further includes a video designation transmission unit 4001, a video display unit 4002, a request transmission unit 4003, a restoration model reception unit 4004, a requested video generation unit 4005, and a video playback unit 4006.

[0417] The video designation sending unit 4001 receives, for example, a designation of a free viewpoint video from the user 440 via a video designation screen, and input of time information for playing the free viewpoint video. The video designation sending unit 4001 transmits identification information for uniquely identifying the free viewpoint video for which the designation has been accepted to the server device 3610. The video designation sending unit 4001 also notifies the request sending unit 4003 of a request including the time information for which the input has been accepted.

[0418] The request sending unit 4003 sends a request including the time information notified by the video designation sending unit 4001 to the server device 3610. Alternatively, the request sending unit 4003 acquires time information input by the user 440 from the video display unit 4002 via the video playback screen on which the free viewpoint video is played, and sends a request including the acquired time information to the server device 3610.

[0419] During playback, the video display unit 4002 plays, on the video playback screen, free viewpoint video, with frame images being view images notified at a predetermined frame cycle by the video playback unit 4006. The video display unit 4002 also accepts time information input by the user 440 on the video playback screen on which the free viewpoint video is played, and notifies the request transmission unit 4003 of the time information.

[0420] As described above, the time information included in the request notified to the request sending unit 905 includes: time information based on a playback instruction; time information based on a stop instruction; and time information based on various operations during stop.

[0421] Furthermore, the video display unit 4002 accepts viewpoint information input by the user 440 while the free viewpoint video is stopped on the video playback screen on which the free viewpoint video is being played, and notifies the requested video generation unit 4005 of this.

[0422] Furthermore, the video display unit 4002 displays the view image notified by the video playback unit 4006 when time information or viewpoint information is input during stoppage on the video playback screen at the notified timing.

[0423] The restoration model receiving unit 4004 receives the trained restoration model transmitted from the server device 3610 and notifies the requested video generating unit 4005 of the trained restoration model.

[0424] The requested video generation unit 4005 inputs default viewpoint information or viewpoint information notified by the video display unit 4002 into the trained restoration model notified by the restoration model receiving unit 4004, thereby executing the trained restoration model and generating a view image. The requested video generation unit 4005 also notifies the video playback unit 4006 of the generated view image.

[0425] During playback, the video playback unit 4006 notifies the video display unit 4002 of the view image notified by the request video generation unit 4005 as a frame image at a predetermined frame period. Also, during stoppage, the video playback unit 4006 notifies the video display unit 4002 of the view image notified by the request video generation unit 4005.

[0426] <Display Screen of Client Terminal> Next, the display screen (video selection screen, video playback screen) of the client terminal 3620 according to the sixth embodiment will be described. The display screen of the client terminal 3620 according to the sixth embodiment is similar to the display screen of the client terminal 420 according to the first embodiment (FIGS. 10 to 13). However, the video selection screen 1000 of the client terminal 3620 according to the sixth embodiment may be configured to allow the user to select a free viewpoint video and to input time information that specifies the start position of playback.

[0427] <Flow of free viewpoint video playback processing> Next, a description will be given of the flow of free viewpoint video playback processing by the free viewpoint video playback system 3600 according to the sixth embodiment. Fig. 41 is a third sequence diagram showing the flow of free viewpoint video playback processing by the free viewpoint video playback system.

[0428] In step S4120_1, the client terminal 3620 receives a designation from the user 440 regarding the free viewpoint video to be displayed, and transmits, to the server device 3610, identification information for uniquely identifying the designated free viewpoint video.

[0429] In step S4120_2, the client terminal 3620 receives the time information T ３ The input of time information T ３ to the server device 3610.

[0430] In step S4110_1, the server device 3610 reads out a group of trained reconstruction models for generating view images included in the specified free viewpoint video. In addition, the server device 3610 reads out a group of trained reconstruction models for generating view images included in the specified free viewpoint video. ３ A trained reconstruction model F θ３ , and then sequentially transmits the data to the client terminal 3620.

[0431] In step S4120_3, the client terminal 3620 receives the trained reconstruction models sequentially transmitted from the server device 3610, and adds default viewpoint information (θ ０ , φ０ ) is input. As a result, the client terminal 3620 inputs the time information T ３ The default viewpoint information (θ ０ , φ ０ ) view images are generated sequentially.

[0432] In step S4120_4, the client terminal 3620 receives the stop instruction and transmits the received stop instruction to the server device 3610. As a result, the server device 3610 transmits the trained reconstruction model F θ１０ After transmitting the trained reconstruction model to the client terminal 3620 until the time information T ３ The default viewpoint information (θ ０ , φ ０ ) view image X ３ ~X １０ It is possible to play free viewpoint videos with the frame images as follows.

[0433] In step S4120_5, the client terminal 3620 accepts an instruction to move the indicator 1112' on the seek bar 1112. The client terminal 3620 sequentially transmits to the server device 3610 time information on each position of the indicator 1112' that is being moved.

[0434] In step S4110_2, the server device 3610 transmits to the client terminal 3620 a trained reconstruction model corresponding to the time information of each position of the moving indicator 1112' every time the server device 3610 receives the time information of each position of the moving indicator 1112' from the client terminal 3620. At this time, the server device 3610 does not transmit a trained reconstruction model that has already been transmitted to the client terminal 3620, but transmits a trained reconstruction model that has not been transmitted to the client terminal 3620. In the example of FIG. 41 , the indicator 1112' of the seek bar 1112 is moved in the direction of the time information T １ Since the server device 3610 has been moved to the position θ２ , F θ１ is transmitted to the client terminal 3620.

[0435] In step S4120_6, the client terminal 3620 adds default viewpoint information (θ ０ , φ ０ ) to generate a view image. As a result, a view image corresponding to the time information of each position of the moving indicator 1112' is displayed on the client terminal 3620. As described above, in the example of FIG. 41, the indicator 1112' of the seek bar 1112 is displayed at the time information T １ Therefore, the client terminal 3620 obtains view images X as view images corresponding to the time information of each position of the moving pointer 1112′. １０ ~X １ is displayed.

[0436] In step S4120_7, the client terminal 3620 receives the viewpoint information (θ ｘ , φ ｘ ) input is accepted.

[0437] At step S 4120 _ 8 , when the play button 1114 is pressed, the client terminal 3620 transmits a play instruction to the server device 3610 .

[0438] In step S4110_3, the server device 3610 receives the time information T １ A trained reconstruction model F θ１ , sequentially transmit the trained reconstruction models to the client terminal 3620. However, the server device 3610 does not transmit the trained reconstruction models that have already been transmitted to the client terminal 3620, but transmits the trained reconstruction models that have not been transmitted to the client terminal 3620.

[0439] In step S4120_9, the client terminal 3620 adds viewpoint information (θ ｘ , φ ｘ ) is input. As a result, the client terminal 3620 inputs the time information T １ The view image is based on the input viewpoint information (θｘ , φ ｘ ) view images are generated sequentially.

[0440] In step S4120_10, the client terminal 3620 receives a stop instruction and transmits the received stop instruction to the server device 3610. As a result, the server device 3610 transmits the trained reconstruction model F θ１１ After transmitting the trained reconstruction model to the client terminal 3620 until the time information T １ The view image is based on the input viewpoint information (θ ｘ , φ ｘ ) view image X １ ~X １１ It is possible to play free viewpoint videos with the frame images as follows.

[0441] <Summary> As is clear from the above description, the server device 3610 according to the sixth embodiment includes one or more memories and one or more processors, and the one or more memories hold one or more trained restoration models (first restoration models) that have been trained in advance to be able to restore a scene from a first time to a second time using time-series captured images from a plurality of viewpoints obtained by capturing a scene from each of the plurality of viewpoints in a temporally continuous manner. The one or more trained restoration models (first restoration models) are time-series trained restoration models for a first time interval that generate time-series view images for the first time interval.

[0442] Furthermore, the one or more processors: receive a request including time information for a scene from a client terminal; and transmit at least a portion of the stored trained restoration models (first restoration models) in a transmission format executable by the client terminal in response to the received request from the client. Specifically, transmit, in a transmission format executable by the client terminal, a time-series trained restoration model (first restoration model) for a first time interval from a trained restoration model (first restoration model) corresponding to the time information included in the request to a trained restoration model (first restoration model) corresponding to a predetermined termination condition. In this way, a free viewpoint video, in which time-series view images corresponding to viewpoint information are used as frame images and generated using at least a portion of the trained restoration model (first restoration model), is played back on the client terminal.

[0443] As described above, according to the sixth embodiment, it is possible to provide a mechanism for playing back free viewpoint video that is different from the first to fifth embodiments.

[0444] [Seventh Embodiment] In the sixth embodiment, the model storage unit 606 has been described as holding one trained restoration model for each piece of time information, and one trained restoration model generating a view image at one piece of time information. However, the trained restoration model is not limited to this, and the model storage unit 606 may hold trained restoration models capable of generating view images at multiple pieces of continuous time information. Hereinafter, the seventh embodiment will be described, focusing on the differences from the sixth embodiment.

[0445] <Example of Trained Recovery Model> First, a trained recovery model stored in the model storage unit 606 in the server device 3610 according to the seventh embodiment will be described. Fig. 42 is a diagram showing an example of a trained recovery model stored in the model storage unit of the server device according to the seventh embodiment.

[0446] As shown in FIG. 42, the trained reconstruction model stored in the model storage unit 606 is associated with time information. Specifically, the time information T １ ~T ３is a trained reconstruction model F θ１_θ３ is associated with the time information T ４ ~T ６ is a trained reconstruction model F θ４_θ６ Similarly, in the example of FIG. 42, the time information T ７ ~T ９ , T １０ ~T １２ Then, the trained reconstruction model F θ７_θ９ , F θ１０_θ１２ , respectively. That is, each model has its own corresponding (supporting) temporal information. The correspondence between the temporal information and the trained reconstruction model may be achieved by directly associating the temporal information with the trained reconstruction model, or by indirectly associating the temporal information with the trained reconstruction model via other data.

[0447] Note that the trained reconstruction model F shown in FIG. θ１_θ３ ~F θ１０_θ１２ is the trained reconstruction model F shown in FIG. θ１_θ３ ~F θ１０_θ１２ This is the same pre-trained reconstruction model as

[0448] <Specific Example of Processing by Server Apparatus> Next, a specific example of processing by the selection unit 3703 of the server apparatus 3610 according to the seventh embodiment will be described.

[0449] (1) Specific Example 1 of Processing by the Selector Fig. 43A is a first diagram illustrating a specific example of processing by the server device according to the seventh embodiment. Fig. 43A illustrates a specific example of processing when the selector 3703 receives a notification of identification information of a specified free viewpoint video from the video specification accepting unit 3701 and a notification of time information included in the request from the request accepting unit 3702.

[0450] As shown in FIG. 43A , the selection unit 3703, which has been notified of the identification information for uniquely identifying the specified free viewpoint video, selects a trained restoration model F for generating a view image included in the specified free viewpoint video. θ１_θ３ ~F θ１０_θ１２ is read from the model storage unit 606.

[0451] The selection unit 3703 also selects the trained reconstruction model F θ１_θ３ ~F θ１０_θ１２ Among them, the time information T ３ A trained reconstruction model F corresponding to θ１_θ３ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained reconstruction model F θ１_θ３ As a result, the client terminal 3620 receives the default viewpoint information (θ ０ , φ ０ ) based on the trained reconstruction model F θ１_θ３ In addition, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) based on the time information T ３ view image at (for example, view image X ３ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ３ A free viewpoint video with the frame images is played.

[0452] Next, the selection unit 3703 selects the trained reconstruction model F as the next trained reconstruction model. θ４_θ６ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained reconstruction model F θ４_θ６ As a result, the client terminal 3620 receives the default viewpoint information (θ ０ , φ ０ ) based on the trained reconstruction model F θ４_θ６ In addition, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) ４ ~T ６ view image at (for example, view image X ４ ~X ６ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ４ ~X ６ A free viewpoint video with the frame images is played.

[0453] Thereafter, the selection unit 3703 repeats the same process until an end condition is transmitted from the client terminal 3620. In the example of FIG. 43A, the client terminal 3620 transmits time information T １０ is shown to have been sent.

[0454] The client terminal 3620 receives time information T １０ is transmitted, the selection unit 3703 selects the time information T １０ A trained reconstruction model F corresponding to θ１０_θ１２ The selection unit 3703 also selects the identified trained reconstruction model F θ１０_θ１２ to the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the trained reconstruction model F θ１０_θ１２ As a result, the client terminal 3620 receives the default viewpoint information (θ ０ , φ ０ ) based on the trained reconstruction model F θ１０_θ１２ In addition, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) based on the time information T １０ view image at (for example, view image X １０ ) is generated. Furthermore, in the client terminal 3620, the generated view image X １０ A free viewpoint video with the frame images is played.

[0455] (2) Specific Example 2 of Processing by the Selector As described above, the time information T ３ From the time information T corresponding to the termination condition １０ A trained reconstruction model F θ１_θ３ ~ Trained reconstruction model F θ１０_θ１２ is transmitted to the client terminal 3620. As a result, the view image X ３ ~ View Image X １０It is assumed that a free viewpoint video having frame images of is played back on the client terminal 3620. Furthermore, it is assumed that a request including time information is transmitted from the client terminal 3620, and viewpoint information is input by the user 440 at the client terminal 3620. In this case, the request receiving unit 3702 receives the request and notifies the selecting unit 3703.

[0456] Here, a specific example of processing by the selection unit 3703 when a request (time information) is notified from the request reception unit 3702 will be described. Fig. 43B is a second diagram showing a specific example of processing by the server device according to the seventh embodiment, and shows a specific example of processing by the selection unit 3703 when the request is notified from the request reception unit 3702.

[0457] As shown in FIG. 43B, the selection unit 3703 selects a trained reconstruction model F θ１_θ３ ~F θ１０_θ１２ The time information included in the request (in the example of FIG. 43B, T １ ) the corresponding trained reconstruction model F θ１_θ３ The trained reconstruction model F θ１_θ３ has already been transmitted to the client terminal 3620. Therefore, the selection unit 3703 transmits the trained reconstruction model F θ１_θ３ , and the trained reconstruction model F θ１_θ３ However, it is not transmitted to the client terminal 3620.

[0458] On the other hand, the client terminal 3620 receives time information (in the example of FIG. 43B, T １ ) and viewpoint information input by the user 440 (in the example of FIG. 43B, (θ ｘ , φ ｘ )) and the trained reconstruction model F θ１_θ３ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T １ view image at (for example, view image X １ ) is generated. Furthermore, in the client terminal 3620, the generated view image X１ A free viewpoint video with the frame images is played.

[0459] In addition, the client terminal 3620 receives the next time information (in the example of FIG. 43B, T ２ ) and viewpoint information input by the user 440 (in the example of FIG. 43B, (θ ｘ , φ ｘ )) and the trained reconstruction model F θ１_θ３ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T ２ view image at (for example, view image X ２ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ２ A free viewpoint video with the frame images is played.

[0460] Furthermore, the client terminal 3620 receives the next time information (in the example of FIG. 43B, T ３ ) and viewpoint information input by the user 440 (in the example of FIG. 43B, (θ ｘ , φ ｘ )) and the trained reconstruction model F θ１_θ３ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T ３ view image at (for example, view image X ３ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ３ A free viewpoint video with the frame images is played.

[0461] Next, the selection unit 3703 selects the trained reconstruction model F as the next trained reconstruction model. θ４_θ６ The trained reconstruction model F θ４_θ６ has already been transmitted to the client terminal 3620. Therefore, the selection unit 3703 transmits the trained reconstruction model F θ４_θ６ , and the trained reconstruction model F θ４_θ６However, it is not transmitted to the client terminal 3620.

[0462] On the other hand, the client terminal 3620 receives the next time information (T ４ ) and viewpoint information input by the user 440 (in the example of FIG. 43B, (θ ｘ , φ ｘ )) and the trained reconstruction model F θ４_θ６ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T ４ view image at (for example, view image X ４ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ４ A free viewpoint video with the frame images is played.

[0463] In addition, the client terminal 3620 receives the next time information (in the example of FIG. 43B, T ５ ) and viewpoint information input by the user 440 (in the example of FIG. 43B, (θ ｘ , φ ｘ )) and the trained reconstruction model F θ４_θ６ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T ５ view image at (for example, view image X ５ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ５ A free viewpoint video with the frame images is played.

[0464] Furthermore, the client terminal 3620 receives the next time information (in the example of FIG. 43B, T ６ ) and viewpoint information input by the user 440 (in the example of FIG. 43B, (θ ｘ , φ ｘ )) and the trained reconstruction model F θ４_θ６ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ) based on the time information T ６ view image at (for example, view image X ６ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ６ A free viewpoint video with the frame images is played.

[0465] Thereafter, the selection unit 3703 repeats the same process until an end condition is transmitted from the client terminal 3620. In the example of FIG. 43B, the client terminal 3620 transmits time information T １１ is shown to have been sent.

[0466] The client terminal 3620 receives time information T １１ is transmitted, the selection unit 3703 selects the time information T １１ A trained reconstruction model F corresponding to θ１０_θ１２ The trained reconstruction model F θ１０_θ１２ has already been transmitted to the client terminal 3620. Therefore, the selection unit 3703 transmits the trained reconstruction model F θ１０_θ１２ , and the trained reconstruction model F θ１０_θ１２ However, it is not transmitted to the client terminal 3620.

[0467] On the other hand, the client terminal 3620 receives the time information T １１ and viewpoint information input by the user 440 (in the example of FIG. 43B, (θ ｘ , φ ｘ )) and the trained reconstruction model F θ１０_θ１２ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T １１ view image at (for example, view image X １１ ) is generated. Furthermore, in the client terminal 3620, the generated view image X １１ A free viewpoint video with the frame images is played.

[0468] <Flow of free viewpoint video playback processing> Next, a flow of free viewpoint video playback processing by the free viewpoint video playback system 3600 according to the seventh embodiment will be described. Fig. 44 is a fourth sequence diagram showing the flow of free viewpoint video playback processing by the free viewpoint video playback system. Here, the description will focus on differences from the third sequence diagram shown in Fig. 41. The differences from the third sequence diagram shown in Fig. 41 are that the fourth sequence diagram of Fig. 44 includes the processing of step S4410_1 instead of the processing of step S4110_1, and does not include the processing of steps S4110_2 and S4110_3.

[0469] In step S4410_1, the server device 3610 reads out a group of trained reconstruction models for generating view images included in the specified free viewpoint video. In addition, the server device 3610 reads out a group of trained reconstruction models for generating view images included in the specified free viewpoint video. ３ A trained reconstruction model F θ１_θ３ The server device 3610 sequentially transmits the trained reconstruction model F θ１０_θ１２ After transmitting the trained reconstruction model to the client terminal 3620, transmission of the trained reconstruction model is stopped.

[0470] The reason why the processes of steps S4110_2 and S4110_3 are not included in the fourth sequence diagram of FIG. 44 is as follows.

[0471] That is, the time information T １ , T ２ , T １１ A trained reconstruction model F for generating a view image corresponding to θ１_θ３ and the trained reconstruction model F θ１０_θ１２ has already been transmitted to the client terminal 3620 in step S4410_1. Therefore, the selection unit 3703 transmits the trained reconstruction model F θ１_θ３ and the trained reconstruction model F θ１０_θ１２ In addition, the trained reconstruction model F θ１_θ３ and the trained reconstruction model F θ１０_θ１２ However, it is not transmitted to the client terminal 3620.

[0472] <Summary> As is clear from the above explanation, the one or more memories included in the server device 3610 according to the seventh embodiment hold a trained restoration model (second restoration model) for generating a time series of view images of a first time interval, the trained restoration model (second restoration model) being a time series of a second time interval that is longer than the first time interval.

[0473] Furthermore, the one or more processors provided in the server device 3610 according to the seventh embodiment: Transmit, in a transmission format executable on the client terminal 3620, a trained restoration model (second restoration model) of a time series of a second time interval from a trained restoration model (second restoration model) corresponding to the time information included in the request to a trained restoration model (second restoration model) corresponding to a predetermined termination condition.

[0474] As a result, according to the seventh embodiment, a mechanism for playing back free viewpoint video that is different from that of the sixth embodiment can be constructed.

[0475] [Eighth Embodiment] In the seventh embodiment, the model storage unit 606 stores a trained restoration model that generates view images at three consecutive pieces of time information as a trained restoration model that generates view images at multiple consecutive pieces of time information. However, the model storage unit 606 may store a trained restoration model that generates view images at a full time range as a trained restoration model that generates view images at multiple consecutive pieces of time information. Note that the full time range here refers to a finite time range captured by an imaging device, and in the eighth embodiment, it is described as, for example, three minutes. Note that if the frame rate is 30 fps, a three-minute free viewpoint video will include 5,400 frame images.

[0476] <Example of Trained Recovery Model> First, a trained recovery model stored in the model storage unit 606 in the server device 3610 according to the eighth embodiment will be described. Fig. 45 is a diagram showing an example of a trained recovery model stored in the model storage unit of the server device according to the eighth embodiment.

[0477] As shown in FIG. 45, the trained reconstruction model stored in the model storage unit 606 is associated with time information. Specifically, the time information T １ ~T ５４００ is a trained reconstruction model F θ１_θ５４００ The trained reconstruction model F shown in FIG. θ１_θ５４００ is the trained reconstruction model F shown in FIG. θ１_θ５４００ is the same as

[0478] <Specific Example of Processing by Server Apparatus> Next, a specific example of processing by the selection unit 3703 of the server apparatus 3610 according to the eighth embodiment will be described.

[0479] (1) Specific Example 1 of Processing by the Selector Fig. 46A is a first diagram illustrating a specific example of processing by the server device according to the eighth embodiment. Fig. 46A illustrates a specific example of processing when the selector 3703 receives a notification of identification information of a specified free viewpoint video from the video specification accepting unit 3701 and a notification of time information included in the request from the request accepting unit 3702.

[0480] As shown in FIG. 46A , the selection unit 3703, which has received a notification of the identification information of the specified free viewpoint video, selects a trained restoration model F θ１_θ５４００ is read from the model storage unit 606.

[0481] The selection unit 3703 also selects the trained reconstruction model F θ１_θ５４００ to the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the trained reconstruction model F θ１_θ５４００ As a result, the client terminal 3620 receives the default viewpoint information (θ ０ , φ ０) based on the trained reconstruction model F θ１_θ５４００ In addition, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) based on the time information T ３ View image X in ３ Furthermore, the client terminal 3620 generates the generated view image X ３ A free viewpoint video with the frame images is played.

[0482] In addition, the client terminal 3620 receives the next time information (in the example of FIG. 46A, T ４ ) and default viewpoint information (in the example of FIG. 46A, (θ ０ , φ ０ )) and the trained reconstruction model F θ１_θ５４００ In addition, the client terminal 3620 receives viewpoint information (θ ０ , φ ０ ) based on the time information T ４ view image at (for example, view image X ４ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ４ A free viewpoint video with the frame images is played.

[0483] Thereafter, the client terminal 3620 repeats the same process until the user 440 inputs the termination condition. In the example of FIG. 46A, the user 440 inputs the time information T １０ is input.

[0484] As a termination condition, time information T １０ When the time information T １０ and viewpoint information input by the user 440 (in the example of FIG. 43A, (θ ０ , φ ０ )) and the trained reconstruction model F θ１_θ５４００ In addition, the client terminal 3620 receives viewpoint information (θ ０ , φ ０ ) based on the time information T １０view image at (for example, view image X １０ ) is generated. Furthermore, in the client terminal 3620, the generated view image X １０ A free viewpoint video with the frame images is played.

[0485] (2) Specific Example 2 of Processing by the Selection Unit As described above, the trained reconstruction model F θ１_θ５４００ is transmitted, and the time information T ３ From the time information T corresponding to the termination condition １０ View image X at each time point ３ ~X １０ is generated by the client terminal 3620. Also, the generated view image X ３ ~X １０ It is assumed that a free viewpoint video having frame images of is played back by the client terminal 3620. In addition, it is assumed that a request including time information is transmitted from the client terminal 3620, and viewpoint information is input by the user 440 to the client terminal 3620. In this case, the request receiving unit 3702 receives the request and notifies the selecting unit 3703.

[0486] Here, a specific example of processing by the selection unit 3703 when a request (time information) is notified from the request reception unit 3702 will be described. Fig. 46B is a second diagram showing a specific example of processing by the server device according to the eighth embodiment, and shows a specific example of processing by the selection unit 3703 when a request is notified from the request reception unit 3702.

[0487] As shown in FIG. 46B, the selection unit 3703 selects the time information included in the request (in the example of FIG. 46B, T １ ) the corresponding trained reconstruction model F θ１_θ５４００ The trained reconstruction model F θ１_θ５４００ has already been transmitted to the client terminal 3620. Therefore, the selection unit 3703 transmits the trained reconstruction model F θ１_θ５４００ , and the trained reconstruction model F θ１_θ５４００ However, it is not transmitted to the client terminal 3620.

[0488] On the other hand, the client terminal 3620 receives time information (in the example of FIG. 43B, T １ ) and viewpoint information input by the user 440 (in the example of FIG. 43B, (θ ｘ , φ ｘ )) and the trained reconstruction model F θ１_θ５４００ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T １ view image at (for example, view image X １ ) is generated. Furthermore, in the client terminal 3620, the generated view image X １ A free viewpoint video with the frame images is played.

[0489] In addition, the client terminal 3620 receives the next time information (in the example of FIG. 43B, T ２ ) and viewpoint information input by the user 440 (in the example of FIG. 43B, (θ ｘ , φ ｘ )) and the trained reconstruction model F θ１_θ５４００ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T ２ view image at (for example, view image X ２ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ２ A free viewpoint video with the frame images is played.

[0490] Thereafter, the client terminal 3620 repeats the same process until the user 440 inputs the termination condition. In the example of FIG. 46B, the user 440 inputs the time information T １１ is input.

[0491] As a termination condition, time information T １１ When the time information T １１ and viewpoint information input by the user 440 (in the example of FIG. 46B, (θ ｘ , φ ｘ)) and the trained reconstruction model F θ１_θ５４００ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T １１ view image at (for example, view image X １１ ) is generated. Furthermore, in the client terminal 3620, the generated view image X １１ A free viewpoint video with the frame images is played.

[0492] <Flow of Free Viewpoint Video Playback Processing> Next, a flow of free viewpoint video playback processing by the free viewpoint video playback system 3600 according to the eighth embodiment will be described. Fig. 47 is a fifth sequence diagram showing the flow of free viewpoint video playback processing by the free viewpoint video playback system. Here, the description will focus on differences from the fourth sequence diagram shown in Fig. 44. The differences from the fourth sequence diagram shown in Fig. 44 are that, in the fifth sequence diagram of Fig. 47, the processing of step S4710_1 is included instead of the processing of step S4410_1, and the stop instructions accepted in steps S4120_4 and S4120_10 are not transmitted to the server device 3610.

[0493] In step S4710_1, the server device 3610 generates a trained reconstruction model F for generating a view image included in the specified free viewpoint video. θ１_θ５４００ The server device 3610 also reads out the trained reconstruction model F θ１_θ５４００ is transmitted to the client terminal 3620.

[0494] 47 , the stop instructions received in steps S4120_4 and S4120_10 are not sent to the server device 3610 because the server device 3610 does not have any new trained reconstruction models to send. That is, in step S4710_1, all trained reconstruction models that can be sent have already been sent to the client terminal 3620.

[0495] <Summary> As is clear from the above description, the one or more memories included in the server device 3610 according to the eighth embodiment hold a trained restoration model (third restoration model) for generating time-series view images of the first time interval.

[0496] Furthermore, the one or more processors included in the server device 3610 according to the eighth embodiment: Transmit the trained reconstruction model (third reconstruction model) in a transmission format that can be executed by the client terminal 3620.

[0497] As a result, according to the eighth embodiment, a mechanism for playing back free viewpoint video that is different from those of the sixth and seventh embodiments can be constructed.

[0498] [Ninth Embodiment] In the sixth embodiment, the model storage unit 606 has been described as holding one trained restoration model for each piece of time information, and one trained restoration model generates a view image for one piece of time information. However, the trained restoration model held by the model storage unit 606 for each piece of time information is not limited to this. For example, the model storage unit 606 may hold a trained differential restoration model that generates a difference image from a view image generated by a trained restoration model of the previous piece of time information. Hereinafter, the ninth embodiment will be described, focusing on differences from the sixth embodiment.

[0499] <Example of Trained Recovery Model> First, a trained recovery model stored in the model storage unit 606 in the server device 3610 according to the ninth embodiment will be described. Fig. 48 is a diagram showing an example of a trained recovery model stored in the model storage unit of the server device according to the ninth embodiment.

[0500] As shown in Fig. 48, the trained key recovery model and the trained differential recovery model stored in the model storage unit 606 are associated with time information. １ is a trained key recovery model F θ１ is associated with the time information T ２ ~T ３ is a trained differential restoration model ΔF θ１, ΔF θ２ Similarly, in the example of FIG. 48, the time information T ４ ~T ６ is a trained key recovery model F θ４ , the trained differential restoration model ΔF θ１ , ΔF θ２ In addition, the time information T ７ ~T ９ is a trained key recovery model F θ７ , the trained differential restoration model ΔF θ１ , ΔF θ２ is associated with the time information T １０ ~T １１ is a trained key recovery model F θ１０ , the trained differential restoration model ΔF θ１ The association between the time information and the trained key recovery model (or the trained differential recovery model) may be achieved by directly associating the time information with the trained key recovery model (or the trained differential recovery model) or by indirectly associating the time information with the trained key recovery model (or the trained differential recovery model) via other data.

[0501] Note that the trained key recovery model F shown in FIG. θ１ , F θ４ , F θ７ , F θ１０ is the trained key recovery model F shown in FIG. θ１ , F θ４ , F θ７ , F θ１０ 48. ２ , T ３ , T ５ , T ６ , T ８ , T ９ , T １１ The trained differential restoration model ΔF θ１ , ΔF θ２ is the corresponding trained differential restoration model ΔF θ１ , ΔF θ２ This is the same pre-trained differential restoration model as

[0502] <Specific Example of Processing by Server Apparatus> Next, a specific example of processing by the selection unit 3703 of the server apparatus 410 according to the ninth embodiment will be described.

[0503] (1) Specific Example 1 of Processing by the Selector Fig. 49A is a first diagram illustrating a specific example of processing by the server device according to the ninth embodiment. Fig. 49A illustrates a specific example of processing when the selector 3703 receives a notification of identification information of a specified free viewpoint video from the video specification accepting unit 3701 and a notification of time information included in the request from the request accepting unit 3702.

[0504] As shown in FIG. 49A , upon receiving the notification of the identification information of the specified free viewpoint video, the selection unit 3703 selects the following as a trained restoration model for generating a view image included in the specified free viewpoint video: θ１ , F θ４ , F θ７ , F θ１０ , Each time information T ２ , T ３ , T ５ , T ６ , T ８ , T ９ , T １１ The trained differential restoration model ΔF θ１ , ΔF θ２ , are read from the model storage unit 606.

[0505] The selection unit 3703 also selects the trained key recovery model and trained differential recovery model that match the time information T ３ As a trained key recovery model and a trained differential recovery model corresponding to θ１ , Time information T ２ , T ３ The trained differential restoration model ΔF θ１ , ΔF θ２ , and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the trained key recovery model F θ１ , time information T ２ , T ３ The trained differential restoration model ΔFθ１ , ΔF θ２ As a result, the client terminal 3620 receives the following: Default viewpoint information (θ ０ , φ ０ ) based on the trained key recovery model F θ１ is executed, and the default viewpoint information (θ ０ , φ ０ ) based on the time information T １ View image X in １ is generated, and the default viewpoint information (θ ０ , φ ０ ) based on the trained differential restoration model ΔF θ１ is executed, and the default viewpoint information (θ ０ , φ ０ ) based on the time information T ２ The difference image ΔX in １ is generated, and the generated difference image ΔX １ is the generated view image X １ is added to the view image X ２ is generated, and the default viewpoint information (θ ０ , φ ０ ) based on the trained differential restoration model ΔF θ２ is executed, and the default viewpoint information (θ ０ , φ ０ ) based on the time information T ３ The difference image ΔX in ２ is generated, and the generated difference image ΔX ２ is the generated view image X ２ is added to the view image X ３ is generated.

[0506] Furthermore, the client terminal 3620 generates the view image X ３ A free viewpoint video with the frame images is played.

[0507] Next, the selection unit 3703 selects the trained key recovery model F corresponding to the next time information (next time) as the next trained recovery model. θ４and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the trained key recovery model F θ４ As a result, the client terminal 3620 receives the default viewpoint information (θ ０ , φ ０ ) based on the trained key recovery model F θ４ is executed, and the default viewpoint information (θ ０ , φ ０ ) based on the time information T ４ View image X in ４ Furthermore, the client terminal 3620 generates the generated view image X ４ A free viewpoint video with the frame images is played.

[0508] Next, the selection unit 3703 selects a trained differential reconstruction model ΔF corresponding to the next time information (next time) as the next trained reconstruction model. θ１ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the trained differential restoration model ΔF θ１ As a result, the client terminal 3620 receives the following: Default viewpoint information (θ ０ , φ ０ ) based on the trained differential restoration model ΔF θ１ is executed, and the default viewpoint information (θ ０ , φ ０ ) based on the time information T ５ The difference image ΔX in ４ is generated, and the generated difference image ΔX ４ is the generated view image X ４ is added to the view image X ５ is generated.

[0509] Furthermore, the client terminal 3620 generates the view image X ５ A free viewpoint video with the frame images is played.

[0510] Next, the selection unit 3703 selects a trained differential reconstruction model ΔF corresponding to the next time information (next time) as the next trained reconstruction model. θ２ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the trained differential restoration model ΔF θ２ As a result, the client terminal 3620 receives the following: Default viewpoint information (θ ０ , φ ０ ) based on the trained differential restoration model ΔF θ２ is executed, and the default viewpoint information (θ ０ , φ ０ ) based on the time information T ６ The difference image ΔX in ５ is generated, and the generated difference image ΔX ５ is the generated view image X ５ is added to the view image X ６ is generated.

[0511] Furthermore, the client terminal 3620 generates the view image X ６ A free viewpoint video with the frame images is played.

[0512] Thereafter, the selection unit 3703 repeats the same process until an end condition is transmitted from the client terminal 3620. In the example of FIG. 49A, the client terminal 3620 transmits time information T １０ is shown to have been sent.

[0513] The client terminal 3620 receives time information T １０ is transmitted, the selection unit 3703 selects the time information T １０ A trained reconstruction model F corresponding to θ１０ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained reconstruction model F θ１０ As a result, the client terminal 3620 receives the default viewpoint information (θ ０ , φ ０) based on the trained key recovery model F θ１０ is executed, and the default viewpoint information (θ ０ , φ ０ ) based on the time information T １０ View image X in １０ Furthermore, the client terminal 3620 generates the generated view image X １０ A free viewpoint video with the frame images is played.

[0514] (2) Specific Example 2 of Processing by the Selector As described above, the time information T ３ From the time information T corresponding to the termination condition １０ A trained key recovery model F θ１ ~Trained key recovery model F θ１０ is transmitted to the client terminal 3620. As a result, the view image X ３ ~ View Image X １０ It is assumed that a free viewpoint video having frame images of is played back on the client terminal 3620. Furthermore, it is assumed that a request including time information is transmitted from the client terminal 3620, and viewpoint information is input by the user 440 at the client terminal 3620. In this case, the request receiving unit 3702 receives the request and notifies the selecting unit 3703.

[0515] Here, a specific example of processing by the selection unit 3703 when the request (time information) is notified from the request reception unit 3702 will be described. Fig. 49B is a second diagram showing a specific example of processing by the server device according to the ninth embodiment, and shows a specific example of processing by the selection unit 3703 when the request is notified from the request reception unit 3702.

[0516] As shown in FIG. 49B, the selection unit 3703 selects the time information included in the request (in the example of FIG. 49B, T １ ) corresponding to the trained key recovery model F θ１ The trained key recovery model F θ１ has already been transmitted to the client terminal 3620. Therefore, the selection unit 3703 transmits the trained key recovery model F θ１, and the trained key recovery model F θ１ However, it is not transmitted to the client terminal 3620.

[0517] On the other hand, the client terminal 3620 receives viewpoint information input by the user 440 (in the example of FIG. 49B, (θ ｘ , φ ｘ )) based on the trained key recovery model F θ１ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T １ view image at (for example, view image X １ ) is generated. Furthermore, in the client terminal 3620, the generated view image X １ A free viewpoint video with the frame images is played.

[0518] Next, the selection unit 3703 selects a trained differential reconstruction model ΔF corresponding to the next time information (next time) as the next trained reconstruction model. θ１ The trained differential restoration model ΔF θ１ has already been transmitted to the client terminal 3620. Therefore, the selection unit 3703 transmits the trained differential restoration model ΔF θ１ , and the trained differential restoration model ΔF θ１ However, it is not transmitted to the client terminal 3620.

[0519] On the other hand, the client terminal 3620 receives viewpoint information input by the user 440 (in the example of FIG. 49B, (θ ｘ , φ ｘ )) based on the trained differential restoration model ΔF θ１ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T ２ The difference image ΔX in １ In addition, the client terminal 3620 generates the generated view image X １ The difference image ΔX １By adding ｘ , φ ｘ ) based on the time information T ２ View image X in ２ Furthermore, the client terminal 3620 generates the generated view image X ２ A free viewpoint video with the frame images is played.

[0520] Next, the selection unit 3703 selects a trained differential reconstruction model ΔF corresponding to the next time information (next time) as the next trained reconstruction model. θ２ The trained differential restoration model ΔF θ２ has already been transmitted to the client terminal 3620. Therefore, the selection unit 3703 transmits the trained differential restoration model ΔF θ２ , and the trained differential restoration model ΔF θ２ However, it is not transmitted to the client terminal 3620.

[0521] On the other hand, the client terminal 3620 receives viewpoint information input by the user 440 (in the example of FIG. 49B, (θ ｘ , φ ｘ )) based on the trained differential restoration model ΔF θ２ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T ３ The difference image ΔX in ２ In addition, the client terminal 3620 generates the generated view image X ２ The difference image ΔX ２ By adding ｘ , φ ｘ ) based on the time information T ３ View image X in ３ Furthermore, the client terminal 3620 generates the generated view image X ３ A free viewpoint video with the frame images is played.

[0522] Thereafter, the selection unit 3703 repeats the same process until an end condition is transmitted from the client terminal 3620. In the example of FIG. 49B, the client terminal 3620 transmits time information T １１ is shown to have been sent.

[0523] The client terminal 3620 receives time information T １１ is transmitted, the selection unit 3703 selects the time information T １１ The trained differential restoration model ΔF corresponding to θ１ The selection unit 3703 also selects the identified trained differential restoration model ΔF θ１ to the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the trained differential restoration model ΔF θ１ As a result, the client terminal 3620 receives the viewpoint information (θ ｘ , φ ｘ ) based on the trained differential restoration model ΔF θ１ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T １１ The difference image ΔX in １０ In addition, the client terminal 3620 generates the generated view image X １０ The difference image ΔX １０ By adding ｘ , φ ｘ ) based on the time information T １１ View image X in １１ Furthermore, the client terminal 3620 generates the generated view image X １１ A free viewpoint video with the frame images is played.

[0524] <Flow of Free Viewpoint Video Playback Processing> Next, a flow of free viewpoint video playback processing by the free viewpoint video playback system 3600 according to the ninth embodiment will be described. Fig. 50 is a sixth sequence diagram showing the flow of free viewpoint video playback processing by the free viewpoint video playback system. Here, the description will focus on differences from the third sequence diagram shown in Fig. 41. The differences from the third sequence diagram shown in Fig. 41 are that, in the sixth sequence diagram of Fig. 50, the processing of step S5010_1 is included instead of the processing of step S4110_1, and the processing of step S4110_2 is not included, and the processing of step S5010_2 is included instead of the processing of step S4110_3.

[0525] In step S5010_1, the server device 3610 reads out a group of trained key recovery models and trained differential recovery models for generating view images included in the specified free viewpoint video. In addition ... ３ As a trained key recovery model and a trained differential recovery model associated with θ１ , the trained differential restoration model ΔF θ１ , ΔF θ２ Furthermore, the server device 3610 sequentially transmits the identified trained key recovery model and trained differential recovery model to the client terminal 3620. The server device 3610 identifies the trained key recovery model F θ１０ After transmitting the trained key recovery model and the trained differential recovery model to the client terminal 3620, transmission of the trained key recovery model and the trained differential recovery model is stopped.

[0526] The sixth sequence diagram of FIG. 50 does not include the processing of step S4110_2 for the following reason.

[0527] That is, the time information T １ , T ２ a trained key recovery model F to generate a view image corresponding to θ１ and the trained differential restoration model ΔF θ１has already been transmitted to the client terminal 3620 in step S5010_1. Therefore, the selection unit 3703 transmits the trained key recovery model F θ１ and the trained differential restoration model ΔF θ１ In addition, the trained key recovery model F θ１ and the trained differential restoration model ΔF θ１ However, it is not transmitted to the client terminal 3620.

[0528] In step S5010_2, the server device 3610 selects the time information T １１ The trained differential restoration model ΔF corresponding to θ１ and identify the trained differential restoration model ΔF θ１ is transmitted to the client terminal 3620.

[0529] <Summary> As is clear from the above description, the one or more memories included in the server device 3610 according to the ninth embodiment: Hold a trained key recovery model (fourth recovery model) of a time series of a third time interval for generating time series view images of a third time interval that is longer than the first time interval. Hold a trained differential recovery model (fourth differential recovery model) of a time series of a first time interval for generating time series view images of a first time interval, the trained differential recovery model (fourth differential recovery model) for generating difference images representing differences from view images generated just the first time interval before.

[0530] Furthermore, one or more processors included in the server device 3610 according to the ninth embodiment: transmit, in a transmission format executable on the client terminal 3620, a trained key recovery model (fourth recovery model) of a time series of a third time interval from a trained key recovery model (fourth recovery model) corresponding to the time information included in the request to a trained key recovery model (fourth recovery model) corresponding to a predetermined termination condition. Transmit, in a transmission format executable on the client terminal 3620, a trained differential recovery model (fourth differential recovery model) of a time series of a first time interval from a trained differential recovery model (fourth differential recovery model) corresponding to the time information included in the request to a trained differential recovery model (fourth differential recovery model) corresponding to the predetermined termination condition.

[0531] As a result, according to the ninth embodiment, a mechanism for playing back free viewpoint video that is different from those of the sixth to eighth embodiments can be constructed.

[0532] [Tenth Embodiment] In the sixth embodiment, a case has been described in which one imaging device captures the three-dimensional scene 140 from the same viewpoint. However, for example, two imaging devices may capture the three-dimensional scene 140 from the same viewpoint. This makes it possible to divide the three-dimensional scene 140 into two spaces and generate trained restoration models that generate view images in each space. The tenth embodiment will be described below, focusing on the differences from the sixth embodiment.

[0533] <Functional Configuration of Server Device> First, the functional configuration of the server device 3610 according to the tenth embodiment will be described. Fig. 51 is a third diagram showing an example of the functional configuration of the server device. The difference from the functional configuration shown in Fig. 37 is that, in the case of Fig. 51, a request receiving unit 5111 included in a restoration model providing unit 5110 of the server device 3610 has a different function from the request receiving unit 3702 included in the restoration model providing unit 3611 of the server device 3610 shown in Fig. 37.

[0534] Specifically, the request receiving unit 5111 included in the restoration model providing unit 5110 of the server device 3610 receives a request including time information and spatial information. In addition, the request receiving unit 5111 included in the restoration model providing unit 5110 of the server device 3610 notifies the selection unit 3703 of the time information and spatial information.

[0535] <Functional Configuration of Client Terminal> Next, the functional configuration of the client terminal 3620 will be described. Fig. 52 is a third diagram showing an example of the functional configuration of a client terminal. The difference from Fig. 40 is that in the case of Fig. 52, a video designation transmission unit 5211, a video display unit 5212, and a request transmission unit 5213 included in a free viewpoint video playback unit 5210 have functions different from the corresponding functional units of the client terminal 3620 shown in Fig. 40.

[0536] Specifically, a video designation transmission unit 5211 and a video display unit 5212 included in a free viewpoint video playback unit 5210 of the client terminal 3620 accept input of spatial information in addition to time information. Also, a request transmission unit 5213 is notified of a request including time information and spatial information from the video designation transmission unit 5211 or the video display unit 5212, and transmits the request to the server device 3610.

[0537] <Example of Trained Recovery Model> Next, a trained recovery model stored in the model storage unit 606 in the server device 3610 according to the tenth embodiment will be described. Fig. 53 is a diagram showing an example of a trained recovery model stored in the model storage unit of the server device according to the tenth embodiment.

[0538] As shown in FIG. 53, the trained reconstruction model stored in the model storage unit 606 is associated with time information. Specifically, the time information T １ The trained space-reconstruction model F θ１ and the trained spatial 2 reconstruction model F θ１ is associated with the time information T ２ The trained space-reconstruction model F θ２ and the trained spatial 2 reconstruction model F θ２ Similarly, in the example of FIG. 53, the time information T ３ ~T １１Then, the trained space 1 reconstruction model F θ３ and the trained spatial 2 reconstruction model F θ３ ~Trained space 1 reconstruction model F θ１１ and the trained spatial 2 reconstruction model F θ１１ The correspondence between the time information and the trained space 1 reconstruction model and the trained space 2 reconstruction model may be performed by directly associating the time information with the trained space 1 reconstruction model and the trained space 2 reconstruction model, or may be performed by indirectly associating the time information with the trained space 1 reconstruction model and the trained space 2 reconstruction model via other data.

[0539] Note that the trained space 1 reconstruction model F shown in FIG. θ１ ~F θ１１ and the trained spatial 2 reconstruction model F θ１ ~F θ１１ is the space 1 reconstruction model F shown in FIG. θ１ ~F θ１１ and the trained spatial 2 reconstruction model F θ１ ~F θ１１ is the same as

[0540] <Specific Example of Processing by Server Apparatus> Next, a specific example of processing by the selection unit 3703 of the server apparatus 3610 according to the tenth embodiment will be described.

[0541] (1) Specific Example 1 of Processing by Selector Fig. 54A is a first diagram showing a specific example of processing by the server device according to the tenth embodiment. Fig. 54A shows a process in which the selector 3703 receives a notification of identification information of a specified free viewpoint video from the video designation accepting unit 3701, and receives time information T ３ This shows a specific example of the process when a notification is received.

[0542] As shown in FIG. 54A , upon receiving the notification of the identification information of the specified free viewpoint video, the selection unit 3703 selects the following as a trained restoration model for generating a view image included in the specified free viewpoint video: θ１ ~F θ１１ , and a trained spatial 2 reconstruction model F θ１ ~F θ１１, are read from the model storage unit 606.

[0543] The selection unit 3703 also selects the trained reconstruction model that has the time information T ３ and a trained space 1 reconstruction model F corresponding to the default spatial information (space 1, space 2). θ３ and the trained spatial 2 reconstruction model F θ３ Identify.

[0544] The selection unit 3703 also selects the identified trained space 1 reconstruction model F θ３ and the trained spatial 2 reconstruction model F θ３ to the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the notified trained space 1 reconstruction model F θ３ and the trained spatial 2 reconstruction model F θ３ As a result, the client terminal 3620 receives the default viewpoint information (in the example of FIG. 54A , (θ ０ , φ ０ )) based on the trained spatial 1 reconstruction model F θ３ In addition, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) based on the time information T ３ view image at (for example, view image X ３_１ Similarly, the client terminal 3620 generates default viewpoint information (in the example of FIG. 54A , (θ ０ , φ ０ )) based on the trained spatial 2 reconstruction model F θ３ In addition, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) based on the time information T ３ view image at (for example, view image X ３_２ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ３_１ , X ３_２ A free viewpoint video with the frame images is played.

[0545] Next, the selection unit 3703 selects the trained space 1 reconstruction model F corresponding to the next time information (next time) as the next trained reconstruction model. θ４ and the trained spatial 2 reconstruction model F θ４ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained space 1 reconstruction model F θ４ and the trained spatial 2 reconstruction model F θ４ As a result, the client terminal 3620 receives the default viewpoint information (in the example of FIG. 54A , (θ ０ , φ ０ )) based on the trained spatial 1 reconstruction model F θ４ In addition, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) based on the time information T ４ view image at (for example, view image X ４_１ Similarly, the client terminal 3620 generates default viewpoint information (in the example of FIG. 54A , (θ ０ , φ ０ )) based on the trained spatial 2 reconstruction model F θ４ In addition, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) based on the time information T ４ view image at (for example, view image X ４_２ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ４_１ , X ４_２ A free viewpoint video with the frame images is played.

[0546] Thereafter, the selection unit 3703 repeats the same process until an end condition is transmitted from the client terminal 3620. In the example of FIG. 54A, the client terminal 3620 transmits time information T １０ is shown to have been sent.

[0547] The client terminal 3620 receives time information T １０is transmitted, the selection unit 3703 selects the time information T １０ A trained space-1 reconstruction model F corresponding to θ１０ and the trained spatial 2 reconstruction model F θ１０ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained space 1 reconstruction model F θ１０ and the trained spatial 2 reconstruction model F θ１０ As a result, the client terminal 3620 receives the default viewpoint information (θ ０ , φ ０ ) based on the trained spatial 1 reconstruction model F θ１０ In addition, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) based on the time information T １０ view image at (for example, view image X １０_１ ) is generated.

[0548] Similarly, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) based on the trained spatial 2 reconstruction model F θ１０ In addition, the client terminal 3620 uses the default viewpoint information (θ ０ , φ ０ ) based on the time information T １０ view image at (for example, view image X １０_２ ) is generated.

[0549] Furthermore, the client terminal 3620 generates the view image X １０_１ , view image X １０_２ A free viewpoint video with the frame images is played.

[0550] (2) Specific Example 2 of Processing by the Selector As described above, the time information T ３ From the time information T corresponding to the termination condition １０ As a trained reconstruction model up to θ３~Trained space 1 reconstruction model F θ１０ , ・Trained space 2 reconstruction model F θ３ ~Trained space 2 reconstruction model F θ１０ , are transmitted to the client terminal 3620. As a result, the view image X ３_１ ~ View Image X １０_１ and view image X ３_２ ~ View Image X １０_２ It is assumed that a free viewpoint video having frame images of is played back on the client terminal 3620. Furthermore, it is assumed that a request including time information and spatial information is transmitted from the client terminal 3620, and viewpoint information is input by the user 440 to the client terminal 3620. In this case, the request receiving unit 3702 receives the request and notifies the selecting unit 3703.

[0551] Here, a specific example of processing by the selection unit 3703 when a request (time information, space information) is notified from the request reception unit 3702 will be described. Fig. 54B is a second diagram showing a specific example of processing by the server device according to the tenth embodiment, and shows a specific example of processing by the selection unit 3703 when the request is notified from the request reception unit 3702.

[0552] As shown in FIG. 54B , the selection unit 3703 selects a trained reconstruction model from among the trained reconstruction models that have already been read out, based on the time information included in the request (in the example of FIG. 54B , T １ ) and the trained space 1 reconstruction model F corresponding to the spatial information (space 1 in the example of FIG. 54B). θ１ Identify.

[0553] The selection unit 3703 also selects the identified trained space 1 reconstruction model F θ１ to the model transmitting unit 3704. As a result, the model transmitting unit 3704 transmits the notified trained space 1 reconstruction model F θ１ As a result, the client terminal 3620 receives the viewpoint information (θ ｘ , φ ｘ ) based on the trained spatial 1 reconstruction model F θ１ In addition, the client terminal 3620 receives viewpoint information (θｘ , φ ｘ ) based on the time information T １ view image at (for example, view image X １_１ ) is generated. Furthermore, in the client terminal 3620, the generated view image X １_１ A free viewpoint video with the frame images is played.

[0554] Next, the selection unit 3703 selects the trained space 1 reconstruction model F corresponding to the next time information (next time) as the next trained reconstruction model. θ２ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained space 1 reconstruction model F θ２ As a result, the client terminal 3620 receives the viewpoint information (θ ｘ , φ ｘ ) based on the trained spatial 1 reconstruction model F θ２ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T ２ view image at (for example, view image X ２_１ ) is generated. Furthermore, in the client terminal 3620, the generated view image X ２_１ A free viewpoint video with the frame images is played.

[0555] Next, the selection unit 3703 selects the trained space 1 reconstruction model F corresponding to the next time information (next time) as the next trained reconstruction model. θ３ The trained space 1 reconstruction model F θ３ has already been transmitted to the client terminal 3620. Therefore, the selection unit 3703 transmits the trained space 1 reconstruction model F θ３ , and the trained spatial 1 reconstruction model F θ３ However, it is not transmitted to the client terminal 3620.

[0556] Thereafter, the selection unit 3703 repeats the same process until an end condition is transmitted from the client terminal 3620. In the example of FIG. 54B, the client terminal 3620 transmits time information T １１ is shown to have been sent.

[0557] The client terminal 3620 receives time information T １１ is transmitted, the selection unit 3703 selects the time information T １１ A trained space-1 reconstruction model F corresponding to θ１１ and notifies the model transmitting unit 3704. As a result, the model transmitting unit 3704 identifies the notified trained space 1 reconstruction model F θ１１ As a result, the client terminal 3620 receives the viewpoint information (θ ｘ , φ ｘ ) based on the trained spatial 1 reconstruction model F θ１１ In addition, the client terminal 3620 receives viewpoint information (θ ｘ , φ ｘ ) based on the time information T １１ view image at (for example, view image X １１_１ ) is generated. Furthermore, in the client terminal 3620, the generated view image X １１_１ A free viewpoint video with the frame images is played.

[0558] <Flow of Free Viewpoint Video Playback Processing> Next, we will explain the flow of free viewpoint video playback processing by the free viewpoint video playback system 3600. Fig. 55 is a seventh sequence diagram showing the flow of free viewpoint video playback processing by the free viewpoint video playback system.

[0559] In step S4120_1, the client terminal 3620 receives a designation from the user 440 regarding the free viewpoint video to be displayed, and transmits, to the server device 3610, identification information for uniquely identifying the designated free viewpoint video.

[0560] In step S4120_2, the client terminal 3620 receives the time information T３ The input of time information T ３ to the server device 3610.

[0561] In step S5510_1, the server device 3610 reads a group of trained space 1 reconstruction models and a group of trained space 2 reconstruction models for generating view images included in the specified free viewpoint video. ３ and a trained space 1 reconstruction model F corresponding to the default spatial information (space 1, space 2). θ３ and the trained spatial 2 reconstruction model F θ３ , and then sequentially transmits the data to the client terminal 3620.

[0562] In step S5520_3, the client terminal 3620 receives the trained space 1 reconstruction model and the trained space 2 reconstruction model sequentially transmitted from the server device 3610, and adds default viewpoint information (θ ０ , φ ０ ) is input. As a result, the client terminal 3620 inputs the time information T ３ The default spatial information (space 1, space 2) and the default viewpoint information (θ ０ , φ ０ ) view images are generated sequentially.

[0563] In step S4120_4, the client terminal 3620 receives the stop instruction and transmits the received stop instruction to the server device 3610. As a result, the server device 3610 transmits the trained space 1 reconstruction model F θ１０ and the trained spatial 2 reconstruction model F θ１０ After transmitting the trained reconstruction model to the client terminal 3620 until the time information T ３ , default spatial information (space 1, space 2) and viewpoint information (θ ０ , φ ０ ) as a view image according to the view image X ３_１ ~X １０_１, View image X ３_２ ~X １０_２ , and can play free viewpoint videos with frame images.

[0564] In step S4120_5, the client terminal 3620 accepts an instruction to move the indicator 1112' on the seek bar 1112. The client terminal 3620 sequentially transmits to the server device 3610 time information on each position of the indicator 1112' that is being moved.

[0565] In step S5510_2, the server device 3610 transmits the trained space 1 reconstruction model and the trained space 2 reconstruction model corresponding to the time information of each position to the client terminal 3620 every time it receives time information of each position of the moving indicator 1112' from the client terminal 3620. At this time, the server device 3610 does not transmit the trained space 1 reconstruction model and the trained space 2 reconstruction model that have already been transmitted to the client terminal 3620, but transmits the trained space 1 reconstruction model and the trained space 2 reconstruction model that have not been transmitted to the client terminal 3620. In the example of FIG. 55 , the indicator 1112' of the seek bar 1112 is moved to the position indicated by the time information T １ Since the server device 3610 has been moved to the position of θ２ , F θ１ and the trained spatial 2 reconstruction model F θ２ , F θ１ is transmitted to the client terminal 3620.

[0566] In step S5520_7, the client terminal 3620 accepts input of space information (space 1).

[0567] In step S4120_7, the client terminal 3620 receives the viewpoint information (θ ｘ , φ ｘ ) input is accepted.

[0568] At step S 4120 _ 8 , when the play button 1114 is pressed, the client terminal 3620 transmits a play instruction to the server device 3610 .

[0569] In step S5510_3, the server device 3610 receives the time information T １ and a trained space 1 reconstruction model F associated with the spatial information (space 1) θ１ , sequentially transmit the trained reconstruction models to the client terminal 3620. However, the server device 3610 does not transmit the trained reconstruction models that have already been transmitted to the client terminal 3620, but transmits the trained reconstruction models that have not been transmitted to the client terminal 3620.

[0570] In step S5520_9, the client terminal 3620 adds viewpoint information (θ ｘ , φ ｘ ) is input. As a result, the client terminal 3620 inputs the time information T １ , a view image according to the space information (space 1), and the input viewpoint information (θ ｘ , φ ｘ ) view images are generated sequentially.

[0571] In step S4120_10, the client terminal 3620 receives a stop instruction and transmits the received stop instruction to the server device 3610. As a result, the server device 3610 transmits the trained space 1 reconstruction model F θ１１ After transmitting the trained reconstruction model to the client terminal 3620 until the time information T １ and a view image according to the space information (space 1), which is input viewpoint information (θ ｘ , φ ｘ ) view image X １_１ ~X １１_１ A free viewpoint video with the frame images is played.

[0572] <Summary> As is clear from the above description, the server device 3610 according to the tenth embodiment includes one or more memories and one or more processors, and the one or more memories hold: a trained space 1 restoration model or a trained space 2 restoration model (first restoration model) for generating a time series of view images of a first time interval for a specific space, the trained space 1 restoration model or the trained space 2 restoration model (first restoration model) of the time series of the first time interval.

[0573] Furthermore, one or more processors included in the server device 3610 according to the tenth embodiment: Transmit, in a transmission format executable on the client terminal 3620, a time series trained space 1 reconstruction model or trained space 2 reconstruction model (first reconstruction model) for a first time interval from a trained space 1 reconstruction model or trained space 2 reconstruction model (first reconstruction model) corresponding to the time information included in the request to a trained space 1 reconstruction model or trained space 2 reconstruction model (first reconstruction model) corresponding to a predetermined termination condition. The time series trained space 1 reconstruction model or trained space 2 reconstruction model (first reconstruction model) for the first time interval is a trained reconstruction model corresponding to the spatial information included in the request.

[0574] In this way, according to the tenth embodiment, it is possible to construct a mechanism for playing back free viewpoint video about a specific space.

[0575] Eleventh Embodiment In the first to fifth embodiments described above, the server device 410 is configured to generate view images in real time when playing back free viewpoint video on the client terminal 420. However, the timing of generating view images by the server device 410 is not limited to this. For example, while playing back free viewpoint video on the client terminal 420, the server device 410 may be configured to generate in advance view images corresponding to time information ahead of the current time information.

[0576] Furthermore, in the first to fifth embodiments, a view image corresponding to the position of the indicator or the position of the mouse pointer when the video display area is dragged is generated while the indicator of the seek bar is being moved on the client terminal 420, or while the video display area is being dragged with the mouse pointer. However, the timing of generating a view image by the server device 410 is not limited to this. For example, a configuration may be adopted in which a destination position is predicted according to the movement direction of the indicator on the client terminal 420 or the movement direction of the dragged video display area, and a view image corresponding to the predicted position is generated in advance.

[0577] Similarly, in the sixth to tenth embodiments, the server device 3610 is configured to transmit a trained restoration model in real time when the client terminal 3620 plays back a free viewpoint video. However, the timing of transmission of the trained restoration model by the server device 3610 is not limited to this. For example, while the client terminal 3620 is playing back a free viewpoint video, the server device 3610 may be configured to transmit in advance a trained restoration model corresponding to time information ahead of the current time information. Alternatively, the server device 3610 may be configured to transmit a trained restoration model corresponding to time information before and after the requested time information.

[0578] In the sixth to tenth embodiments, a trained restoration model corresponding to the position of the indicator of the seek bar is transmitted while the indicator is being moved on the client terminal 3620. However, the timing of transmitting the trained restoration model by the server device 3610 is not limited to this. For example, the server device 3610 may be configured to predict the destination position according to the movement direction of the indicator on the client terminal 3620, and transmit in advance a trained restoration model corresponding to the predicted position.

[0579] In the above first to tenth embodiments, a view image of a certain viewpoint is generated by performing volume rendering processing on a combination of color and opacity output from a restoration model. However, the method of generating the view image is not limited to this. For example, a feature image may be generated by performing volume rendering processing on a feature vector output from the restoration model, and an RGB image may be generated from the generated feature image using a multilayer perceptron (MLP), a convolutional neural network (CNN), or the like, to obtain the view image.

[0580] In the fifth embodiment, the three-dimensional scene 140 is photographed from the same viewpoint using two image capturing devices, and the three-dimensional scene 140 is divided into two spaces, thereby generating trained restoration models that generate view images in each space. However, the method of dividing the space is not limited to this. For example, the space may be divided into a background region and a region excluding the background, and a trained restoration model that generates a view image in the background region and a trained restoration model that generates a view image in the region excluding the background may be generated.

[0581] Furthermore, although the fifth embodiment is shown as a modification of the first embodiment, it may be a modification of any of the second to fourth embodiments. Similarly, although the tenth embodiment is shown as a modification of the sixth embodiment, it may be a modification of any of the seventh to ninth embodiments.

[0582] In addition, in each of the above embodiments, a system using a reconstruction model that has been trained in advance by the NeRF technology has been described. However, a system using a reconstruction model that can generate new viewpoints or a composite system may also be used.

[0583] For example, instead of the NeRF technique, a system may be used that uses a restoration model that has been trained in advance using the 3D Gaussian Splatting technique.

[0584] For example, the system may use an image generation model that has been trained in advance using image-based rendering techniques or transformer techniques that do not involve explicit reconstruction of a three-dimensional scene.

[0585] [Other Embodiments] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, a-b, a-c, bc, or a-bc. It may also include multiple instances of any element, such as a-a, a-bb-b, a-a-bb-cc-c, etc. Furthermore, it also includes adding elements other than the enumerated elements (a, b, and c), such as having d, as in a-b-c-d.

[0586] Furthermore, in this specification (including claims), when expressions such as "using data as input / based on / according to / in response to" (including similar expressions) are used, unless otherwise specified, this includes cases where various data itself is used as input, or where various data that has been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) is used as input. Furthermore, when it is stated that a result is obtained "based on / according to / in response to data," this includes cases where the result is obtained based solely on the data in question, as well as cases where the result is obtained in response to other data, factors, conditions, and / or states other than the data in question. Furthermore, when it is stated that "data is output," unless otherwise specified, this includes cases where various data itself is used as output, or where various data that has been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) is output.

[0587] Furthermore, when the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that include any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, and physically connection / coupling. These terms should be interpreted appropriately depending on the context in which they are used, but any connection / coupling form that is not intentionally or naturally excluded should be interpreted as being included in these terms without any restrictions.

[0588] Furthermore, in this specification (including the claims), when the expression "A configured to B" is used, it may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting / configuration of element A being configured / set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Furthermore, if element A is a dedicated processor or dedicated arithmetic circuit, it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0589] Furthermore, when words implying containing or possessing (e.g., "comprising / including" and "having") are used in this specification (including the claims), they are intended to be open-ended terms that include cases where something other than the object indicated by the object of the term is contained or possessed. When the object of such words implying containing or possessing does not specify a quantity or suggests a singular number (e.g., an expression using the article "a" or "an"), the expression should be construed as not being limited to a specific number.

[0590] Furthermore, although expressions such as "one or more" or "at least one" are used in some places in this specification (including the claims) and expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

[0591] Furthermore, if a particular advantage / result is described in this specification as being obtained with respect to a particular configuration of an embodiment, it should be understood that the same advantage / result can also be obtained with one or more other embodiments having the same configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, etc., and that the effect is not necessarily obtained with the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or states, etc. are satisfied, and the effect does not necessarily occur in a claimed invention that defines the same or a similar configuration.

[0592] Furthermore, in this specification (including claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may cooperate to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that hardware may include an electronic circuit, a device including an electronic circuit, etc.

[0593] Furthermore, in this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices (memories) may store only a portion of the data, or may store the entire data.

[0594] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, partial deletions, etc. are possible within the scope of the conceptual idea and spirit of the present invention derived from the content defined in the claims and their equivalents. For example, in all of the above-described embodiments, the numerical values ​​used in the explanations are shown as examples and are not limited to these. Furthermore, the order of each operation in the embodiments is shown as an example and is not limited to these.

[0595] The disclosed technology may take the form of the following supplementary notes: (Supplementary Note 1) A server device comprising: one or more memories; and one or more processors, wherein the one or more memories hold one or more restoration models for generating time-series free-viewpoint images, the one or more restoration models being pre-trained so as to be able to restore a scene from a first time to a second time using time-series captured images from a plurality of viewpoints obtained by capturing a scene continuously over time from each of the plurality of viewpoints, and the one or more processors receive from a client a request including viewpoint information and time information for the scene, and use the one or more restoration models to generate time-series images corresponding to the viewpoint information and the time information included in the request received from the client, and transmit the images in a transmission format allowing video playback by the client. (Supplementary Note 2) The server device according to Supplementary Note 1, wherein the one or more processors generate time-series images corresponding to the viewpoint information using one or more restoration models ranging from a restoration model corresponding to the time information included in the request from the client to a restoration model corresponding to a predetermined termination condition. (Supplementary Note 3) The server device according to Supplementary Note 2, wherein the one or more memories hold a first restoration model of a time series of a first time interval for generating time-series free-viewpoint images of the first time interval, and the one or more processors generate time-series images of the first time interval according to the viewpoint information using a first restoration model of the time series of the first time interval from a first restoration model corresponding to the time information to a first restoration model corresponding to a predetermined termination condition. (Supplementary Note 4) The server device according to Supplementary Note 2, wherein the one or more memories hold a second restoration model of a time series of a second time interval longer than the first time interval for generating time-series free-viewpoint images of the first time interval, and the one or more processors generate time-series images of the first time interval according to the viewpoint information using a second restoration model of the time series of the second time interval from the second restoration model corresponding to the time information to a second restoration model corresponding to a predetermined termination condition.(Supplementary Note 5) The server device according to Supplementary Note 2, wherein the one or more memories hold a third restoration model for generating time-series free viewpoint images of a first time interval, and the one or more processors use the third restoration model to generate time-series images of the first time interval according to the viewpoint information from the time information until a predetermined termination condition is met. (Supplementary Note 6) The server device according to Supplementary Note 1, wherein the request includes spatial information, and the one or more processors use a restoration model of the time series from the restoration model corresponding to the time information to a restoration model corresponding to a predetermined termination condition, the restoration model corresponding to the spatial information, to generate time-series images according to the viewpoint information. (Supplementary Note 7) The server device according to Supplementary Note 6, wherein the space specified by the spatial information is a predetermined area within the space or an area within the space excluding a background. (Supplementary Note 8) The server device according to any of Supplements 1 to 7, wherein the one or more processors, when a video is designated by the client, generate time-series images according to default viewpoint information using restoration models corresponding to the designated video, the restoration models being time-series restoration models ranging from a restoration model corresponding to default time information to a restoration model corresponding to a predetermined termination condition, and transmit the generated images in a transmission format enabling video playback by the client, and receive the request from the client in response to having transmitted the time-series images in a transmission format enabling video playback by the client. (Supplementary Note 9) The server device according to Supplementary Note 8, wherein the one or more processors, each time a request including time information is transmitted from the stopped client, generate an image according to the viewpoint information using a restoration model corresponding to the time information included in the transmitted request, and each time a request including viewpoint information is transmitted from the stopped client, generate an image according to the viewpoint information included in the transmitted request.(Supplementary Note 10) The server device according to Supplementary Note 9, wherein the one or more processors, when a request including time information based on a video playback instruction is transmitted from the stopped client, start a process of generating time-series images according to the viewpoint information from a restoration model corresponding to the time information included in the transmitted request, and when a request including time information based on a video stop instruction is transmitted from the client during playback, stop the process of generating time-series images according to the viewpoint information included in the transmitted request, with the restoration model corresponding to the time information included in the transmitted request as the last one. (Supplementary Note 11) The server device according to any of Supplements 1 to 10, wherein the...

Claims

1. A server device comprising: one or more memories; and one or more processors; wherein the one or more memories hold one or more restoration models for generating time-series free viewpoint images, the one or more restoration models being pre-trained to be capable of restoring a scene from a first time to a second time using time-series captured images from multiple viewpoints obtained by shooting a scene from each of the multiple viewpoints in succession over time; and the one or more processors receive from a client a request including viewpoint information and time information for the scene; and use the one or more restoration models to generate time-series images according to the viewpoint information and the time information included in the request received from the client, and transmit the images in a transmission format allowing video playback by the client.

2. The server device according to claim 1, wherein the one or more processors generate a time-series image according to the viewpoint information using one or more restoration models ranging from a restoration model corresponding to the time information included in the request from the client to a restoration model corresponding to a predetermined termination condition.

3. The server device according to claim 2, wherein the one or more memories hold a first restoration model of a time series of a first time interval for generating a time series of free viewpoint images of the first time interval, and the one or more processors generate images of the time series of the first time interval according to the viewpoint information using a first restoration model of the time series of the first time interval from a first restoration model corresponding to the time information to a first restoration model corresponding to a predetermined termination condition.

4. The server device according to claim 2, wherein the one or more memories hold a second restoration model for generating a time series of free viewpoint images of a first time interval, the second restoration model being a time series of a second time interval longer than the first time interval, and the one or more processors generate a time series of images of the first time interval according to the viewpoint information using the second restoration model of the time series of the second time interval from the second restoration model corresponding to the time information to a second restoration model corresponding to a predetermined termination condition.

5. The server device according to claim 2, wherein the one or more memories hold a third restoration model for generating time-series free viewpoint images of a first time interval, and the one or more processors use the third restoration model to generate, from the time information, time-series images of the first time interval according to the viewpoint information up to a predetermined end condition.

6. The server device according to claim 1, wherein the request includes spatial information, and the one or more processors generate a time-series image according to the viewpoint information using a restoration model corresponding to the spatial information, the restoration model being a time series from a restoration model corresponding to the time information to a restoration model corresponding to a predetermined termination condition.

7. The server device according to claim 6, wherein the space specified by the spatial information is a predetermined area within the space, or an area within the space excluding the background.

8. The server device according to claim 1, wherein the one or more processors, when a video is specified by the client, generate time-series images according to default viewpoint information using a time-series restoration model corresponding to the specified video, the time-series restoration model ranging from a restoration model corresponding to default time information to a restoration model corresponding to a specified termination condition, and transmit the time-series images in a transmission format enabling video playback by the client; and receive the request from the client in response to transmitting the time-series images in a transmission format enabling video playback by the client.

9. The server device according to claim 8, wherein the one or more processors: each time a request including time information is sent from the stopped client, generate an image corresponding to the viewpoint information using a restoration model corresponding to the time information included in the sent request; and each time a request including viewpoint information is sent from the stopped client, generate an image corresponding to the viewpoint information included in the sent request.

10. The server device according to claim 9, wherein the one or more processors, when a request including time information based on an instruction to play a video is transmitted from the stopped client, start a process of generating time-series images according to the viewpoint information from a restoration model corresponding to the time information contained in the transmitted request, and when a request including time information based on an instruction to stop playing a video is transmitted from the client during playback, stop the process of generating time-series images according to the viewpoint information contained in the transmitted request, ending with the restoration model corresponding to the time information contained in the transmitted request.

11. The server device according to claim 1, wherein the one or more processors generate the time-series images at time intervals according to the frame period and / or display mode when the client plays back a video, the communication load between the client, or the processing load when generating the time-series images.

12. The server device according to claim 1, wherein the one or more processors generate an image predicted based on an operation on the client using the restoration model.

13. A server device comprising: one or more memories; and one or more processors, wherein the one or more memories hold one or more restoration models for generating time-series free viewpoint images, the one or more restoration models being pre-trained to be able to restore a scene from a first time to a second time using time-series captured images from multiple viewpoints obtained by shooting a scene continuously over time from each of the multiple viewpoints, and the one or more processors receive from a client a request including time information for the scene, and transmit one or more restoration models corresponding to the time information included in the request received from the client in a transmission format executable by the client, thereby causing the client to play a free viewpoint video using the time-series images corresponding to the viewpoint information generated using the one or more restoration models as frame images.

14. The server device according to claim 13, wherein the one or more processors transmit a time-series restoration model from a restoration model corresponding to the time information included in the request from the client to a restoration model corresponding to a predetermined termination condition in a transmission format executable by the client.

15. The server device according to claim 14, wherein the one or more memories hold a first restoration model of a time series of a first time interval for generating a time series of free viewpoint images of the first time interval, and the one or more processors transmit the first restoration models of the time series of the first time interval from the first restoration model corresponding to the time information to the first restoration model corresponding to a predetermined termination condition in a transmission format executable by the client.

16. The server device according to claim 14, wherein the one or more memories hold a second restoration model for generating a time series of free viewpoint images of a first time interval, the second restoration model being a time series of a second time interval longer than the first time interval, and the one or more processors transmit the second restoration models of the time series of the second time interval from the second restoration model corresponding to the time information to the second restoration model corresponding to a predetermined termination condition in a transmission format executable by the client.

17. The server device according to claim 14, wherein the one or more memories hold a third restoration model for generating a time series of free viewpoint images for a first time interval, and the one or more processors transmit the third restoration model in a transmission format executable in the client.

18. The server device according to claim 13, wherein the request includes spatial information, and the one or more processors transmit a restoration model corresponding to the spatial information, which is a time series restoration model from a restoration model corresponding to the time information to a restoration model corresponding to a predetermined termination condition, in a transmission format executable by the client.

19. The server device according to claim 18, wherein the space specified by the spatial information is a predetermined area within the space, or an area within the space excluding the background.

20. The server device according to claim 14, wherein the one or more processors transmit, each time a request including time information is transmitted from the stopped client, a restoration model corresponding to the time information contained in the transmitted request in a transmission format executable by the client.

21. A server device as described in claim 14, which transmits, in a transmission format executable by the client, time-series restoration models ranging from a restoration model corresponding to the time information to a restoration model corresponding to a specified termination condition, the time-series restoration models being thinned out depending on the frame period and / or display mode when the client displays a video and the communication load between the client and the server.

22. The server device according to claim 16, wherein the one or more processors transmit information for identifying the restoration model, including model parameters or hyperparameters of the restoration model, to the client.

23. The server device according to claim 13, wherein the one or more processors transmit a restoration model predicted based on an operation on the client in a transmission format executable by the client.

Images