Image processing apparatus, image processing method and program

Abstract

To provide a technique for providing virtual viewpoint images, which makes it difficult to restore a three-dimensional shape of a subject based on plural virtual viewpoint images.SOLUTION: An image processing apparatus is configured to: acquire plural virtual viewpoint images in which each is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; output videos shown by the plural virtual viewpoint images for display; and limit storage operations of the displayed virtual viewpoint images that follow user instructions in the case when prescribed conditions are met.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program, and in particular to a technology for generating a virtual viewpoint video. [Background technology]

[0002] In recent years, a technology for generating a virtual viewpoint image has been known. For example, a technology called volumetric video has been attracting attention. In this technology, multiple cameras are installed at different positions, and synchronous shooting is performed from multiple viewpoints. Then, a virtual viewpoint image from an arbitrary viewpoint can be generated using images from multiple viewpoints obtained by such shooting. According to such a technology, for example, highlight scenes of a sport such as soccer or basketball can be viewed from an arbitrary angle. In addition, it is also possible to view images from within the field during a game, which was not possible in the past. In this way, a user can view images from various positions or angles that are different from conventional images, that is, images with added value.

[0003] Such a virtual viewpoint image can be generated, for example, by a server acquiring images from multiple cameras and performing a series of processes such as foreground / background separation, three-dimensional shape estimation, three-dimensional model generation, and rendering. Meanwhile, cloud technology has been developed in recent years. The processing capabilities of information devices such as PCs and tablet terminals have also improved. Furthermore, the speed of communication has improved due to the development of network infrastructure. For these reasons, virtual viewpoint operation and video viewing, which were previously performed using dedicated devices, can now be performed using general users' tablet terminals or PCs. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-325920 A Summary of the Invention [Problem to be solved by the invention]

[0005] If a user can freely manipulate the virtual viewpoint, the subject can obtain virtual viewpoint images of the subject from multiple directions. Using these virtual viewpoint images, the user may be able to reconstruct the three-dimensional shape of the subject. For example, such manipulations can be performed using an application with a photogrammetry function. As a result, there is a possibility that the rights of the provider of virtual viewpoint video content may be infringed.

[0006] Patent Document 1 discloses a technique for limiting the subject area for which a virtual viewpoint image is generated. According to the technique described in Patent Document 1, two images captured in different directions are displayed on a display device. When two points are specified from the two images, a reconstructable area is identified based on the specified two points, and an image of this area is generated. However, such a technique is insufficient to prevent the restoration of three-dimensional shape data of a subject based on virtual viewpoint images of the subject from multiple directions.

[0007] An object of the present disclosure is to make it difficult to restore the three-dimensional shape of a subject based on multiple virtual viewpoint images in a technology for providing virtual viewpoint images. [Means for solving the problem]

[0008] An image processing apparatus according to an embodiment of the present invention includes the following components: an acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting an image represented by the plurality of virtual viewpoint images for display; a control means for limiting a saving operation of the displayed virtual viewpoint image in accordance with a user instruction when a predetermined condition is satisfied; Equipped with. Effect of the Invention

[0009] In a technology for providing virtual viewpoint images, it is possible to make it difficult to restore the three-dimensional shape of a subject based on a plurality of virtual viewpoint images. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus according to an embodiment. [Diagram 2] FIG. 1 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to an embodiment. [Diagram 3] 1 is a flowchart of an image processing method according to an embodiment. [Figure 4] 4A to 4C are diagrams illustrating generation of a virtual viewpoint image. [Diagram 5] 5A and 5B are diagrams for explaining conditions related to parameters of a virtual viewpoint. [Figure 6] FIG. 1 is a block diagram showing an example of the functional arrangement of an image processing system according to an embodiment. [Figure 7] 1 is a flowchart of an image processing method according to an embodiment. [Figure 8] FIG. 1 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to an embodiment. [Figure 9] 1 is a flowchart of an image processing method according to an embodiment. [Figure 10] 1 is a flowchart of an image processing method according to an embodiment. [Figure 11] FIG. 1 is a block diagram showing an example of the functional arrangement of an image processing system according to an embodiment. [Figure 12] 1 is a flowchart of an image processing method according to an embodiment. [Figure 13] 1 is a flowchart of an image processing method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although the embodiments describe a plurality of features, not all of these features are necessarily essential, and the plurality of features may be combined in any manner. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] An image processing device according to an embodiment can be realized by a computer including a processor and a memory. Fig. 1 is a block diagram showing a typical hardware configuration example of an image processing device according to an embodiment. An image processing device 100 includes a CPU 101, a main storage device 102, an auxiliary storage device 103, an external I / F 104, and a GPU 105. These components are connected to each other via a bus 106 so as to be able to communicate with each other.

[0013] The CPU 101 is a central processing unit that controls the entire image processing device 100. The CPU 101 can perform various calculations and decisions, image processing, and data input / output.

[0014] The main memory device 102 can function as a work area or a temporary storage area for data for the CPU 101. The main memory device 102 can be implemented using a storage medium such as a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0015] The auxiliary storage device 103 can store information such as various programs, various setting information, various image data, camera parameters, three-dimensional shape data, or two-dimensional maps. The auxiliary storage device 103 can be implemented using a storage medium such as a non-volatile memory such as a read only memory (ROM) or a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a tape medium. The auxiliary storage device 103 may be implemented using a combination of multiple storage media to increase capacity or speed. In this case, these storage media may be logically combined into one by using RAID technology or the like.

[0016] The external I / F 104 is a general-purpose input / output interface. The external I / F 104 can communicate with an external device such as a camera, a PC, or a server. The external I / F 104 can be connected to an external device that accepts user operations such as a joystick, a keyboard, or a mouse. The external I / F 104 can be connected to an external storage to input and output files. The external I / F 104 may be an interface having a connection terminal of a physical cable for InfiniBand, Ethernet (registered trademark), Universal Serial Bus (USB), or the like. The external I / F 104 may be a wireless interface for wireless LAN, Bluetooth, or the like.

[0017] The GPU 105 is a computing device for performing image processing operations at high speed. The GPU 105 may have a function of outputting a video signal to an output device capable of displaying an external video, such as a television. The GPU 105 may also execute a process of rendering a virtual viewpoint image. The GPU 105 may also execute a process of generating shape data or color information required to generate a virtual viewpoint image. The GPU 105 may have a video terminal such as DVI, HDMI (registered trademark), Display Port, or SDI in order to output a video signal. The GPU 105 may include a main memory device different from the main memory device 102. The GPU 105 may also perform a part or all of the various calculations performed by the CPU 101 other than the above-mentioned processes. The GPU 105 may not have a function of outputting a video signal. For example, the GPU 105 may only perform calculations such as shape data or color information required for image processing or generating a virtual viewpoint image. In this case, the external I / F 104 may further have a video output function or a function of outputting data required for video output.

[0018] The bus 106 is configured by a general-purpose bus such as PCI Express, and each part of the hardware block can perform bidirectional communication. In FIG. 1, all the hardware blocks are connected to the same bus 106, but the configuration is not limited to this. For example, the main storage device 102 and the CPU 101 may be connected by a bus different from the bus 106. In this case, the main storage device 102 and other blocks may communicate with each other via the CPU 101. In addition, a hardware block such as the auxiliary storage device 103 or the GPU 105 may be connected to the external I / F 104. In this case, communication with the auxiliary storage device 103 or the GPU 105 can be performed via the external I / F 104. The type of the bus 106 is not limited. As the bus 106, any bus that allows each block to communicate with each other in both directions can be used. For example, the bus 106 may be a bus that conforms to a standardized communication standard such as USB. The bus 106 may be a communication bus designed specifically for the purpose.

[0019] In this way, a processor such as the CPU 101 executes a program stored in a memory such as the main storage device 102 or the auxiliary storage device 103, thereby realizing the functions of each unit shown in FIG. 2, which will be described later. Note that the configuration shown in FIG. 1 is merely one example of the configuration of the image processing device 100. For example, some or all of the functions of the image processing device 100 may be realized by dedicated hardware. Also, some of the functions of the image processing device 100 may be realized by externally connected hardware. Also, the main storage device 102 and the auxiliary storage device 103 may be realized by a single piece of hardware. Furthermore, other hardware such as an FPGA, a DSP, or a dedicated image processing LSI may be used instead of the GPU 105.

[0020] Moreover, the image processing device 100 does not have to be configured by a single device. For example, a plurality of devices having similar configurations may be prepared. In this case, the configurations of the individual devices may be different. Furthermore, the image processing device according to an embodiment may be configured by a plurality of information processing devices connected via a network, for example. In an embodiment, the functions of the image processing device 100 can be provided as a cloud service.

[0021] [Embodiment 1] An image processing device according to an embodiment will be described below with reference to the drawings. In the following embodiment, the image processing device 100 generates and outputs a virtual viewpoint video represented by a plurality of virtual viewpoint images. However, the image processing device 100 may acquire a virtual viewpoint video from another device and output the acquired virtual viewpoint video. In the following embodiment, the image processing device 100 has a function of saving the virtual viewpoint image constituting the virtual viewpoint video. Such a saving function is also called an image capture function. That is, the image processing device 100 can save the virtual viewpoint image displayed on the output device according to a user instruction. In the following example, the saved virtual viewpoint image is a two-dimensional image. Note that the saving method is not particularly limited. In one embodiment, the output virtual viewpoint image is saved. In another embodiment, a screenshot of the display on the output device is saved. In a further embodiment, a virtual viewpoint image based on the same time code and virtual viewpoint as the virtual viewpoint image displayed on the output device is generated by the video generation unit 205, and the generated virtual viewpoint image is saved.

[0022] Furthermore, in this embodiment, the saving operation of the virtual viewpoint image is restricted in accordance with a predetermined condition. First, a case where the saving operation of the virtual viewpoint image is restricted in a state where the playback of the virtual viewpoint video is stopped will be described below.

[0023] In the following example, the image processing device 100 can play back a volumetric video. The volumetric video is generated based on a three-dimensional model corresponding to each time. More specifically, the volumetric video is generated based on a three-dimensional model linked to a time code representing the time. The three-dimensional model can change according to time. The image processing device 100 can generate an image of the three-dimensional model corresponding to a specific time from a virtual viewpoint as a virtual viewpoint image. Note that image processing such as adding a background may be further performed on the virtual viewpoint image.

[0024] As described later, the video output unit 206 of the image processing device 100 sequentially outputs virtual viewpoint images of a three-dimensional model corresponding to a specified time. Here, playing a volumetric video corresponds to continuously updating a specified time. That is, a plurality of virtual viewpoint images are generated based on a three-dimensional model corresponding to each different time. A virtual viewpoint video composed of these multiple virtual viewpoint images is played as a volumetric video. Hereinafter, sequentially displaying virtual viewpoint images generated based on a three-dimensional model corresponding to each successive time is referred to as playing a volumetric video.

[0025] On the other hand, a plurality of virtual viewpoint images may be generated based on a specific three-dimensional model corresponding to a specific time. In this specification, the continuous display of a virtual viewpoint image generated based on a specific three-dimensional model corresponding to a specific time is called stopping of the volumetric video. Stopping the volumetric video corresponds to continuously specifying the same time. Stopping the volumetric video includes sequentially displaying a plurality of virtual viewpoint images generated based on a specific three-dimensional model corresponding to a specific time. These plurality of virtual viewpoint images may be generated based on different virtual viewpoints. In this way, when the volumetric video is stopped, the video output unit 206 of the image processing device 100 can output a virtual viewpoint video composed of virtual viewpoint images from various directions for a three-dimensional model corresponding to the same time. On the other hand, stopping the volumetric video may include the continuous display of the same one virtual viewpoint image. Stopping the volumetric video includes a case where the progression of the time code is temporarily stopped (i.e., paused).

[0026] 2 is a schematic diagram showing an example of a functional configuration of an image processing device 100 according to an embodiment. The image processing device 100 includes a model acquisition unit 201, a space construction unit 202, a time designation unit 203, a viewpoint designation unit 204, a video generation unit 205, a video output unit 206, an image storage unit 207, an image storage unit 208, and a storage control unit 209.

[0027] The model acquisition unit 201 acquires data of a three-dimensional model corresponding to each time. In this embodiment, the model acquisition unit 201 acquires a three-dimensional model of a volumetric video linked to a time code instructed by the space configuration unit 202. The three-dimensional model can represent content such as sports such as soccer, a concert, or a play. Specifically, the three-dimensional model can represent the three-dimensional shape of an object. The three-dimensional model can also represent the texture of an object. The model acquisition unit 201 may acquire a three-dimensional model for each of a plurality of objects. For example, the three-dimensional model can represent the three-dimensional shape of a person such as a player or a performer, or an object such as a ball. The three-dimensional model may also represent the three-dimensional shape of a structure such as a stadium or stage equipment that serves as a background. Note that some objects may not change with time. For example, a structure that serves as a background may be stationary regardless of a time code. A three-dimensional model corresponding to such an object may be linked to a plurality of time codes. A range of continuous time codes may be linked to one three-dimensional model.

[0028] The model acquisition unit 201 may acquire these three-dimensional models via a network such as the Internet or a local area network (LAN). The model acquisition unit 201 may read these three-dimensional models from a storage medium such as an optical medium, a USB memory, a HDD, or a magnetic tape medium. The model acquisition unit 201 may acquire three-dimensional models associated with all time codes at once. In addition, the model acquisition unit 201 may sequentially acquire three-dimensional models necessary for the space construction unit 202 to construct a virtual space for a specific time code.

[0029] The space construction unit 202 constructs a three-dimensional virtual space. The space construction unit 202 can construct a virtual space corresponding to a specific time code designated by the time designation unit 203. Specifically, the space construction unit 202 can acquire a three-dimensional model linked to the specific time code from the model acquisition unit 201. Then, the space construction unit 202 can place the acquired three-dimensional model in the virtual space.

[0030] The time designation unit 203 designates a time. The video generation unit 205, which will be described later, generates a virtual viewpoint image of the three-dimensional model corresponding to the time designated by the time designation unit 203. In this embodiment, the time designation unit 203 can designate a time code. Then, the time designation unit 203 outputs the designated time code to the space construction unit 202 so as to construct a three-dimensional virtual space corresponding to the time code.

[0031] During normal playback, the time designation unit 203 advances the time code as real time passes. For example, the time designation unit 203 can advance the time code so that it advances 60 frames per second. On the other hand, the time designation unit 203 can stop counting up the time code. In this case, the volumetric video that is output or displayed stops. Note that the number of frames per second during normal playback is not limited to 60 frames. The number of frames per second may be, for example, 50 frames, 30 frames, 24 frames, 59.94 frames, 29.97 frames, or 23.976 frames.

[0032] The time designation unit 203 can control the time code in response to an instruction from the user. For example, the time designation unit 203 can stop counting up the time code when receiving a pause instruction from the user. Also, the time designation unit 203 can increase the number of frames counted up or down per second when receiving a fast-forward or fast-rewind instruction from the user. Furthermore, the time designation unit 203 can also control the time code for special playback. For example, the time designation unit 203 can accept an operation to directly designate a time code.

[0033] The viewpoint designation unit 204 designates a virtual viewpoint. The viewpoint designation unit 204 can set a virtual viewpoint in the virtual space constructed by the space construction unit 202. The virtual viewpoint corresponds to a virtual camera that takes pictures in the virtual space. The virtual camera is a concept for explaining a virtual viewpoint used to generate a virtual viewpoint image. The virtual camera is a virtual camera that is different from a plurality of imaging devices that are actually installed around the imaging area to generate a three-dimensional model. That is, the virtual viewpoint image can be considered as an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. In other words, it can be said that the virtual viewpoint image is an image that imitates an image captured by a camera when it is assumed that a camera exists at the position of the virtual viewpoint set in the virtual space. The image generation unit 205 described later generates an image of a three-dimensional model arranged in the virtual space from the virtual viewpoint as a virtual viewpoint image.

[0034] The viewpoint designation unit 204 can operate the virtual viewpoint based on a user operation. For example, the viewpoint designation unit 204 can move the virtual viewpoint in a virtual space. The viewpoint designation unit 204 can also set parameters for the virtual viewpoint. Specifically, the viewpoint designation unit 204 can perform camera operations such as zooming. The position and orientation of the virtual viewpoint thus set correspond to the position and orientation of the virtual camera. In this way, a virtual viewpoint image from a free viewpoint can be generated.

[0035] In this embodiment, the viewpoint designation unit 204 sets camera parameters of a virtual camera corresponding to a virtual viewpoint. The camera parameters may include external parameters and internal parameters. Information included in the camera parameters is not particularly limited. The camera parameters may include, for example, parameters representing a three-dimensional position of the camera and information representing a three-dimensional orientation of the camera. The orientation of the camera can be represented by information representing rotation in the pan, tilt, and roll directions. The camera parameters may also include information such as the size of the field of view (angle of view), the number of pixels of a virtual sensor, or a focal length.

[0036] The method of setting the virtual viewpoint is not particularly limited. For example, the virtual viewpoint may be linked to a character or object in the virtual space, such as an avatar operated by the user. For example, a virtual camera may correspond to the field of view of the character. The virtual viewpoint may also be set so that its relative position with respect to these characters or objects is fixed. In this case, too, the virtual viewpoint can move in accordance with the movement of the character or object.

[0037] The video generation unit 205 can obtain a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time. As described above, the video generation unit 205 can generate, as a virtual viewpoint image, an image of a three-dimensional model corresponding to a time specified by the time specification unit 203 from a virtual viewpoint specified by the viewpoint specification unit 204. For example, as described above, the video generation unit 205 can generate, as a virtual viewpoint image, an image of a three-dimensional model from a specified virtual viewpoint in a virtual space corresponding to a specific time code. The video generation unit 205 can generate the virtual viewpoint image as a two-dimensional image in a format that can be viewed by a user. Furthermore, the video generation unit 205 can sequentially generate virtual viewpoint images of a three-dimensional model corresponding to a specified time. For example, the video generation unit 205 can generate a plurality of virtual viewpoint images for virtual spaces corresponding to different time codes. Furthermore, the video generation unit 205 can generate a plurality of virtual viewpoint images for virtual spaces corresponding to the same time code. Furthermore, the video generation unit 205 can generate a plurality of virtual viewpoint images each corresponding to a different virtual viewpoint. In this way, the video generator 205 can generate a virtual viewpoint video made up of a plurality of virtual viewpoint images.

[0038] The video output unit 206 outputs a video represented by a plurality of virtual viewpoint images for display. The video output unit 206 can sequentially output virtual viewpoint images of a three-dimensional model corresponding to a specified time, which are generated by the video generation unit 205. For example, the video output unit 206 can output a virtual viewpoint video composed of a plurality of virtual viewpoint images generated by the video generation unit 205 to an output device such as a display. At this time, the video output unit 206 may output the virtual viewpoint video via a video signal output terminal such as HDMI (registered trademark), DisplayPort, or SDI.

[0039] The image storage unit 207 stores the virtual viewpoint image. The image storage unit 207 can store the virtual viewpoint image according to a user instruction. Furthermore, the image storage unit 207 can store the displayed virtual viewpoint image. Specifically, the image storage unit 207 can store, among the virtual viewpoint images constituting the virtual viewpoint video, the virtual viewpoint image output by the video output unit 206 or the virtual viewpoint image displayed on the output device.

[0040] The image saving unit 207 can store the virtual viewpoint image generated by the video generating unit 205 in the image storage unit 208. The image saving unit 207 can save the virtual viewpoint image as a two-dimensional image file conforming to a format such as JPEG, BMP, PNG, or MPEG. Here, the saving operation of the virtual viewpoint image by the image saving unit 207 is restricted by the saving control unit 209. In this example, the image saving unit 207 stores the virtual viewpoint image in the image storage unit 208 in response to the decision by the saving control unit 209 to permit saving.

[0041] The saving control unit 209 restricts the saving operation of the virtual viewpoint image when a predetermined condition is satisfied. In this example, the saving control unit 209 judges whether or not to permit saving of the virtual viewpoint image. The saving control unit 209 can prohibit saving of the virtual viewpoint image in response to the judgment that the predetermined condition is satisfied. In this way, the saving control unit 209 can judge whether or not it is permitted to save the virtual viewpoint image as a two-dimensional image file.

[0042] In this example, the predetermined condition is that the volumetric video is stopped. That is, the storage control unit 209 determines whether or not the virtual viewpoint image may be stored depending on whether or not the volumetric video is stopped. The storage control unit 209 determines whether or not the virtual viewpoint image may be stored depending on whether or not the volumetric video is being played. On the other hand, the storage control unit 209 determines not to permit the storage of the virtual viewpoint image depending on whether or not the volumetric video is stopped.

[0043] The save control unit 209 may notify the user that the saving operation of the virtual viewpoint image is restricted. For example, if saving of the image is not permitted, the save control unit 209 can notify the user that the image cannot be saved. The save control unit 209 can notify the user via an output device that displays the virtual viewpoint video.

[0044] Fig. 3 is a flowchart showing a processing flow of an image processing method performed by the image processing device 100 according to an embodiment. A virtual space image is generated according to the flowchart shown in Fig. 3. Furthermore, by repeating the flowchart shown in Fig. 3, a plurality of virtual space images are generated, that is, a virtual space video is generated.

[0045] In S301, the time designation unit 203 sets a time code. When the volumetric video is stopped, the time designation unit 203 sets the time code to a fixed value. In this case, the configuration of the virtual space constructed in S302 becomes fixed. This realizes the stopping of the volumetric video. On the other hand, when playing the volumetric video, the time designation unit 203 counts up the time code.

[0046] In S302, the space construction unit 202 acquires a three-dimensional model of the volumetric video corresponding to the time code designated by the time designation unit 203 from the model acquisition unit 201. Then, the space construction unit 202 constructs a virtual space as described above.

[0047] In S303, the viewpoint designation unit 204 sets the parameters of the virtual viewpoint in response to the user's operation as described above, thus determining the virtual viewpoint of the volumetric video.

[0048] In S304, the video generation unit 205 generates a virtual viewpoint image that the user can see, according to the parameters of the virtual viewpoint set in S303. The virtual viewpoint image thus generated may be displayed on an output device.

[0049] FIG. 4 is a diagram for explaining generation of a virtual viewpoint image based on volumetric video data created by shooting a soccer match. FIG. 4 shows a schematic representation of a virtual space at a certain time and a state in which shooting is performed in this virtual space using a virtual camera. FIG. 4(A) is a schematic representation of the state of the virtual space from a bird's-eye view above the stadium. A soccer match is played on a soccer court 401. Spectator seats 402 are seats at a soccer stadium. Seats are installed in the spectator seats 402 in the same way as in a soccer stadium in the real world.

[0050] The player 403 is a soccer player. The player 403 is represented as a three-dimensional model in the volumetric video. In the following, for the sake of explanation, one player 403 on the soccer court 401 will be described. However, two or more players may exist in the virtual space. In a soccer game, there are usually 11 players per team, that is, a total of 22 players from the two competing teams, plus referees on the soccer court 401.

[0051] Virtual cameras 404a and 404b are virtual cameras operated by viewpoint designation unit 204. Virtual cameras 404a and 404b virtually capture images of player 403. Note that virtual cameras 404a and 404b may be invisible in the virtual space. For example, when virtual camera 404b facing toward virtual camera 404a captures an image, virtual camera 404a does not have to appear in the captured image of virtual camera 404b.

[0052] The virtual cameras 404a and 404b can move to any location, including the field where the soccer game is played. The virtual cameras 404a and 404b can also move freely in three dimensions in the up-down, left-right, and height directions. However, the positions of the virtual cameras 404a and 404b may be restricted. For example, the virtual cameras 404a and 404b may be movable only within a predetermined area such as the spectator seats 402. An area in which the virtual cameras 404a and 404b cannot move may be set in the virtual space. As described above, the virtual viewpoint may be the viewpoint of an avatar placed in the virtual space. In this case, the movable range of the avatar may be limited to a predetermined area such as the spectator seats 402. Furthermore, the movement of the virtual viewpoint in the height direction may be limited to a certain range. In this case, the certain range may be set according to the height of the avatar.

[0053] Fig. 4(B) is a schematic representation of an example of a virtual viewpoint image obtained by virtual camera 404a. Fig. 4(C) is a schematic representation of an example of a virtual viewpoint image obtained by virtual camera 404b. The image generator 205 can render these two-dimensional images so that the user can view them.

[0054] In S305, the image storage unit 207 determines whether or not a user has instructed to save the virtual viewpoint image. If it is determined that a user has instructed to save the virtual viewpoint image, the process proceeds to S306. If it is determined that a user has not instructed to save the virtual viewpoint image, the process illustrated in FIG. 3 ends.

[0055] In S306, the storage control unit 209 determines whether or not to restrict the saving operation of the virtual viewpoint image. In this example, the storage control unit 209 determines whether or not the volumetric video has stopped. The storage control unit 209 can inquire of the time designation unit 203 whether or not the volumetric video has stopped. If it is determined that the volumetric video has stopped, the process proceeds to S307. If not, the process proceeds to S308.

[0056] In S307, the image storage unit 207 does not store the virtual viewpoint image. In this case, the image storage unit 207 can notify the user who instructed to store the virtual viewpoint image that the image cannot be stored. For example, the image storage unit 207 can display a UI indicating that the image cannot be stored on an output device that displays the virtual viewpoint video.

[0057] In S308, the image saving unit 207 saves the virtual viewpoint image generated in S304 in the image storage unit 208.

[0058] According to the above embodiment, the virtual viewpoint image cannot be saved when the volumetric video is stopped. For example, the user may try to save a virtual viewpoint image of a three-dimensional model corresponding to a specific time code from the position of the virtual camera 404a as shown in FIG. 4(B) when the volumetric video is stopped. Furthermore, the user may try to save a virtual viewpoint image of a three-dimensional model corresponding to the same time code from the position of the virtual camera 404b as shown in FIG. 4(C) by moving the virtual viewpoint. However, in the above embodiment, the virtual viewpoint image cannot be saved when the three-dimensional model of the subject is stationary. This makes it difficult to save multiple virtual viewpoint images of a three-dimensional model corresponding to the same time code. This makes it difficult to restore the three-dimensional shape of the subject based on the virtual viewpoint image.

[0059] (Variation 1) As described above, the storage control unit 209 restricts the image storage unit 207 from storing the virtual viewpoint image in accordance with a predetermined condition. In the above example, the storage control unit 209 prohibits the image storage unit 207 from storing the virtual viewpoint image when the volumetric video is stopped. On the other hand, the predetermined condition is not limited to the volumetric video being stopped. In the following, a case will be described in which the predetermined condition includes satisfying a first condition related to the virtual viewpoint. In the following example, the storage control unit 209 can restrict the image storage unit 207 from storing the virtual viewpoint image in accordance with a specified virtual viewpoint.

[0060] For example, the storage control unit 209 can limit the saving operation of the virtual viewpoint image by the image saving unit 207 based on whether the virtual viewpoint satisfies a first condition. More specifically, the storage control unit 209 can determine whether to limit the saving operation of the virtual viewpoint image by the image saving unit 207 based on whether the parameters of the virtual viewpoint satisfy the first condition. For example, the storage control unit 209 can determine whether to limit the saving operation of the virtual viewpoint image by the image saving unit 207 according to the position of the virtual viewpoint specified by the viewpoint specifying unit 204. In one embodiment, the first condition is that the position of the virtual viewpoint is not within a specific range. For example, the storage control unit 209 can determine to limit the saving operation of the virtual viewpoint image according to the determination that the position of the virtual viewpoint is not within a specific range.

[0061] This process will be described with reference to Fig. 5(A). The photographing-permitted area 701 indicates the range in which photographing by the virtual camera (i.e., saving of a virtual viewpoint image) is permitted. Since the virtual camera 404a is within the photographing-permitted area 701, the saving control unit 209 allows saving of the virtual viewpoint image obtained by the virtual camera 404a. On the other hand, since the virtual camera 404b is not within the photographing-permitted area 701, the saving control unit 209 does not allow saving of the virtual viewpoint image obtained by the virtual camera 404a.

[0062] Furthermore, the saving control unit 209 can determine whether or not to restrict the saving operation of the virtual viewpoint image by the image saving unit 207, depending on the orientation of the virtual viewpoint specified by the viewpoint specifying unit 204, or a combination of the position and orientation. For example, the saving control unit 209 can determine to restrict the saving operation of the virtual viewpoint image, depending on whether it is determined that the orientation of the virtual viewpoint is not within a specific range.

[0063] This process will be described with reference to Figures 5(B) and (C). The permitted shooting direction range 703 indicates the range of the orientation of the virtual camera in which shooting by the virtual camera is permitted. As shown in Figure 5(B), since the orientation 702 of the virtual camera 404a is within the permitted shooting direction range 703, the storage control unit 209 permits the storage of the virtual viewpoint image obtained by the virtual camera 404a. On the other hand, as shown in Figure 5(C), since the orientation 704 of the virtual camera 404b is not within the permitted shooting direction range 703, the storage control unit 209 does not permit the storage of the virtual viewpoint image obtained by the virtual camera 404a.

[0064] The above-mentioned conditions regarding the position of the virtual viewpoint and the conditions regarding the direction of the virtual viewpoint may be combined. For example, when the direction 702 of the virtual camera 404a is within the photographing permitted direction range 703 and the virtual camera 404a is within the photographing permitted area 701, the storage control unit 209 may permit the storage of the virtual viewpoint image obtained by the virtual camera 404a. In other words, when the virtual camera 404a faces the direction 704, even if the virtual camera 404a is within the photographing permitted area 701, the storage control unit 209 can restrict the operation of saving the virtual viewpoint image obtained by the virtual camera 404a. In this way, the conditions regarding the camera parameters of the virtual camera, such as the photographing permitted area 701 and the photographing permitted direction range 703, can be appropriately combined.

[0065] Furthermore, the storage control unit 209 can determine whether or not to limit the operation of storing the virtual viewpoint image by the image storage unit 207, depending on the resolution of the three-dimensional model of the subject on the virtual viewpoint image. Such a resolution can be determined based on the parameters of the virtual viewpoint. For example, the resolution can be expressed by the number of pixels on the virtual viewpoint image of a portion corresponding to one three-dimensional shape component, such as a voxel or texture, of the three-dimensional model corresponding to the subject. Note that the three-dimensional shape component corresponding to the texture can be, for example, one pixel constituting the texture, or an entire texture represented by vertex coordinates. In such an example, the first condition may be that the resolution is smaller than one pixel. Such a resolution can be determined based on the position, angle of view, number of pixels, etc. of the virtual camera. Therefore, such a condition regarding the resolution is also included in the first condition regarding the virtual viewpoint.

[0066] In such a modified example, in S306, the storage control unit 209 judges whether or not to restrict the operation of storing the virtual viewpoint image according to the virtual viewpoint as described above. If it is judged that the operation of storing the virtual viewpoint image by the image storage unit 207 is restricted according to the above conditions, the process proceeds to S307. If not, the process proceeds to S308.

[0067] When the parameters of the virtual viewpoint satisfy the first condition, the storage control unit 209 may further control the time designation unit 203 to prohibit stopping of the volumetric video.

[0068] According to this modification, the virtual viewpoint corresponding to the virtual viewpoint image to be saved is limited. Therefore, it is possible to suppress saving of virtual viewpoint images of a three-dimensional model from various directions in order to restore the three-dimensional shape of the subject. In addition, by bringing the virtual viewpoint closer to the three-dimensional model, it is possible to suppress saving of a virtual viewpoint image of a three-dimensional model having a high resolution. In this way, according to this modification, it is possible to make it difficult to restore the three-dimensional shape of the subject.

[0069] (Variation 2) In the following, a case will be described in which the predetermined condition includes satisfying a second condition related to the number of times the virtual viewpoint image is saved. For example, the save control unit 209 can limit the saving operation of the virtual viewpoint image by the image saving unit 207 according to the number of times the virtual viewpoint image is saved. The second condition may include a condition related to the number of times the virtual viewpoint image of the three-dimensional model is saved at a specific time. For example, the second condition may be that the number of times the virtual viewpoint image is saved at a specific time code is equal to or greater than a predetermined number. In such an example, a limit can be set on the number of virtual viewpoint images that can be saved at the same time code of the volumetric video.

[0070] In this modification, the image storage unit 207 stores image recording information corresponding to the virtual viewpoint image in the image storage unit 208 in association with the virtual viewpoint image. This image recording information may include time information related to the virtual viewpoint image. This time information may indicate a time code corresponding to the virtual viewpoint image. For example, the image storage unit 207 can store the time code of the volumetric video being played back, acquired from the time designation unit 203, as the image recording information. Meanwhile, the time information may indicate the real time when the virtual viewpoint image is stored. The recorded time may be based on Universal Time. Furthermore, the image recording information may include information on the virtual viewpoint corresponding to the virtual viewpoint image. For example, the image recording information may include camera parameters including external parameters and internal parameters set by the viewpoint designation unit 204 when the image storage unit 207 stores the virtual viewpoint image.

[0071] In addition, the storage control unit 209 determines whether or not to restrict the saving operation of the virtual viewpoint image according to the number of times the virtual viewpoint image is saved. For example, the storage control unit 209 can acquire the current time code designated by the time designation unit 203. Then, the storage control unit 209 determines the number of times the virtual viewpoint image has been saved in the past at the current time code by referring to the image storage unit 208. The number of times the virtual viewpoint image has been saved at the current time code may be the number of times the virtual viewpoint image has been saved at the same time code as the current time code. On the other hand, the number of times the virtual viewpoint image has been saved at the time code near the current time code may be the number of times the virtual viewpoint image has been saved. For example, the storage control unit 209 may determine the number of times the virtual viewpoint image has been saved within a predetermined period near the current time code (for example, within one second before and after). In addition, the volumetric video may be divided into a plurality of scenes in advance. For example, the volumetric video may be divided in units of one second or the like according to the time code. In this case, the storage control unit 209 can determine the number of times the virtual viewpoint image has been saved at the same scene as the current time code.

[0072] In this example, the second condition is that the number of times of saving is a predetermined number or more. That is, in response to the determination that the user has saved the virtual viewpoint image for the current time code a predetermined number or more in the past, the saving control unit 209 restricts the saving operation of the virtual viewpoint image. On the other hand, in response to the determination that the user has not saved the virtual viewpoint image for the current time code a predetermined number or more in the past, the saving control unit 209 permits saving of the virtual viewpoint image. Note that the predetermined number of times is not particularly limited, and may be once, twice, three times, or four or more times.

[0073] In another embodiment, the storage control unit 209 may determine the number of times the virtual viewpoint image is stored for the entire volumetric video. In this example, the second condition is that the number of times the virtual viewpoint image is stored for the entire volumetric video is equal to or greater than a predetermined number. In such an example, the number of times the virtual viewpoint image is stored for the entire volumetric video can be limited.

[0074] In another embodiment, the storage control unit 209 may determine the number of times the virtual viewpoint image is saved within a predetermined period of real time (i.e., the frequency of saving). In this example, the second condition is that the number of times the virtual viewpoint image is saved is equal to or greater than a threshold value. For example, the storage control unit 209 may limit the saving operation of the virtual viewpoint image in response to determining that the virtual viewpoint image has already been saved 10 times within 15 minutes in real time. Note that the length of the predetermined period is not particularly limited, and may be longer than 15 minutes, such as 1 hour, 1 day, or 1 week, or may be shorter than 15 minutes, such as 5 minutes or 1 minute. The threshold value is also not particularly limited, and may be longer or shorter than 10 times, for example.

[0075] Furthermore, the storage control unit 209 may use a combination of the above conditions to determine whether or not to restrict the operation of storing the virtual viewpoint image.

[0076] In such a modified example, in S306, the storage control unit 209 determines whether or not to limit the saving operation of the virtual viewpoint image according to the number of times the virtual viewpoint image has been saved, as described above. If it is determined that the saving operation of the virtual viewpoint image by the image saving unit 207 is to be limited according to the above conditions, the process proceeds to S307. If not, the process proceeds to S308.

[0077] According to this modification, the number of times the virtual viewpoint image is saved is limited. Therefore, it becomes difficult for the user to save more than a predetermined number of virtual viewpoint images. That is, it is possible to prevent the user from saving a large number of virtual viewpoint images of a three-dimensional model from various directions in order to restore the three-dimensional shape of the subject. In this way, according to this modification, it is possible to make it difficult to restore the three-dimensional shape of the subject.

[0078] (Variation 3) So far, an example of restricting the saving operation of a virtual viewpoint image when the volumetric video is stopped, an example of restricting the saving operation of a virtual viewpoint image according to a virtual viewpoint, and an example of restricting the saving operation of a virtual viewpoint image according to the number of times of saving have been described. These conditions can be used in combination. There is no particular limit to the way in which the conditions are combined. For example, it may be determined that a predetermined condition is satisfied when all of a plurality of conditions are satisfied. It may also be determined that a predetermined condition is satisfied when at least one of a plurality of conditions is satisfied.

[0079] For example, the storage control unit 209 may limit the saving operation of the virtual viewpoint image in response to determining that either or both of the conditions of the volumetric video being stopped and the condition related to the virtual viewpoint parameters are satisfied. Also, the storage control unit 209 may limit the saving operation of the virtual viewpoint image in response to determining that either or both of the conditions of the volumetric video being stopped and the condition related to the number of times the virtual viewpoint image is saved are satisfied. Furthermore, the storage control unit 209 may limit the saving operation of the virtual viewpoint image in response to determining that either or both of the conditions of the virtual viewpoint parameters and the condition related to the number of times the virtual viewpoint image is saved are satisfied.

[0080] In the above example, when the save control unit 209 restricts the saving operation of the virtual viewpoint image, the image saving unit 207 does not save the virtual viewpoint image in the image storage unit 208. However, the manner of restriction by the save control unit 209 is not limited to the above. For example, the save control unit 209 may reduce the quality of the virtual viewpoint image to be saved in response to a determination that a predetermined condition is satisfied. For example, the save control unit 209 can control the image saving unit 207 to reduce the virtual viewpoint image to be saved in response to a determination that a predetermined condition is satisfied. In this case, the image saving unit 207 can save the reduced virtual viewpoint image in the image storage unit 208. In this way, the image saving unit 207 can lower the resolution of the three-dimensional model in the virtual viewpoint image to be saved in the image storage unit 208.

[0081] As described above, the resolution of the three-dimensional model in the virtual viewpoint image can be expressed by the number of pixels on the virtual viewpoint image that correspond to one three-dimensional shape component of the three-dimensional model corresponding to the subject. In this case, the save control unit 209 can control the resolution of the three-dimensional model in the virtual viewpoint image that the image save unit 207 saves in the image storage unit 208 to be smaller than one pixel.

[0082] As another example, when the save control unit 209 restricts the save operation of the virtual viewpoint image, the playback state of the volumetric video or the parameters of the virtual viewpoint may be changed so that the saving of the virtual viewpoint image is permitted. For example, the save control unit 209 can control the time designation unit 203 to release the stopped state of the volumetric video and play the volumetric video in response to a user instruction to save the virtual viewpoint image. Even in such a configuration, while saving of the virtual viewpoint image is not permitted when the volumetric video is stopped, the playback of the volumetric video is resumed in response to a user instruction, so that the virtual viewpoint image can be saved.

[0083] [Embodiment 2] An image processing system 1000 according to an embodiment will be described below with reference to FIG. 6. FIG. 6 shows an example of a functional configuration of the image processing system 1000 according to this embodiment. The image processing system 1000 includes a server device 1001 and a client device 1002. In this embodiment, as in the first embodiment, the saving operation of the virtual viewpoint image is restricted according to a predetermined condition. On the other hand, this embodiment differs from the first embodiment in that the server device 1001 performs rendering processing of the volumetric video, and the virtual viewpoint video is sent to the client device 1002 by communication. Specifically, the server device 1001 generates the virtual viewpoint video. In addition, the client device 1002 receives the virtual viewpoint video from the server device 1001 and plays the virtual viewpoint video. In this embodiment, the client device 1002 does not receive three-dimensional shape data of the object. The hardware configurations of the server device 1001 and the client device 1002 may be the same as the hardware configuration of the image processing device 100.

[0084] The server device 1001 has a model acquisition unit 201, a space construction unit 202, a time designation unit 203, a viewpoint designation unit 204, a video generation unit 205, a save control unit 209, and a communication unit 1003. The client device 1002 has a video output unit 206, an image save unit 207, an image storage unit 208, a communication unit 1003, and an operation unit 1004. The same functional units as those in the first embodiment are denoted by the same reference symbols. The differences from the first embodiment will be described below.

[0085] The communication unit 1003 communicates with the server device 1001 or the client device 1002. The server device 1001 and the client device 1002 can communicate bidirectionally using the communication unit 1003. The configuration of the communication unit 1003 can be selected so that a transmission band required for communication between the server device 1001 and the client device 1002 is secured. The communication unit 1003 may communicate via a network. For example, the communication unit 1003 may communicate via a local network such as Ethernet. In this case, the communication unit 1003 can communicate via, for example, various switching hubs or router devices. At least a part of the network may be the Internet or a dedicated line. In this case, the server device 1001 and the client device 1002 can be installed in physically separate locations.

[0086] The operation unit 1004 accepts a user operation. Moreover, the operation unit 1004 can send information indicating the user operation to each unit of the server device 1001, such as the time designation unit 203 and the viewpoint designation unit 204, via communication by the communication unit 1003. For example, the operation unit 1004 can accept a user operation on the virtual viewpoint and send information indicating the user operation to the viewpoint designation unit 204. Furthermore, the operation unit 1004 can accept a user operation related to the reproduction of the volumetric video. The operation unit 1004 can accept, for example, a user operation for designating a time code, playing, stopping, and performing special playback. The operation unit 1004 can transmit information indicating these user operations to the time designation unit 203. In this way, the user can operate the displayed virtual viewpoint video. Furthermore, the operation unit 1004 can accept a user operation for saving a virtual viewpoint image. The image saving unit 207 can save the virtual viewpoint image in the image storage unit 208 according to such a user operation.

[0087] In the server device 1001 according to this embodiment, the video generation unit 205 acquires a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time. The communication unit 1003 outputs the video represented by the plurality of virtual viewpoint images generated by the video generation unit 205 to the client device 1002 for display. The video data transmitted by the communication unit 1003 may be compressed by any encoding method such as MPEG or H.264. The storage control unit 209 restricts the storage operation of the displayed virtual viewpoint image according to a user instruction, which is performed by the image storage unit 207, to the case where a predetermined condition is satisfied.

[0088] In addition, in the client device 1002 according to this embodiment, the communication unit 1003 acquires a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time. In addition, the video output unit 206 outputs a video represented by the plurality of virtual viewpoint images to an output device for display. In addition, the image storage unit 207 can store in the image storage unit 208, among the virtual viewpoint images constituting the virtual viewpoint video sent from the server device 1001, the virtual viewpoint image output by the video output unit 206 or the virtual viewpoint image displayed on the output device. At this time, the image storage unit 207 can limit the saving operation of the displayed virtual viewpoint image according to the user's instruction in accordance with the control by the saving control unit 209 when a predetermined condition is satisfied.

[0089] Figures 7(A) and (B) are flowcharts showing the processing flow of an image processing method performed by an image processing system 1000 according to an embodiment. A virtual space image is generated and displayed by repeating the flowcharts shown in Figures 7(A) and (B). In the following, a configuration in which saving of a virtual viewpoint image is restricted when the volumetric video is stopped, as in the first embodiment, will be described. Figure 7(A) shows processing performed by a server device 1001. Also, Figure 7(B) shows processing performed by a client device 1002.

[0090] First, a flowchart of the process performed by the server device 1001 shown in Fig. 7(A) will be described. In S1101, the communication unit 1003 of the server device 1001 acquires information indicating a user operation from the operation unit 1004. For example, the communication unit 1003 can receive information indicating a user operation for changing a virtual viewpoint or a user operation related to the playback of a volumetric video. Based on the received information, the viewpoint designation unit 204 can set a virtual viewpoint. Also, based on the received information, the time designation unit 203 can perform time code control related to the playback of a volumetric video.

[0091] S301, S302, S303, and S304 are performed in the same manner as in the first embodiment. The determination in S306 is also performed in the same manner as in the first embodiment. That is, the save control unit 209 can determine whether or not to restrict the save operation of the virtual viewpoint image according to a predetermined condition. If the save operation of the virtual viewpoint image is restricted, the process proceeds to S1102. If not, the process proceeds to S1103. In this example, if it is determined that the volumetric video is stopped, the process proceeds to S1102. If not, the process proceeds to S1103.

[0092] In S1102, the saving control unit 209 determines to restrict the saving operation of the virtual viewpoint image. In this example, the saving control unit 209 determines to prohibit saving of the virtual viewpoint image. In S1103, the saving control unit 209 determines not to restrict the saving operation of the virtual viewpoint image. In this example, the saving control unit 209 determines to permit saving of the virtual viewpoint image.

[0093] In S1104, the video generation unit 205 transmits the virtual viewpoint image generated in S304 to the client device 1002 via the communication unit 1003. In addition, the storage control unit 209 transmits information (storage permission information) indicating whether or not to restrict the operation of saving the virtual viewpoint image to the client device 1002 via the communication unit 1003. For example, the storage control unit 209 may transmit the determination results in S1102 to S1103 to the client device 1002. The storage control unit 209 may transmit information specifying a method of restricting the saving of the virtual viewpoint image to the client device 1002.

[0094] Next, a flowchart of the process performed by the client device 1002 shown in FIG. 7(B) will be described.

[0095] In S1105, the operation unit 1004 accepts a user operation. For example, the operation unit 1004 can accept a user operation related to the playback of a virtual viewpoint and a volumetric video. Then, the operation unit 1004 transmits operation information indicating the accepted user operation to the server device 1001 via the communication unit 1003. The information transmitted in this manner is received in S1101 as described above.

[0096] In S1106, the communication unit 1003 of the client device 1002 acquires the information transmitted from the server device 1001 in S1104 as described above. That is, the communication unit 1003 can acquire the virtual viewpoint image generated by the server device 1001 and the storage permission information. The communication unit 1003 transmits the acquired virtual viewpoint image to the video output unit 206. In addition, the communication unit 1003 transmits the storage permission information to the image storage unit 207.

[0097] The determination in S305 is performed in the same manner as in the first embodiment. If it is determined that there is an instruction to save the virtual viewpoint image, the process proceeds to S1107. If it is determined that there is no instruction to save the virtual viewpoint image, the process shown in FIG. 7B ends.

[0098] In S1107, the image saving unit 207 judges whether or not saving permission information indicating that the saving operation of the virtual viewpoint image is restricted has been sent from the server device 1001. If saving permission information indicating that the saving operation of the virtual viewpoint image is restricted has been sent from the server device 1001, the process proceeds to S307. If such information has not been sent, the process proceeds to S308. The process of S307 is performed in the same manner as in the first embodiment. In this example, the image saving unit 207 does not save the virtual viewpoint image. Also, in S308, the image saving unit 207 saves the virtual viewpoint image acquired in S1104 in the image storage unit 208.

[0099] In the above embodiment, the communication unit 1003 of the client device 1002 acquires the storage permission information together with the virtual viewpoint image in S1106. On the other hand, the image storage unit 207 may acquire the storage permission information by inquiring of the server device 1001 in S1107. In this manner, the server device 1001 may transmit the storage permission information in response to an inquiry from the client device 1002, instead of sending the storage permission information together with the virtual viewpoint image in S1104.

[0100] According to the above embodiment, the server device 1001 generates a virtual viewpoint image. Furthermore, the server device 1001 determines whether to restrict the operation of saving the virtual viewpoint image. When the three-dimensional model of the subject is stationary, the user cannot save the virtual viewpoint image in the client device 1002. This makes it difficult to restore the three-dimensional shape of the subject based on the virtual viewpoint image.

[0101] As in the first embodiment, the predetermined condition is not limited to the volumetric video being stopped. As in the first modification, the save control unit 209 may limit the save operation of the virtual viewpoint image by the image save unit 207 in accordance with the designated virtual viewpoint. This process can be realized by determining in S306 whether or not the save control unit 209 limits the save operation of the virtual viewpoint image in the same manner as in the first modification.

[0102] Also, similarly to the above-mentioned modified example 2, the saving control unit 209 may limit the saving operation of the virtual viewpoint image by the image saving unit 207 depending on the number of times the virtual viewpoint image is saved. In such an example, the image saving unit 207 can transmit the image recording information to the server device 1001 via the communication unit 1003 in S308. The image saving unit 207 can receive the image recording information via the communication unit 1003. The image saving unit 207 may store the received image recording information in an image recording information saving unit (not shown). Also, the saving control unit 209 can determine whether or not to limit the saving operation of the virtual viewpoint image in S306, similarly to the modified example 2. At this time, the saving control unit 209 can determine whether or not a predetermined condition regarding the number of times the virtual viewpoint image is saved is satisfied, by referring to the image recording information received from the image saving unit 207.

[0103] Moreover, similarly to the first embodiment, the conditions for restricting the storage of virtual viewpoint images can be used in combination.

[0104] (Variation 4) The configuration of the image processing system 1000 is not limited to that shown in Fig. 6. For example, the connection between the server device 1001 and the client device 1002 is not limited to a one-to-one connection. For example, a plurality of client devices 1002 may be connected to the server device 1001.

[0105] Furthermore, the server device 1001 may operate on multiple physical devices. In this case, the server device 1001 may be recognized as a single logical device by the client device 1002. Furthermore, the server device 1001 may be realized using a virtual service such as cloud technology. In this case, the server device 1001 may have multiple logical function units that perform the above-mentioned processes using software technology.

[0106] At this time, the multiple client devices 1002 may view volumetric video in a common virtual space. Also, the playback state of the volumetric video (e.g., playback, stop, or time code of the display target) may be common among the multiple client devices 1002. For example, when an operation to stop the volumetric video is performed by one client device 1002, the volumetric video may also be stopped by the other client devices 1002. Similarly, the multiple client devices 1002 may view a virtual viewpoint video from a commonly set virtual viewpoint. For example, when an operation to specify a virtual viewpoint is performed by one client device 1002, the virtual viewpoint image displayed on the other client devices 1002 may change according to the specified virtual viewpoint.

[0107] Also, when a first user performs an operation to save a virtual viewpoint image, another second user may perform an operation to stop the volumetric video. In such a case, in S306, the saving control unit 209 can determine that the volumetric video is stopped. Then, the saving control unit 209 can determine that the first user is not permitted to save the virtual viewpoint image.

[0108] In addition, the storage control unit 209 may control the time designation unit 203 so as not to accept an instruction to stop the volumetric video when parameters of a virtual viewpoint designated by at least one of the multiple client devices 1002 satisfy a first condition.

[0109] Hereinafter, a process of determining whether or not to limit the saving operation of the virtual viewpoint image in the same manner as in the modified example 2 when a plurality of client devices 1002 are connected to the server device 1001 will be described. In this example, the virtual viewpoint image is saved according to a user instruction by each of a plurality of users. Meanwhile, the saving control unit 209 can receive the image recording information via the communication unit 1003 as described above, and can store the received image recording information in an image recording information saving unit (not shown). Here, the saving control unit 209 can store the image recording information for all the client devices 1002. Also, the saving control unit 209 can store the image recording information in association with the user of the client device 1002. For example, the saving control unit 209 can store the image recording information together with user information such as a user ID.

[0110] As described above, in S306, the saving control unit 209 can determine whether or not to limit the saving operation of the virtual viewpoint image according to the number of times the virtual viewpoint image is saved. Here, the number of times the virtual viewpoint image is saved may be the total number of times the virtual viewpoint image is saved according to the user instructions by each of the multiple users. That is, in S306, the saving control unit 209 can determine whether or not to limit the saving operation of the virtual viewpoint image according to the total number of times the virtual viewpoint image is saved for all users. In other words, the saving control unit 209 can cross-sectionally check for all users whether or not the number of times the virtual viewpoint image for a specific time code is saved is equal to or greater than a predetermined number.

[0111] For example, assume that the number of times that a virtual viewpoint image can be saved at the same time code is 10, and that user A has already saved five virtual viewpoint images for a specific time code, and user B has already saved five virtual viewpoint images. In this case, even if user C performs an operation to save another virtual viewpoint image for the same time code, user C cannot save the virtual viewpoint image because a predetermined number of virtual viewpoint images have already been saved by users A and B. In another example, if a virtual viewpoint image can only be saved up to 10 times within 15 minutes in real time, and users A and B have saved a total of 10 virtual viewpoint images, user C will wait 15 minutes before saving a virtual viewpoint image. Note that the time is not limited to 15 minutes and may be longer or shorter. Also, the number of times that a virtual viewpoint image can be saved is not limited to 10 times and may be shorter or longer.

[0112] Furthermore, the predetermined condition may include a condition for each user. Also, the predetermined condition may include a condition for each user in addition to the condition for all users. For example, the number of times that can be saved for all users may be set to 10 or less, and the number of times that can be saved for each user may be set to 5 or less. In this way, the upper limit of the number of times that can be saved for each user may be different from the upper limit of the number of times that can be saved for all users.

[0113] [Embodiment 3] An image processing device according to an embodiment will be described below with reference to the drawings. In the following embodiment, the image processing device 1600 generates and outputs a virtual viewpoint video represented by a plurality of virtual viewpoint images, similar to the image processing device 100 according to the first embodiment. The image processing device 1600 according to this embodiment restricts a user operation on a video when a predetermined condition related to a communication method of the video to an output device is satisfied. For example, the image processing device 1600 can restrict a user operation on a video when the video transmitted to the output device is not encrypted. First, a configuration for restricting an operation to stop a volumetric video when an output video signal is not encrypted will be described below.

[0114] 8 is a schematic diagram showing an example of a functional configuration of an image processing device 1600 according to this embodiment. The hardware configuration of the image processing device 1600 may be similar to that of the image processing device 100. The image processing device 1600 includes a model acquisition unit 201, a space construction unit 202, a time designation unit 203, a viewpoint designation unit 204, a video generation unit 205, a video output unit 206, an encryption unit 1601, and an operation restriction unit 1602.

[0115] The same functional units as those in the first embodiment are given the same reference symbols. For example, the video generation unit 205 can acquire a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time. Also, the video output unit 206 can output the video represented by the plurality of virtual viewpoint images for display by an output device. The following describes the differences from the first embodiment.

[0116] The encryption unit 1601 can encrypt the virtual viewpoint video generated by the video generation unit 205. The encryption unit 1601 can encrypt a video signal indicating the virtual viewpoint video according to a setting by a user. The encryption method is not particularly limited, but for example, the HDCP method can be used. As the encryption method, a method that makes it impossible for a device that receives the video signal to record the video signal may be used. Also, as the encryption method, a method that makes it impossible for a device that receives the video signal to reproduce the original video signal based on the video signal recorded may be used.

[0117] The operation limiting unit 1602 limits user operations on the video when a predetermined condition regarding a communication method of the video to the output device is satisfied. In this example, the operation limiting unit 1602 limits user operations on the video depending on whether or not the virtual viewpoint video output by the image processing device is encrypted. For example, the operation limiting unit 1602 can limit user operations on the video depending on whether or not it is determined that the virtual viewpoint video output by the image processing device is not encrypted. Also, the operation limiting unit 1602 can determine not to limit user operations on the video depending on whether or not it is determined that the virtual viewpoint video output by the image processing device is encrypted. In this example, the operation limiting unit 1602 rejects an operation to stop the volumetric video when the virtual viewpoint video output by the image processing device is not encrypted. According to such a configuration, the volumetric video cannot be stopped. However, the operation limiting unit 1602 may limit user operations on the video depending on whether or not some communication method that makes it difficult to use the output video is used.

[0118] Fig. 9 is a flowchart showing a processing flow of an image processing method performed by the image processing device 1600 according to one embodiment. A virtual space image is generated according to the flowchart shown in Fig. 9. Moreover, a virtual space video is generated and output by repeating the flowchart shown in Fig. 9.

[0119] In S1701, the time designation unit 203 determines whether or not a volumetric video stop instruction has been received from the user. If a volumetric video stop instruction has not been received, the process proceeds to S301. If a volumetric video stop instruction has been received, the process proceeds to S1702.

[0120] In S1702, the operation restriction unit 1602 determines whether or not the video signal has been encrypted by the encryption unit 1601. If the video signal has been encrypted, the process proceeds to S1703. If the video signal has not been encrypted, the process proceeds to S1704.

[0121] In S1703, the operation restriction unit 1602 determines to stop the volumetric video in accordance with a user instruction. That is, in the following S301, the time designation unit 203 sets the time code to a fixed value. Note that in the following S1701, the time designation unit 203 can resume playback of the volumetric video in response to determining that a playback instruction (or a pause release instruction) for the volumetric video has been received from the user. In this case, in the following S301, the time designation unit 203 counts up the time code.

[0122] In S1704, the operation limiting unit 1602 determines not to stop the volumetric video according to the user's instruction. That is, despite receiving the instruction to stop the volumetric video, the time designation unit 203 counts up the time code in the following S301. In this case, the operation limiting unit 1602 may notify the user that the user's operation is restricted. For example, the operation limiting unit 1602 can notify the user who has issued the instruction to stop the volumetric video that the volumetric video cannot be stopped. Specifically, the operation limiting unit 1602 can control the video output unit 206 to display a UI indicating that stopping cannot be performed on the output device that displays the virtual viewpoint video.

[0123] Steps S301 to S304 are performed in the same manner as in the first embodiment.

[0124] Note that the method of restricting user operations on video when a predetermined condition regarding the communication method of video to the output device is satisfied is not limited to the above method. For example, the encryption / decryption of the virtual viewpoint video may be dynamically switched. Specifically, the operation restriction unit 1602 may control the encryption unit 1601 to encrypt the output virtual viewpoint video in response to an instruction to stop the volumetric video. According to such a configuration, the encrypted virtual viewpoint video is output when the volumetric video is stopped. The operation restriction unit 1602 may control the encryption unit 1601 to decrypt the virtual viewpoint video when resuming the playback of the volumetric video. According to such a configuration, the operation of stopping the volumetric video when the virtual viewpoint video is not encrypted can also be restricted.

[0125] Furthermore, when a stop instruction for the volumetric video is received in a state where the virtual viewpoint image is not encrypted, the operation restriction unit 1602 can control the video generation unit 205 or the video output unit 206 to stop the output of the virtual viewpoint image. When the operation restriction unit 1602 rejects the user operation in this way, the operation restriction unit 1602 may control the output device to output a video prepared in advance. For example, in this case, the video output unit 206 can output a video different from the volumetric video, such as a black screen. Furthermore, the video output unit 206 can notify the user who has issued the stop instruction for the volumetric video that the volumetric video cannot be stopped. While the output of the volumetric video is stopped, the video output unit 206 can display a UI indicating that the video cannot be output on the output device.

[0126] According to the above embodiment, when the volumetric video is stopped, the unencrypted virtual viewpoint image is not output. The user may try to output the virtual viewpoint image including the virtual viewpoint images from various viewpoints of the three-dimensional model corresponding to the same time code by stopping the volumetric video and moving the virtual viewpoint. However, since the virtual viewpoint image output from the image processing device 1600 is encrypted, it is difficult to restore the three-dimensional shape of the subject based on the output virtual viewpoint image.

[0127] (Variation 5) The method in which the operation limiting unit 1602 limits user operations on video is not limited to the above method. When a predetermined condition related to a communication method of the video to the output device is satisfied, the operation limiting unit 1602 can limit user operations that comply with the predetermined condition. As the predetermined condition related to the user operations, for example, the same condition as that of the first modification can be used. Specifically, the operation limiting unit 1602 can reject a user operation that specifies a virtual viewpoint that satisfies a first condition in response to the determination that the video transmitted to the output device is not encrypted. According to such a configuration, the virtual viewpoint video from the virtual viewpoint that satisfies the first condition is not output.

[0128] For example, the operation restriction unit 1602 can acquire information indicating whether or not the virtual viewpoint image is encrypted from the encryption unit 1601. Furthermore, the operation restriction unit 1602 can acquire a user instruction for operating the virtual viewpoint. The operation restriction unit 1602 can determine whether or not to permit operation of the virtual viewpoint according to the information. For example, when the virtual viewpoint image is not encrypted and the parameters of the virtual viewpoint according to the user instruction satisfy a predetermined condition, the operation restriction unit 1602 can control the viewpoint designation unit 204 so as not to accept operation of the virtual viewpoint according to the user instruction.

[0129] 10 is a flowchart showing the processing flow of the image processing method performed by the image processing device 1600 according to one embodiment in this modified example. S301 and S302 are performed in the same manner as in the first embodiment.

[0130] In S1901, the operation restriction unit 1602 determines whether or not a user instruction to operate the virtual viewpoint has been input. If a user instruction has been input, the process proceeds to S1702, and if no operation has been performed, the process proceeds to S304.

[0131] In S1702, the operation restriction unit 1602 determines whether or not the video signal has been encrypted by the encryption unit 1601. If the video signal has been encrypted, the process proceeds to S1903. If the video signal has not been encrypted, the process proceeds to S1902.

[0132] In S1902, the operation restriction unit 1602 determines whether the parameters of the virtual viewpoint satisfy a predetermined condition, as in Modification 1. If the parameters of the virtual viewpoint satisfy the predetermined condition, the process proceeds to S1904. If not, the process proceeds to S1903.

[0133] In S1903, the operation restriction unit 1602 permits the operation of the virtual viewpoint in accordance with the user's instruction. Then, the viewpoint designation unit 204 sets the virtual viewpoint in accordance with the user's instruction. In S1904, the operation restriction unit 1602 rejects the operation of the virtual viewpoint in accordance with the user's instruction. In this case, the viewpoint designation unit 204 does not change the virtual viewpoint.

[0134] S304 is performed in the same manner as in embodiment 1. That is, the video generation unit 205 generates a virtual viewpoint image according to the virtual viewpoint set in S1902 or a virtual viewpoint that has been set previously.

[0135] According to the above embodiment, in a state where the virtual viewpoint video is not encrypted, the output of the virtual viewpoint video from the virtual viewpoint conforming to a predetermined condition is restricted. For example, when the virtual viewpoint video is not encrypted, the user cannot set the virtual viewpoint outside the permitted range (for example, the photographing permitted area 701). In this way, the virtual viewpoint corresponding to the output virtual viewpoint image is limited. On the other hand, when the virtual viewpoint is outside the permitted range, since the virtual viewpoint video output from the image processing device 1600 is encrypted, it is difficult to restore the three-dimensional shape of the subject based on the output virtual viewpoint image.

[0136] As in the above embodiment, when a virtual viewpoint image is not encrypted and a predetermined condition is satisfied, the operation restriction unit 1602 may control the video generation unit 205 or the video output unit 206 to stop outputting the virtual viewpoint image. For example, when a virtual viewpoint image is not encrypted and a parameter of the virtual viewpoint satisfies a predetermined condition, the operation restriction unit 1602 can stop outputting the virtual viewpoint image.

[0137] So far, the conditions under which the output of the virtual viewpoint video is restricted have been described. As in the third modification, these conditions can be used in combination. Furthermore, the manner of restriction by the operation restriction unit 1602 is not limited to the above. For example, as in the third modification, when the output of the virtual viewpoint image is restricted, the operation restriction unit 1602 may reduce the image quality of the virtual viewpoint image output from the video output unit 206.

[0138] [Embodiment 4] An image processing system 2000 according to an embodiment will be described below with reference to Fig. 11. Fig. 11 shows an example of the functional configuration of the image processing system 2000. The image processing system 2000 includes a server device 2001 and a client device 2002, similar to the second embodiment. Also, similar to the second embodiment, the server device 2001 renders a volumetric video, and a virtual viewpoint video is sent to the client device 2002 via communication. The hardware configurations of the server device 2001 and the client device 2002 may be similar to the hardware configuration of the image processing device 100.

[0139] In this embodiment, similarly to the third embodiment, when a predetermined condition related to the video communication method between the client device 2002 and the output device is satisfied, a user operation on the video is restricted. First, a configuration for restricting a user operation to stop the volumetric video when the video signal to be output is not encrypted will be described below.

[0140] The server device 2001 has a model acquisition unit 201, a space construction unit 202, a time designation unit 203, a viewpoint designation unit 204, a video generation unit 205, an operation restriction unit 1602, and a communication unit 1003. The client device 2002 has a video output unit 206, a communication unit 1003, an operation unit 1004, and an encryption unit 1601. The same functional units as those in the second and third embodiments are denoted by the same reference symbols. The differences from the second and third embodiments will be described below.

[0141] The operation restriction unit 1602 acquires information indicating a communication method of video between the client device 2002 and the output device. In this embodiment, the operation restriction unit 1602 can acquire information indicating whether or not the virtual viewpoint video has been encrypted from the encryption unit 1601 via the communication unit 1003. In addition, the operation restriction unit 1602 can control the time designation unit 203 so as to restrict a user operation to stop the volumetric video.

[0142] Figures 12(A) and (B) are flowcharts showing the processing flow of an image processing method performed by an image processing system 2000 according to an embodiment. A virtual space image is generated and output by repeating the flowcharts shown in Figures 12(A) and (B). Figure 12(A) shows the processing performed by a server device 2001. Also, Figure 12(B) shows the processing performed by a client device 2002.

[0143] S1101 is performed in the same manner as in the second embodiment. In this embodiment, an instruction to stop the volumetric video can be received from the user in S1101. Furthermore, S1701 to S1704 are performed in the same manner as in the third embodiment. That is, if the virtual viewpoint video has been encrypted, the operation restriction unit 1602 determines in S1703 to stop the volumetric video. Furthermore, if the virtual viewpoint video has not been encrypted, the operation restriction unit 1602 determines in S1704 not to stop the volumetric video.

[0144] Steps S301 to S304 are performed in the same manner as in embodiment 2. In step S2102, the video generation unit 205 transmits the virtual viewpoint image generated in step S304 to the client device 2002 via the communication unit 1003.

[0145] S1105 is performed in the same manner as in embodiment 3. In S2103, the encryption unit 1601 transmits information (encryption state information) indicating whether or not the virtual viewpoint video has been encrypted to the operation restriction unit 1602 via the communication unit 1003. In S2104, the communication unit 1003 of the client device 1002 acquires the virtual viewpoint image generated by the server device 1001. Thereafter, the video output unit 206 can output the virtual viewpoint image to the output device for display.

[0146] Even in the above embodiment, when the volumetric video is stopped, the unencrypted virtual viewpoint video is not output, so it is difficult to restore the three-dimensional shape of the subject based on the virtual viewpoint images from various viewpoints obtained by stopping the volumetric video and moving the virtual viewpoint.

[0147] (Variation 6) As in the third embodiment, the image processing system 2000 can restrict user operations on video that comply with predetermined conditions when a predetermined condition related to a method of communicating the video to an output device is satisfied. Here, the predetermined condition is not limited to the above. For example, as in the fifth modification, the predetermined condition may be a condition related to a virtual viewpoint designated by a user operation.

[0148] Fig. 13 is a flowchart showing the process flow of the image processing method in this modified example, which is performed by the server device 2001. The client device 2002 can operate according to the flowchart in Fig. 12(B).

[0149] S1101, S301, and S302 are performed in the same manner as in FIG. 12(A). That is, in S1101, a user instruction for operating the virtual viewpoint can be received. Also, S1901, S1702, and S1902 to S1904 are performed in the same manner as in Modification 5. That is, if the virtual viewpoint video has been encrypted, in S1903, the viewpoint designation unit 204 sets the virtual viewpoint according to the user instruction. Also, if the virtual viewpoint video has not been encrypted and the virtual viewpoint does not satisfy a predetermined condition, in S1903, the viewpoint designation unit 204 sets the virtual viewpoint according to the user instruction. On the other hand, if the virtual viewpoint video has not been encrypted and the virtual viewpoint satisfies a predetermined condition, in S1904, the viewpoint designation unit 204 does not accept the operation of the virtual viewpoint according to the user instruction.

[0150] Even with the above modification, if the virtual viewpoint video is not encrypted, the output of the virtual viewpoint video from the virtual viewpoint that complies with the predetermined conditions is restricted, making it difficult to restore the three-dimensional shape of the subject based on the output virtual viewpoint image.

[0151] As in the fourth modification, the configuration of the image processing system 2000 is not limited to that shown in Fig. 11. For example, the server device 2001 may operate on multiple physical devices or may operate on the cloud.

[0152] Also, a plurality of client devices 2002 may be connected to the server device 2001. Furthermore, similarly to the fourth modification, a plurality of client devices 1002 may watch a volumetric video in a common virtual space. In this case, the encryption setting may be different for each client device 2002. For example, a first user may encrypt a virtual viewpoint video, but a second user may not encrypt the virtual viewpoint video. In such a case, the operation restriction unit 1602 may determine whether to restrict a user operation on the video depending on whether all of the plurality of client devices 2002 encrypt the virtual viewpoint video output. For example, if at least one of the plurality of client devices 2002 does not encrypt the virtual viewpoint video output, the operation restriction unit 1602 may determine in S1704 not to stop the volumetric video. Note that the server device 2001 may reject a connection from a client device 2002 that is not set to encrypt the virtual viewpoint video to be output.

[0153] As in the third embodiment, the encryption / non-encryption of the virtual viewpoint video may be dynamically switched. For example, the operation restriction unit 1602 may notify the encryption unit 1601 to encrypt the virtual viewpoint video to be output in response to an instruction to stop the volumetric video. In this case, when the encryption unit 1601 starts encrypting the virtual viewpoint video, it can transmit information indicating that the virtual viewpoint video is being encrypted to the operation restriction unit 1602. Upon receiving such information, the operation restriction unit 1602 can control the time designation unit 203 to stop the volumetric video. Furthermore, the operation restriction unit 1602 may notify the encryption unit 1601 to decrypt the virtual viewpoint video when the playback of the volumetric video is resumed.

[0154] (Other embodiments) In the third and fourth embodiments, a configuration has been described in which a user operation to stop the volumetric video is restricted when a predetermined condition regarding a communication method of the video to the output device is satisfied. In this way, the operation restriction unit 1602 can restrict the output of the video when the volumetric video is stopped according to a predetermined condition. Here, the predetermined condition is not limited to when the video signal is not encrypted. For example, the operation restriction unit 1602 may restrict the video output operation when the volumetric video is stopped according to a virtual viewpoint. Specifically, as in the first modification, the operation restriction unit 1602 may restrict the video output operation when the volumetric video is stopped when the parameters of the virtual viewpoint satisfy a predetermined condition. Furthermore, the operation restriction unit 1602 may control the viewpoint designation unit 204 to reject the operation of the virtual viewpoint when the volumetric video is stopped so that virtual viewpoint videos from various viewpoints are not output when the volumetric video is stopped.

[0155] Also, in Modification 1, when the parameters of the virtual viewpoint satisfy a predetermined condition, the time designation unit 203 may not accept a user instruction to stop the volumetric video. Similarly, in Modification 4, when the parameters of the virtual viewpoint operated by at least one of the multiple client devices 1002 satisfy a predetermined condition, the time designation unit 203 may not accept a user instruction to stop the volumetric video. Also, in Modifications 2 and 4, when a predetermined condition related to the number of times the virtual viewpoint is saved is satisfied, the time designation unit 203 may not accept a user instruction to stop the volumetric video. In these cases as well, when the predetermined condition is satisfied, the output of the image while the volumetric video is stopped is restricted.

[0156] In addition, in the first and second embodiments, a configuration for restricting the saving operation of the virtual viewpoint image when a predetermined condition is satisfied has been described. Thus, the image processing device according to one embodiment has the video generation unit 205 or the communication unit 1003 that acquires a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time. Also, the image processing device has the video output unit 206 or the communication unit 1003 that outputs a video represented by the plurality of virtual viewpoint images. Here, the video output unit 206 or the communication unit 1003 can output a video in a first mode in which the plurality of virtual viewpoint images of the three-dimensional model corresponding to different times are sequentially output so that a volumetric video is played. Also, the video output unit 206 or the communication unit 1003 can output a video in a second mode in which the plurality of virtual viewpoint images of the three-dimensional model corresponding to a specific time are sequentially output so that the volumetric video is stopped. The image processing device has a save control unit 209 or an operation restriction unit 1602 that, when certain conditions are satisfied, permits image output in the first mode while restricting the image output operation or the saving operation of the virtual viewpoint image in the second mode.

[0157] According to such a configuration, when a predetermined condition is satisfied, it is possible to restrict the stopping of the volumetric video or the saving of the virtual viewpoint image. Therefore, it is possible to make it difficult to restore the three-dimensional shape of the subject based on the stopped volumetric video or based on the saved virtual viewpoint image. For example, it is possible to make it difficult to restore the three-dimensional shape of the subject based on the virtual viewpoint images from a plurality of directions output while the volumetric video is stopped. It is also possible to make it difficult to restore the three-dimensional shape of the subject based on the saved virtual viewpoint images from a plurality of directions. On the other hand, even when the predetermined condition is satisfied, it is possible to permit the playback of the volumetric video. Therefore, it is possible to improve the user experience when watching the volumetric video.

[0158] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0159] The disclosure of this specification includes the following image processing device, image processing method, and program. (Item 1) an acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting an image represented by the plurality of virtual viewpoint images for display; A control means for limiting a saving operation of the displayed virtual viewpoint image in accordance with a user instruction when a predetermined condition is satisfied; An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the control means prohibits saving of the virtual viewpoint image in response to determining that the predetermined condition is satisfied. (Item 3) 2. The image processing device according to claim 1, wherein the control means reduces the quality of the virtual viewpoint image to be stored in response to the determination that the predetermined condition is satisfied. (Item 4) 4. The image processing device according to claim 1, wherein the control means performs control to reduce the virtual viewpoint image to be stored in response to determining that the predetermined condition is satisfied. (Item 5) The image processing device described in item 1, 3, or 4, characterized in that the control means controls the resolution of the virtual viewpoint image to be stored in response to determining that the specified condition is satisfied so that a portion corresponding to one component of the three-dimensional model is smaller than one pixel on the virtual viewpoint image. (Item 6) the output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to a designated time; 6. The image processing device according to any one of items 1 to 5, wherein the predetermined condition includes the same time being continuously specified. (Item 7) 7. The image processing device according to item 6, wherein the control means performs control to continuously update the specified time in response to a user instruction to save the virtual viewpoint image. (Item 8) 6. The image processing device according to any one of items 1 to 5, wherein the predetermined condition includes satisfying a first condition regarding the virtual viewpoint. (Item 9) the output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to a designated time; 9. The image processing device according to item 8, wherein the control means prohibits the same time from being continuously specified in response to determining that the first condition is satisfied. (Item 10) 6. The image processing device according to any one of items 1 to 5, wherein the predetermined condition includes satisfying a second condition related to the number of times the virtual viewpoint image is saved. (Item 11) The image processing device according to item 10, characterized in that the second condition includes a condition regarding the number of times a virtual viewpoint image of the three-dimensional model is saved at a specific time, or a condition regarding the number of times the virtual viewpoint image is saved within a specified period of real time. (Item 12) The virtual viewpoint image is stored in accordance with a user instruction by each of a plurality of users; 12. The image processing device according to item 10 or 11, wherein the number of times the virtual viewpoint image is saved is a total number of times the virtual viewpoint image is saved in accordance with the user instruction by each of the multiple users. (Item 13) 13. The image processing device according to any one of items 1 to 12, wherein the control means notifies a user that a restriction has been imposed on the saving operation of the virtual viewpoint image. (Item 14) an acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting an image represented by the plurality of virtual viewpoint images for display by an output device; a control means for restricting a user operation on the image when a predetermined condition related to a communication method of the image to the output device is satisfied; An image processing device comprising: (Item 15) Item 15. The image processing device according to item 14, characterized in that the control means restricts user operations on the image in response to determining that the image to be transmitted to the output device is not encrypted. (Item 16) the output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to a designated time; The image processing device described in any one of items 14 to 15, characterized in that the control means performs control to reject user operations that repeatedly specify the same time when it is determined that the image to be transmitted to the output device is not encrypted. (Item 17) The image processing device described in any one of items 14 to 16, characterized in that the control means performs control to reject a user operation that specifies the virtual viewpoint that satisfies a first condition in response to determining that the image to be transmitted to the output device is not encrypted. (Item 18) 18. The image processing device according to any one of items 16 to 17, wherein the control means performs control so as to output a previously prepared image to the output device when the user operation is rejected. (Item 19) 19. The image processing device according to any one of items 14 to 18, wherein the control means notifies the user that the user operation is restricted. (Item 20) A time designation means for designating a time; A viewpoint designation means for designating a virtual viewpoint, 20. The image processing device according to any one of items 1 to 19, wherein the acquisition means generates an image of the three-dimensional model from the specified virtual viewpoint corresponding to the specified time as the virtual viewpoint image. (Item 21) an acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting a video image represented by the plurality of virtual viewpoint images; a control means for permitting video output in a first mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to different times are sequentially output, while restricting a video output operation in a second mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to a specific time are sequentially output, or a storage operation of the virtual viewpoint image, when a predetermined condition is satisfied; An image processing device comprising: (Item 22) An image processing method performed by an image processing device, comprising: acquiring a plurality of virtual viewpoint images, each image being from a specifiable virtual viewpoint of the three-dimensional model corresponding to a time instant; outputting for display an image represented by the plurality of virtual viewpoint images; restricting a saving operation of the displayed virtual viewpoint image in accordance with a user instruction to a case where a predetermined condition is satisfied; 13. An image processing method comprising: (Item 23) An image processing method performed by an image processing device, comprising: acquiring a plurality of virtual viewpoint images, each image being from a specifiable virtual viewpoint of the three-dimensional model corresponding to a time instant; outputting the video represented by the plurality of virtual viewpoint images for display by an output device; restricting user operations on the video when a predetermined condition related to a communication method of the video to the output device is satisfied; 13. An image processing method comprising: (Item 24) An image processing method performed by an image processing device, comprising: acquiring a plurality of virtual viewpoint images, each image being from a specifiable virtual viewpoint of the three-dimensional model corresponding to a time instant; outputting a video represented by the plurality of virtual viewpoint images; permitting video output in a first mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to different times are sequentially output, while restricting a video output operation in a second mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to a specific time are sequentially output, or a storage operation of the virtual viewpoint images, when a predetermined condition is satisfied; 13. An image processing method comprising: (Item 25) 22. A program for causing a computer to function as the image processing device according to any one of items 1 to 21.

[0160] The present disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present disclosure. Accordingly, the following claims are appended to disclose the scope of the present disclosure. [Explanation of symbols]

[0161] 201: model acquisition unit, 202: space construction unit, 203: time designation unit, 204: viewpoint designation unit, 205: video generation unit, 206: video output unit, 207: image storage unit, 208: image storage unit, 209: storage control unit

Claims

1. an acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting an image represented by the plurality of virtual viewpoint images for display; A control means for limiting a saving operation of the displayed virtual viewpoint image in accordance with a user instruction when a predetermined condition is satisfied; An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the control means prohibits the storage of the virtual viewpoint image in response to the determination that the predetermined condition is satisfied.

3. The image processing device according to claim 1 , wherein the control means reduces a quality of the virtual viewpoint image to be stored in response to the determination that the predetermined condition is satisfied.

4. The image processing device according to claim 1 , wherein the control means performs control to reduce the virtual viewpoint image to be stored in response to the determination that the predetermined condition is satisfied.

5. 5. The image processing device according to claim 4, wherein the control means controls a resolution of the virtual viewpoint image to be stored in response to determining that the predetermined condition is satisfied, so that a portion corresponding to one component of the three-dimensional model is smaller than one pixel on the virtual viewpoint image.

6. the output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to a designated time; The image processing device according to claim 1 , wherein the predetermined condition includes a condition that the same time is continuously specified.

7. 7. The image processing apparatus according to claim 6, wherein the control means performs control to continuously update the designated time in response to a user instruction to save the virtual viewpoint image.

8. The image processing device according to claim 1 , wherein the predetermined condition includes satisfying a first condition regarding the virtual viewpoint.

9. the output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to a designated time; 9. The image processing apparatus according to claim 8, wherein said control means prohibits the same time from being continuously specified in response to the determination that said first condition is satisfied.

10. The image processing device according to claim 1 , wherein the predetermined condition includes satisfying a second condition related to the number of times the virtual viewpoint image is saved.

11. The image processing device according to claim 10, characterized in that the second condition includes a condition regarding the number of times that a virtual viewpoint image of the three-dimensional model is saved at a specific time, or a condition regarding the number of times that the virtual viewpoint image is saved within a predetermined period of real time.

12. The virtual viewpoint image is stored in accordance with a user instruction by each of a plurality of users; The image processing device according to claim 10 , wherein the number of times the virtual viewpoint image is saved is a total number of times the virtual viewpoint image is saved in accordance with the user instructions by each of the plurality of users.

13. 2. The image processing apparatus according to claim 1, wherein said control means notifies a user that a restriction has been imposed on the saving operation of said virtual viewpoint image.

14. an acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting an image represented by the plurality of virtual viewpoint images for display by an output device; a control means for restricting a user operation on the image when a predetermined condition related to a communication method of the image to the output device is satisfied; An image processing device comprising:

15. 15. The image processing device according to claim 14, wherein the control means restricts user operations on the image when it is determined that the image to be transmitted to the output device is not encrypted.

16. the output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to a designated time; 15. The image processing device according to claim 14, wherein the control means performs control so as to reject a user operation of repeatedly specifying the same time when it is determined that the image to be transmitted to the output device is not encrypted.

17. 15. The image processing device according to claim 14, characterized in that the control means performs control so as to reject a user operation that specifies the virtual viewpoint that satisfies a first condition in response to a determination that the image to be transmitted to the output device is not encrypted.

18. 17. The image processing device according to claim 16, wherein the control means performs control so that a prepared image is output to the output device when the user operation is rejected.

19. 15. The image processing device according to claim 14, wherein the control means notifies the user that the user operation is restricted.

20. A time designation means for designating a time; A viewpoint designation means for designating a virtual viewpoint, 2 . The image processing apparatus according to claim 1 , wherein the acquisition means generates, as the virtual viewpoint image, an image of the three-dimensional model from the specified virtual viewpoint and corresponding to the specified time.

21. an acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting a video image represented by the plurality of virtual viewpoint images; a control means for permitting video output in a first mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to different times are sequentially output, while restricting a video output operation in a second mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to a specific time are sequentially output, or a storage operation of the virtual viewpoint image, when a predetermined condition is satisfied; An image processing device comprising:

22. An image processing method performed by an image processing device, comprising: acquiring a plurality of virtual viewpoint images, each image being from a specifiable virtual viewpoint of the three-dimensional model corresponding to a time instant; outputting for display an image represented by the plurality of virtual viewpoint images; restricting a saving operation of the displayed virtual viewpoint image in accordance with a user instruction to a case where a predetermined condition is satisfied; 13. An image processing method comprising:

23. An image processing method performed by an image processing device, comprising: acquiring a plurality of virtual viewpoint images, each image being from a specifiable virtual viewpoint of the three-dimensional model corresponding to a time instant; outputting the video represented by the plurality of virtual viewpoint images for display by an output device; restricting user operations on the video when a predetermined condition related to a communication method of the video to the output device is satisfied; 13. An image processing method comprising:

24. An image processing method performed by an image processing device, comprising: acquiring a plurality of virtual viewpoint images, each image being from a specifiable virtual viewpoint of the three-dimensional model corresponding to a time instant; outputting a video represented by the plurality of virtual viewpoint images; permitting video output in a first mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to different times are sequentially output, while restricting a video output operation in a second mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to a specific time are sequentially output, or a storage operation of the virtual viewpoint images, when a predetermined condition is satisfied; 13. An image processing method comprising:

25. A program for causing a computer to function as the image processing device according to any one of claims 1 to 21.

Images