Image processing device, image processing method, and program

Abstract

The present invention addresses a difficulty in restoring a three-dimensional shape of a subject on the basis of a plurality of virtual viewpoint images in a technology for providing a virtual viewpoint image.　The present invention outputs, to an external device, a plurality of virtual viewpoint images each being an image including a three-dimensional model corresponding to a certain time. The present invention restricts processing for storing a virtual viewpoint image displayed on the external device among the plurality of virtual viewpoint images if a predetermined condition is satisfied.

Description

Image processing device, image processing method, and program

[0001] The present disclosure relates to an image processing device, an image processing method, and a program, and in particular to a technology for generating a virtual viewpoint video.

[0002] In recent years, technologies for generating virtual viewpoint images have become known. For example, a technology called volumetric video has attracted attention. In this technology, multiple cameras are installed at different positions to perform synchronized shooting from multiple viewpoints. Images from multiple viewpoints obtained through such shooting can then be used to generate a virtual viewpoint image from any viewpoint. This technology allows users to view highlight scenes of sports such as soccer or basketball from any angle. It also allows users to view footage from within the field during a game, something that was previously not possible. In this way, users can view footage from various positions or angles, i.e., footage with added value, that differs from conventional footage.

[0003] Such virtual viewpoint images can be generated, for example, by a server acquiring images from multiple cameras and performing a series of processes, such as foreground / background separation, 3D shape estimation, 3D model generation, and rendering. Meanwhile, cloud technology has developed in recent years. The processing capabilities of information devices such as PCs and tablet terminals have also improved. Furthermore, advances in network infrastructure have led to improvements in communication speeds. For these reasons, virtual viewpoint operation and video viewing, which previously required dedicated devices, can now be performed using ordinary users' tablet terminals, PCs, and the like.

[0004] Japanese Patent Application Laid-Open No. 2007-325920

[0005] If a user can freely manipulate the virtual viewpoint, they can obtain virtual viewpoint images of the subject from multiple angles. Using these virtual viewpoint images, the user may be able to reconstruct the three-dimensional shape of the subject. For example, such manipulation can be performed using an application with a photogrammetry function. As a result, there is a possibility that the rights of the provider of virtual viewpoint video content may be infringed.

[0006] Japanese Patent Laid-Open No. 2009-102663 discloses a technique for limiting the object region from which a virtual viewpoint image is generated. According to the technique described in Japanese Patent Laid-Open No. 2009-102663, two images captured from different directions are displayed on a display device. When two points are specified on the two images, a region that can be reconstructed based on the specified two points is identified, and an image of this region is generated. However, this technique is insufficient to prevent the reconstruction of three-dimensional shape data of an object based on virtual viewpoint images of the object from multiple directions.

[0007] The present disclosure aims to make it difficult to restore the three-dimensional shape of a subject based on multiple virtual viewpoint images in a technology for providing virtual viewpoint images.

[0008] An image processing device according to one embodiment has the following configuration: an output unit that outputs a plurality of virtual viewpoint images, each of which is an image including a three-dimensional model corresponding to a certain time, to an external device, and a control unit that limits a process of saving a virtual viewpoint image displayed on the external device from among the plurality of virtual viewpoint images, when a predetermined condition is satisfied.

[0009] In a technology for providing virtual viewpoint images, it is possible to make it difficult to restore the three-dimensional shape of a subject based on a plurality of virtual viewpoint images.

[0010] Other features and advantages of the present disclosure will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar components are designated by the same reference numerals.

[0011] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments of the present disclosure, and are used, together with the description thereof, to explain the principles of the present disclosure. A block diagram illustrating an example of the hardware configuration of an image processing device according to an embodiment. A block diagram illustrating an example of the functional configuration of an image processing device according to an embodiment. A flowchart of an image processing method according to an embodiment. A diagram for explaining generation of a virtual viewpoint image. A diagram for explaining generation of a virtual viewpoint image. A diagram for explaining generation of a virtual viewpoint image. A diagram for explaining conditions related to parameters of a virtual viewpoint. A diagram for explaining conditions related to parameters of a virtual viewpoint. A block diagram illustrating an example of the functional configuration of an image processing system according to an embodiment. A flowchart of an image processing method according to an embodiment. A flowchart of an image processing method according to an embodiment. A block diagram illustrating an example of the functional configuration of an image processing device according to an embodiment. A flowchart of an image processing method according to an embodiment. A flowchart of an image processing method according to an embodiment. A block diagram illustrating an example of the functional configuration of an image processing system according to an embodiment. A flowchart of an image processing method according to an embodiment. A flowchart of an image processing method according to an embodiment.

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] An image processing apparatus according to an embodiment can be realized by a computer including a processor and a memory. Fig. 1 is a block diagram schematically illustrating an example of the hardware configuration of an image processing apparatus according to an embodiment. The image processing apparatus 100 includes a CPU 101, a main memory device 102, an auxiliary memory device 103, an external I / F 104, and a GPU 105. These components are connected to each other via a bus 106 so as to be able to communicate with each other.

[0014] The CPU 101 is a central processing unit that controls the entire image processing device 100. The CPU 101 can perform various calculations and decisions, image processing, and data input / output.

[0015] The main memory device 102 can function as a work area or a temporary data storage area for the CPU 101. The main memory device 102 can be implemented using a storage medium such as a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0016] The auxiliary storage device 103 can store information such as various programs, various setting information, various image data, camera parameters, three-dimensional shape data, or two-dimensional maps. The auxiliary storage device 103 can be implemented using a storage medium such as a non-volatile memory such as a read-only memory (ROM) or flash memory, a hard disk drive (HDD), a solid-state drive (SSD), or tape media. The auxiliary storage device 103 may be implemented using a combination of multiple storage media to increase capacity or speed. In this case, these storage media may be logically combined into one using RAID technology or the like.

[0017] The external I / F 104 is a general-purpose input / output interface. The external I / F 104 can communicate with external devices such as a camera, a PC, or a server. The external I / F 104 can also be connected to external devices that accept user operations, such as a joystick, a keyboard, or a mouse. The external I / F 104 can also be connected to external storage for file input / output. The external I / F 104 may be an interface with a physical cable connection terminal for InfiniBand, Ethernet (registered trademark), Universal Serial Bus (USB), or the like. The external I / F 104 may also be a wireless interface for a wireless LAN, Bluetooth, or the like.

[0018] The GPU 105 is a computing device for performing image processing calculations at high speed. The GPU 105 may have a function of outputting a video signal to an output device capable of displaying an external video, such as a television. The GPU 105 may also perform a process of rendering a virtual viewpoint image. The GPU 105 may also perform a process of generating shape data or color information necessary for generating a virtual viewpoint image. The GPU 105 may have a video terminal such as DVI, HDMI (registered trademark), DisplayPort, or SDI to output a video signal. The GPU 105 may include a main memory device different from the main memory device 102. The GPU 105 may also perform some or all of the various calculations performed by the CPU 101 other than the above-described processes. The GPU 105 may not have a function of outputting a video signal. For example, the GPU 105 may only perform calculations such as shape data or color information necessary for image processing or generating a virtual viewpoint image. In this case, the external I / F 104 may further have a video output function or a function to output data necessary for video output.

[0019] The bus 106 is configured using a general-purpose bus such as PCI Express, allowing each hardware block to communicate bidirectionally. While all hardware blocks are connected to the same bus 106 in FIG. 1 , this configuration is not limited thereto. For example, the main memory device 102 and the CPU 101 may be connected via a bus different from the bus 106. In this case, the main memory device 102 may communicate with other blocks via the CPU 101. Furthermore, hardware blocks such as the auxiliary memory device 103 or the GPU 105 may be connected to the external I / F 104. In this case, communication with the auxiliary memory device 103 or the GPU 105 can be performed via the external I / F 104. The type of the bus 106 is not limited. Any bus that allows each block to communicate bidirectionally can be used as the bus 106. For example, the bus 106 may be a bus conforming to a standardized communication standard such as USB. The bus 106 may also be a dedicated communication bus.

[0020] In this way, a processor such as the CPU 101 executes a program stored in a memory such as the main storage device 102 or the auxiliary storage device 103, thereby realizing the functions of each unit shown in FIG. 2 (described later). Note that the configuration shown in FIG. 1 is merely one example of the configuration of the image processing device 100. For example, some or all of the functions of the image processing device 100 may be realized by dedicated hardware. Furthermore, some of the functions of the image processing device 100 may be realized by externally connected hardware. Furthermore, the main storage device 102 and the auxiliary storage device 103 may be realized by a single piece of hardware. Furthermore, other hardware such as an FPGA, a DSP, or a dedicated image processing LSI may be used instead of the GPU 105.

[0021] Furthermore, the image processing device 100 does not have to be configured as a single device. For example, a plurality of devices having similar configurations may be prepared. In this case, the configurations of the individual devices may be different. Furthermore, the image processing device according to an embodiment may be configured as a plurality of information processing devices connected via a network, for example. In an embodiment, the functions of the image processing device 100 may be provided as a cloud service.

[0022] [Embodiment 1] An image processing device according to one embodiment will be described below with reference to the drawings. In the following embodiment, the image processing device 100 generates and outputs a virtual viewpoint video represented by a plurality of virtual viewpoint images. However, the image processing device 100 may acquire the virtual viewpoint video from another device and output the acquired virtual viewpoint video. In the following embodiment, the image processing device 100 has a function of saving the virtual viewpoint images constituting the virtual viewpoint video. Such a saving function is also called an image capture function. That is, the image processing device 100 can save the virtual viewpoint image displayed on the output device in accordance with a user instruction. In the following example, the saved virtual viewpoint image is a two-dimensional image. Note that the saving method is not particularly limited. In one embodiment, the output virtual viewpoint image is saved. In another embodiment, a screenshot of the display on the output device is saved. In a further embodiment, a virtual viewpoint image based on the same time code and virtual viewpoint as the virtual viewpoint image displayed on the output device is generated by the video generation unit 205, and the generated virtual viewpoint image is saved.

[0023] Furthermore, in this embodiment, the saving operation of the virtual viewpoint image is restricted in accordance with predetermined conditions. First, a case where the saving operation of the virtual viewpoint image is restricted in a state where the playback of the virtual viewpoint video is stopped will be described below.

[0024] In the following example, the image processing device 100 can play volumetric video. The volumetric video is generated based on a three-dimensional model corresponding to each time. More specifically, the volumetric video is generated based on a three-dimensional model linked to a time code representing the time. The three-dimensional model can change over time. The image processing device 100 can generate an image of the three-dimensional model corresponding to a specific time from a virtual viewpoint as a virtual viewpoint image. Note that further image processing, such as adding a background, may be performed on the virtual viewpoint image.

[0025] As will be described later, the video output unit 206 of the image processing device 100 sequentially outputs virtual viewpoint images of a three-dimensional model corresponding to a specified time. Here, playing a volumetric video corresponds to continuously updating the specified time. That is, multiple virtual viewpoint images are generated based on a three-dimensional model corresponding to each different time. A virtual viewpoint video composed of these multiple virtual viewpoint images is played as a volumetric video. Hereinafter, sequentially displaying virtual viewpoint images generated based on a three-dimensional model corresponding to each successive time is referred to as playing a volumetric video.

[0026] Alternatively, multiple virtual viewpoint images may be generated based on a specific three-dimensional model corresponding to a specific time. In this specification, continuously displaying virtual viewpoint images generated based on a specific three-dimensional model corresponding to a specific time is referred to as stopping the volumetric video. Stopping the volumetric video corresponds to continuously specifying the same time. Stopping the volumetric video includes sequentially displaying multiple virtual viewpoint images generated based on a specific three-dimensional model corresponding to a specific time. These multiple virtual viewpoint images may be generated based on different virtual viewpoints. In this way, when the volumetric video is stopped, the video output unit 206 of the image processing device 100 can output a virtual viewpoint video composed of virtual viewpoint images of a three-dimensional model corresponding to the same time from various directions. Alternatively, stopping the volumetric video may include continuously displaying the same single virtual viewpoint image. Stopping the volumetric video also includes temporarily stopping the progression of the time code (i.e., pausing).

[0027] 2 is a schematic diagram showing an example of the functional configuration of an image processing device 100 according to an embodiment. The image processing device 100 includes a model acquisition unit 201, a space construction unit 202, a time designation unit 203, a viewpoint designation unit 204, a video generation unit 205, a video output unit 206, an image saving unit 207, an image storage unit 208, and a save control unit 209.

[0028] The model acquisition unit 201 acquires three-dimensional model data corresponding to each time. In this embodiment, the model acquisition unit 201 acquires a three-dimensional model of a volumetric video linked to a time code specified by the space configuration unit 202. The three-dimensional model may represent content such as a sport such as soccer, a concert, or a play. Specifically, the three-dimensional model may represent the three-dimensional shape of an object. The three-dimensional model may also represent the texture of the object. The model acquisition unit 201 may acquire a three-dimensional model for each of multiple objects. For example, the three-dimensional model may represent the three-dimensional shape of a person such as a player or performer, or an object such as a ball. The three-dimensional model may also represent the three-dimensional shape of a background structure such as a stadium or stage equipment. Note that some objects may not change over time. For example, a background structure may be stationary regardless of the time code. The three-dimensional model corresponding to such an object may be linked to multiple time codes. Furthermore, a range of consecutive time codes may be linked to one three-dimensional model.

[0029] The model acquisition unit 201 may acquire these three-dimensional models via a network such as the Internet or a local area network (LAN). The model acquisition unit 201 may read these three-dimensional models from a storage medium such as optical media, a USB memory, a HDD, or a magnetic tape medium. The model acquisition unit 201 may acquire three-dimensional models associated with all time codes at once. Alternatively, the model acquisition unit 201 may sequentially acquire three-dimensional models necessary for the space construction unit 202 to construct a virtual space for a specific time code.

[0030] The space construction unit 202 constructs a three-dimensional virtual space. The space construction unit 202 can construct a virtual space corresponding to a specific time code designated by the time designation unit 203. Specifically, the space construction unit 202 can acquire a three-dimensional model linked to the specific time code from the model acquisition unit 201. Then, the space construction unit 202 can place the acquired three-dimensional model in the virtual space.

[0031] The time designation unit 203 designates a time. The video generation unit 205, which will be described later, generates a virtual viewpoint image of the three-dimensional model corresponding to the time designated by the time designation unit 203. In this embodiment, the time designation unit 203 can designate a time code. Then, the time designation unit 203 outputs the designated time code to the space construction unit 202 so as to construct a three-dimensional virtual space corresponding to the time code.

[0032] During normal playback, the time designation unit 203 advances the time code as real time passes. For example, the time designation unit 203 can advance the time code so that it advances 60 frames per second. On the other hand, the time designation unit 203 can stop counting up the time code. In this case, the volumetric video being output or displayed stops. Note that the number of frames per second during normal playback is not limited to 60 frames. The number of frames per second may be, for example, 50 frames, 30 frames, 24 frames, 59.94 frames, 29.97 frames, or 23.976 frames.

[0033] The time designation unit 203 can control the time code in response to an instruction from the user. For example, the time designation unit 203 can stop counting up the time code when receiving a pause instruction from the user. Furthermore, the time designation unit 203 can increase the number of frames counted up or down per second when receiving a fast-forward or fast-rewind instruction from the user. Furthermore, the time designation unit 203 can also control the time code for special playback. For example, the time designation unit 203 can accept an operation to directly designate a time code.

[0034] The viewpoint designation unit 204 designates a virtual viewpoint. The viewpoint designation unit 204 can set a virtual viewpoint within the virtual space constructed by the space construction unit 202. The virtual viewpoint corresponds to a virtual camera that captures images within the virtual space. The virtual camera is a concept for explaining the virtual viewpoint used to generate a virtual viewpoint image. A virtual camera is a virtual camera that is different from the multiple imaging devices actually installed around the imaging area to generate a three-dimensional model. In other words, the virtual viewpoint image can be considered to be an image captured from a virtual viewpoint set within a virtual space associated with the imaging area. In other words, the virtual viewpoint image can be said to be an image simulating an image captured by a camera if it were assumed that the camera exists at the position of the virtual viewpoint set within the virtual space. The video generation unit 205, which will be described later, generates an image of the three-dimensional model placed within the virtual space from the virtual viewpoint as the virtual viewpoint image.

[0035] The viewpoint designation unit 204 can operate the virtual viewpoint based on a user operation. For example, the viewpoint designation unit 204 can move the virtual viewpoint in a virtual space. The viewpoint designation unit 204 can also set parameters for the virtual viewpoint. Specifically, the viewpoint designation unit 204 can perform camera operations such as zooming. The position and orientation of the virtual viewpoint thus set correspond to the position and orientation of the virtual camera. In this way, a virtual viewpoint image from any viewpoint can be generated.

[0036] In this embodiment, the viewpoint designation unit 204 sets camera parameters of a virtual camera corresponding to the virtual viewpoint. The camera parameters may include external parameters and internal parameters. The information included in the camera parameters is not particularly limited. The camera parameters may include, for example, parameters representing the three-dimensional position of the camera and information representing the three-dimensional orientation of the camera. The orientation of the camera can be represented by information representing rotation in the pan, tilt, and roll directions. The camera parameters may also include information such as the size of the field of view (angle of view), the number of pixels of the virtual sensor, or the focal length.

[0037] The method for setting the virtual viewpoint is not particularly limited. For example, the virtual viewpoint may be linked to a character or object in the virtual space, such as an avatar operated by a user. For example, a virtual camera may correspond to the field of view of the character. The virtual viewpoint may also be set so that its relative position with respect to these characters or objects is fixed. In this case, too, the virtual viewpoint can move in accordance with the movement of the character or object.

[0038] The video generation unit 205 can acquire multiple virtual viewpoint images, each of which is an image of a three-dimensional model from a specifiable virtual viewpoint corresponding to a certain time. As described above, the video generation unit 205 can generate, as a virtual viewpoint image, an image of a three-dimensional model from a virtual viewpoint specified by the viewpoint specification unit 204 and corresponding to a time specified by the time specification unit 203. For example, as described above, the video generation unit 205 can generate, as a virtual viewpoint image, an image of a three-dimensional model from a specified virtual viewpoint in a virtual space corresponding to a specific time code. The video generation unit 205 can generate the virtual viewpoint image as a two-dimensional image in a format viewable by a user. The video generation unit 205 can also sequentially generate virtual viewpoint images of a three-dimensional model corresponding to a specified time. For example, the video generation unit 205 can generate multiple virtual viewpoint images for virtual spaces each corresponding to a different time code. The video generation unit 205 can also generate multiple virtual viewpoint images for virtual spaces corresponding to the same time code. Furthermore, the video generation unit 205 can generate multiple virtual viewpoint images each corresponding to a different virtual viewpoint. In this way, the video generation unit 205 can generate a virtual viewpoint video made up of a plurality of virtual viewpoint images.

[0039] The video output unit 206 outputs a video represented by a plurality of virtual viewpoint images for display. The video output unit 206 can sequentially output the virtual viewpoint images of a three-dimensional model corresponding to a specified time, which are generated by the video generation unit 205. For example, the video output unit 206 can output a virtual viewpoint video composed of the plurality of virtual viewpoint images generated by the video generation unit 205 to an output device such as a display. In this case, the video output unit 206 may output the virtual viewpoint video via a video signal output terminal such as HDMI (registered trademark), DisplayPort, or SDI.

[0040] The image storage unit 207 stores virtual viewpoint images. The image storage unit 207 can store virtual viewpoint images in accordance with a user instruction. The image storage unit 207 can also store the displayed virtual viewpoint images. Specifically, the image storage unit 207 can store, among the virtual viewpoint images that constitute the virtual viewpoint video, the virtual viewpoint images that are output by the video output unit 206 or the virtual viewpoint images that are displayed on the output device.

[0041] The image saving unit 207 can store the virtual viewpoint image generated by the video generating unit 205 in the image storage unit 208. The image saving unit 207 can save the virtual viewpoint image as a two-dimensional image file in a format such as JPEG, BMP, PNG, or MPEG. The saving operation of the virtual viewpoint image by the image saving unit 207 is restricted by the saving control unit 209. In this example, the image saving unit 207 stores the virtual viewpoint image in the image storage unit 208 in response to the saving control unit 209 determining to permit saving.

[0042] The saving control unit 209 restricts the saving operation of the virtual viewpoint image when a predetermined condition is satisfied. In this example, the saving control unit 209 determines whether or not to permit saving of the virtual viewpoint image. The saving control unit 209 can prohibit saving of the virtual viewpoint image in response to determining that the predetermined condition is satisfied. In this way, the saving control unit 209 can determine whether or not it is permissible to save the virtual viewpoint image as a two-dimensional image file.

[0043] In this example, the predetermined condition is that the volumetric video is stopped. That is, the saving control unit 209 determines whether or not it is permitted to save the virtual viewpoint image depending on whether or not the volumetric video is stopped. The saving control unit 209 determines that it is permitted to save the virtual viewpoint image depending on whether or not it is determined that the volumetric video is being played. On the other hand, the saving control unit 209 determines that it is not permitted to save the virtual viewpoint image depending on whether or not it is determined that the volumetric video is stopped.

[0044] The save control unit 209 may notify the user that the saving operation of the virtual viewpoint image has been restricted. For example, if saving of the image is not permitted, the save control unit 209 can notify the user that the image cannot be saved. The save control unit 209 can notify the user via an output device that displays the virtual viewpoint image, etc.

[0045] 3 is a flowchart showing the processing flow of an image processing method performed by the image processing device 100 according to one embodiment. A virtual space image is generated according to the flowchart shown in FIG. 3. Furthermore, by repeating the flowchart shown in FIG. 3, multiple virtual space images are generated, i.e., a virtual space video is generated.

[0046] In S301, the time designation unit 203 sets a time code. If the volumetric video is stopped, the time designation unit 203 sets the time code to a fixed value. In this case, the configuration of the virtual space constructed in S302 is fixed. This allows the volumetric video to be stopped. On the other hand, if the volumetric video is to be played, the time designation unit 203 counts up the time code.

[0047] In S302, the space construction unit 202 acquires a three-dimensional model of the volumetric video corresponding to the time code designated by the time designation unit 203 from the model acquisition unit 201. Then, the space construction unit 202 constructs a virtual space as described above.

[0048] In S303, the viewpoint designation unit 204 sets the parameters of the virtual viewpoint in response to the user's operation as described above, thus determining the virtual viewpoint for the volumetric video.

[0049] In S304, the video generation unit 205 generates a virtual viewpoint image that can be viewed by the user in accordance with the parameters of the virtual viewpoint set in S303. The virtual viewpoint image thus generated may be displayed on an output device.

[0050] 4A and 4B are diagrams illustrating the generation of a virtual viewpoint image based on volumetric video data created by filming a soccer match. FIG. 4 schematically shows a virtual space at a certain time and how a virtual camera is used to film within this virtual space. FIG. 4A is a schematic diagram showing the state of the virtual space from a bird's-eye view above the stadium. A soccer match is being played on a soccer field 401. Spectator seats 402 are those of a soccer stadium. The spectator seats 402 are installed with seats similar to those in a soccer stadium in the real world.

[0051] The player 403 is a soccer player. The player 403 is represented as a three-dimensional model in the volumetric video. For the sake of explanation, the following will be described with reference to one player 403 on the soccer court 401. However, two or more players may exist in the virtual space. Typically, in a soccer match, there are 11 players per team, i.e., a total of 22 players from the two competing teams, plus referees on the soccer court 401.

[0052] The virtual cameras 404a and 404b are virtual cameras operated by the viewpoint designation unit 204. The virtual cameras 404a and 404b virtually capture images of the player 403. Note that these virtual cameras 404a and 404b may be invisible in the virtual space. For example, when the virtual camera 404b facing the virtual camera 404a captures an image, the virtual camera 404a does not need to appear in the image captured by the virtual camera 404b.

[0053] The virtual cameras 404a and 404b can move to any location, including the field where the soccer game is played. Furthermore, the virtual cameras 404a and 404b can move freely in three dimensions, up and down, left and right, and in the height direction. However, the positions of the virtual cameras 404a and 404b may be restricted. For example, the virtual cameras 404a and 404b may be limited to moving only within a predetermined area, such as the spectator seats 402. Furthermore, an area in the virtual space where the virtual cameras 404a and 404b cannot move may be set. Furthermore, as described above, the virtual viewpoint may be the viewpoint of an avatar placed in the virtual space. In this case, the movable range of the avatar may be limited to a predetermined area, such as the spectator seats 402. Furthermore, the movement of the virtual viewpoint in the height direction may be limited to a certain range. In this case, the certain range may be set according to the height of the avatar.

[0054] Fig. 4(B) is a schematic representation of an example of a virtual viewpoint image obtained by virtual camera 404a. Fig. 4(C) is a schematic representation of an example of a virtual viewpoint image obtained by virtual camera 404b. The video generator 205 can render these two-dimensional images so that the user can view them.

[0055] In S305, the image storage unit 207 determines whether or not the user has issued an instruction to save the virtual viewpoint image. If it is determined that the instruction to save the virtual viewpoint image has been issued, the process proceeds to S306. If it is determined that the instruction to save the virtual viewpoint image has not been issued, the process shown in FIG. 3 ends.

[0056] In S306, the saving control unit 209 determines whether to restrict the saving operation of the virtual viewpoint image. In this example, the saving control unit 209 determines whether the volumetric video has stopped. The saving control unit 209 can inquire of the time designation unit 203 whether the volumetric video has stopped. If it is determined that the volumetric video has stopped, the process proceeds to S307. If not, the process proceeds to S308.

[0057] In S307, the virtual viewpoint image is not saved by the image saving unit 207. In this case, the image saving unit 207 can notify the user who has instructed to save the virtual viewpoint image that saving is not possible. For example, the image saving unit 207 can display a UI indicating that saving is not possible on an output device that displays the virtual viewpoint video.

[0058] In S308, the image saving unit 207 saves the virtual viewpoint image generated in S304 in the image storage unit 208.

[0059] According to the above embodiment, it is not possible to save a virtual viewpoint image when the volumetric video is stopped. For example, a user may attempt to save a virtual viewpoint image of a three-dimensional model corresponding to a specific time code from the position of virtual camera 404a, as shown in FIG. 4B, while the volumetric video is stopped. Furthermore, the user may attempt to save a virtual viewpoint image of a three-dimensional model corresponding to the same time code from the position of virtual camera 404b, as shown in FIG. 4C, by moving the virtual viewpoint. However, in the above embodiment, it is not possible to save a virtual viewpoint image when the three-dimensional model of the subject is stationary. This makes it difficult to save multiple virtual viewpoint images of a three-dimensional model corresponding to the same time code. This can make it difficult to reconstruct the three-dimensional shape of the subject based on the virtual viewpoint image.

[0060] (Variation 1) As described above, the save control unit 209 restricts the saving operation of the virtual viewpoint image by the image saving unit 207 in accordance with a predetermined condition. In the above example, the save control unit 209 prohibits the image saving unit 207 from saving the virtual viewpoint image when the volumetric video is stopped. On the other hand, the predetermined condition is not limited to the volumetric video being stopped. Below, a case will be described in which the predetermined condition includes satisfying a first condition related to the virtual viewpoint. In the following example, the save control unit 209 can restrict the saving operation of the virtual viewpoint image by the image saving unit 207 in accordance with the specified virtual viewpoint.

[0061] For example, the saving control unit 209 can restrict the saving operation of the virtual viewpoint image by the image saving unit 207 based on whether the virtual viewpoint satisfies a first condition. More specifically, the saving control unit 209 can determine whether to restrict the saving operation of the virtual viewpoint image by the image saving unit 207 based on whether the parameters of the virtual viewpoint satisfy the first condition. For example, the saving control unit 209 can determine whether to restrict the saving operation of the virtual viewpoint image by the image saving unit 207 based on the position of the virtual viewpoint specified by the viewpoint specifying unit 204. In one embodiment, the first condition is that the position of the virtual viewpoint is not within a specific range. For example, the saving control unit 209 can determine to restrict the saving operation of the virtual viewpoint image based on the determination that the position of the virtual viewpoint is not within a specific range.

[0062] This processing will be described with reference to Fig. 5(A). The photography-permitted area 701 indicates the range in which photography by the virtual camera (i.e., saving of virtual viewpoint images) is permitted. Because the virtual camera 404a is within the photography-permitted area 701, the saving control unit 209 allows saving of the virtual viewpoint image obtained by the virtual camera 404a. On the other hand, because the virtual camera 404b is not within the photography-permitted area 701, the saving control unit 209 does not allow saving of the virtual viewpoint image obtained by the virtual camera 404a.

[0063] Furthermore, the saving control unit 209 can determine whether or not to restrict the saving operation of the virtual viewpoint image by the image saving unit 207, depending on the orientation of the virtual viewpoint or a combination of the position and orientation specified by the viewpoint specifying unit 204. For example, the saving control unit 209 can determine to restrict the saving operation of the virtual viewpoint image when it is determined that the orientation of the virtual viewpoint is not within a specific range.

[0064] This processing will be described with reference to Figures 5(B) and (C). The permitted image capture direction range 703 indicates the range of virtual camera orientations in which image capture by the virtual camera is permitted. As shown in Figure 5(B), the orientation 702 of the virtual camera 404a is within the permitted image capture direction range 703, so the saving control unit 209 allows the saving of the virtual viewpoint image obtained by the virtual camera 404a. On the other hand, as shown in Figure 5(C), the orientation 704 of the virtual camera 404b is not within the permitted image capture direction range 703, so the saving control unit 209 does not allow the saving of the virtual viewpoint image obtained by the virtual camera 404a.

[0065] The above-described conditions regarding the position of the virtual viewpoint and the conditions regarding the direction of the virtual viewpoint may be combined. For example, when the direction 702 of the virtual camera 404a is within the permitted photographing direction range 703 and the virtual camera 404a is within the permitted photographing area 701, the saving control unit 209 may permit saving of the virtual viewpoint image obtained by the virtual camera 404a. In other words, when the virtual camera 404a is facing the direction 704, the saving control unit 209 can restrict the saving operation of the virtual viewpoint image obtained by the virtual camera 404a even if the virtual camera 404a is within the permitted photographing area 701. In this way, conditions regarding the camera parameters of the virtual camera, such as the permitted photographing area 701 and the permitted photographing direction range 703, can be combined as appropriate.

[0066] Furthermore, the storage control unit 209 can determine whether to limit the operation of storing the virtual viewpoint image by the image storage unit 207, depending on the resolution of the three-dimensional model of the subject in the virtual viewpoint image. Such resolution can be determined based on parameters of the virtual viewpoint. For example, the resolution can be expressed by the number of pixels in the virtual viewpoint image of a portion of the three-dimensional model corresponding to a single three-dimensional shape component, such as a voxel or texture, of the subject. The three-dimensional shape component corresponding to the texture can be, for example, a single pixel constituting the texture, or an entire texture represented by vertex coordinates. In such an example, the first condition may be that the resolution is smaller than one pixel. Such resolution can be determined based on the position, angle of view, number of pixels, etc. of the virtual camera. Therefore, such a condition regarding the resolution is also included in the first condition regarding the virtual viewpoint.

[0067] In this modified example, in S306, the save control unit 209 determines whether or not to restrict the saving operation of the virtual viewpoint image according to the virtual viewpoint as described above. If it is determined that the saving operation of the virtual viewpoint image by the image saving unit 207 should be restricted in accordance with the above conditions, the process proceeds to S307. If not, the process proceeds to S308.

[0068] If the parameters of the virtual viewpoint satisfy the first condition, the storage control unit 209 may further control the time designation unit 203 to prohibit stopping of the volumetric video.

[0069] According to this modification, the virtual viewpoints corresponding to the virtual viewpoint images to be saved are limited. Therefore, it is possible to prevent the saving of virtual viewpoint images of a three-dimensional model from various directions in order to restore the three-dimensional shape of the subject. Furthermore, by bringing the virtual viewpoint closer to the three-dimensional model, it is possible to prevent the saving of virtual viewpoint images of a three-dimensional model with high resolution. In this way, according to this modification, it is possible to make it difficult to restore the three-dimensional shape of the subject.

[0070] (Variation 2) The following describes a case where the predetermined condition includes satisfying a second condition related to the number of times virtual viewpoint images are saved. For example, the saving control unit 209 can limit the saving operation of the virtual viewpoint images by the image saving unit 207 depending on the number of times the virtual viewpoint images are saved. The second condition may include a condition related to the number of times virtual viewpoint images of a three-dimensional model are saved at a specific time. For example, the second condition may be that the number of times virtual viewpoint images are saved at a specific time code is equal to or greater than a predetermined number. In such an example, a limit can be set on the number of virtual viewpoint images that can be saved at the same time code of the volumetric video.

[0071] In this modification, the image saving unit 207 stores image recording information corresponding to the virtual viewpoint image in the image storage unit 208 in association with the virtual viewpoint image. This image recording information may include time information related to the virtual viewpoint image. This time information may indicate a time code corresponding to the virtual viewpoint image. For example, the image saving unit 207 can save the time code of the volumetric video being played, acquired from the time designation unit 203, as the image recording information. Meanwhile, the time information may indicate the real time when the virtual viewpoint image is saved. The recorded time may be based on Universal Time. Furthermore, the image recording information may include information on the virtual viewpoint corresponding to the virtual viewpoint image. For example, the image recording information may include camera parameters, including external parameters and internal parameters, set by the viewpoint designation unit 204 when the image saving unit 207 saved the virtual viewpoint image.

[0072] Furthermore, the saving control unit 209 determines whether to limit the saving operation of the virtual viewpoint image according to the number of times the virtual viewpoint image has been saved. For example, the saving control unit 209 can acquire the current time code specified by the time specification unit 203. Then, the saving control unit 209 determines the number of times the past virtual viewpoint image has been saved for the current time code by referring to the image storage unit 208. Note that the number of times the current time code has been saved may be the number of times the current time code has been saved for the same time code as the current time code. On the other hand, the number of times the current time code has been saved may be the number of times the current time code has been saved for a time code close to the current time code. For example, the saving control unit 209 may determine the number of times the virtual viewpoint image has been saved within a predetermined period close to the current time code (for example, within one second before or after the current time code). Furthermore, the volumetric video may be divided into multiple scenes in advance. For example, the volumetric video may be divided into units of one second or the like according to the time code. In this case, the saving control unit 209 can determine the number of times the same scene as the current time code has been saved.

[0073] In this example, the second condition is that the number of times of saving is equal to or greater than a predetermined number. That is, in response to determining that the user has saved a virtual viewpoint image for the current time code a predetermined number of times or more in the past, the saving control unit 209 restricts the saving operation of the virtual viewpoint image. On the other hand, in response to determining that the user has not saved a virtual viewpoint image for the current time code a predetermined number of times or more in the past, the saving control unit 209 permits saving of the virtual viewpoint image. Note that the predetermined number of times is not particularly limited and may be one time, two times, three times, four times, or more.

[0074] In another embodiment, the storage control unit 209 may determine the number of times the virtual viewpoint image is stored for the entire volumetric video. In this example, the second condition is that the number of times the virtual viewpoint image is stored for the entire volumetric video is equal to or greater than a predetermined number. In such an example, the number of times the virtual viewpoint image is stored for the entire volumetric video can be limited.

[0075] In another embodiment, the storage control unit 209 may determine the number of times a virtual viewpoint image is saved within a predetermined period of real time (i.e., the saving frequency). In this example, the second condition is that this saving number is equal to or greater than a threshold value. For example, the storage control unit 209 may limit the saving operation of a virtual viewpoint image in response to determining that a virtual viewpoint image has already been saved 10 times within 15 minutes in real time. Note that the length of the predetermined period is not particularly limited and may be longer than 15 minutes, such as one hour, one day, or one week, or shorter than 15 minutes, such as five minutes or one minute. The threshold value is also not particularly limited and may be longer or shorter than 10 times, for example.

[0076] The save control unit 209 may also use a combination of the above conditions to determine whether or not to restrict the save operation of the virtual viewpoint image.

[0077] In this modified example, in S306, the saving control unit 209 determines whether or not to limit the saving operation of the virtual viewpoint image, depending on the number of times the virtual viewpoint image has been saved, as described above. If it is determined that the saving operation of the virtual viewpoint image by the image saving unit 207 should be limited in accordance with the above conditions, the process proceeds to S307. If not, the process proceeds to S308.

[0078] According to this modification, the number of times that virtual viewpoint images can be saved is limited. This makes it difficult for a user to save more than a predetermined number of virtual viewpoint images. In other words, it is possible to prevent a user from saving a large number of virtual viewpoint images of a three-dimensional model from various directions in order to restore the three-dimensional shape of a subject. In this way, according to this modification, it is possible to make it difficult to restore the three-dimensional shape of a subject.

[0079] (Variation 3) Up to this point, examples have been described in which the operation of saving a virtual viewpoint image is restricted when volumetric video is stopped, the operation of saving a virtual viewpoint image is restricted depending on the virtual viewpoint, and the operation of saving a virtual viewpoint image is restricted depending on the number of times the image is saved. These conditions can be used in combination. There is no particular limitation on how the conditions are combined. For example, it may be determined that a predetermined condition is satisfied when all of a plurality of conditions are satisfied. It may also be determined that a predetermined condition is satisfied when at least one of a plurality of conditions is satisfied.

[0080] For example, the save control unit 209 may restrict the save operation of the virtual viewpoint image in response to determining that either or both of the conditions of the volumetric video being stopped and the condition related to the virtual viewpoint parameters are satisfied. Also, the save control unit 209 may restrict the save operation of the virtual viewpoint image in response to determining that either or both of the conditions of the volumetric video being stopped and the condition related to the number of times the virtual viewpoint image has been saved are satisfied. Furthermore, the save control unit 209 may restrict the save operation of the virtual viewpoint image in response to determining that either or both of the conditions of the virtual viewpoint parameters and the condition related to the number of times the virtual viewpoint image has been saved are satisfied.

[0081] In the above example, when the saving control unit 209 restricts the saving operation of the virtual viewpoint image, the image saving unit 207 does not save the virtual viewpoint image in the image storage unit 208. However, the manner of restriction by the saving control unit 209 is not limited to the above. For example, the saving control unit 209 may reduce the quality of the virtual viewpoint image to be saved in response to determining that a predetermined condition is satisfied. For example, the saving control unit 209 can control the image saving unit 207 to reduce the virtual viewpoint image to be saved in response to determining that a predetermined condition is satisfied. In this case, the image saving unit 207 can save the reduced virtual viewpoint image in the image storage unit 208. In this way, the image saving unit 207 can lower the resolution of the three-dimensional model in the virtual viewpoint image to be saved in the image storage unit 208.

[0082] As described above, the resolution of the three-dimensional model in the virtual viewpoint image can be expressed by the number of pixels on the virtual viewpoint image that correspond to one three-dimensional shape component of the three-dimensional model corresponding to the subject. In this case, the saving control unit 209 can control the resolution of the three-dimensional model in the virtual viewpoint image that the image saving unit 207 saves in the image storage unit 208 to be smaller than one pixel.

[0083] As another example, when the saving control unit 209 restricts the saving operation of the virtual viewpoint image, the playback state of the volumetric video or the parameters of the virtual viewpoint may be changed so that saving of the virtual viewpoint image is permitted. For example, the saving control unit 209 can control the time designation unit 203 to release the stopped state of the volumetric video and play the volumetric video in response to a user instruction to save the virtual viewpoint image. Even in this configuration, while saving of the virtual viewpoint image is not permitted when the volumetric video is stopped, playback of the volumetric video is resumed in response to a user instruction, making it possible to save the virtual viewpoint image.

[0084] [Embodiment 2] An image processing system 1000 according to an embodiment will be described below with reference to FIG. 6 . FIG. 6 shows an example of the functional configuration of the image processing system 1000 according to this embodiment. The image processing system 1000 includes a server device 1001 and a client device 1002. Similar to embodiment 1, this embodiment restricts the saving operation of virtual viewpoint images according to predetermined conditions. However, this embodiment differs from embodiment 1 in that the server device 1001 performs rendering processing of volumetric video and the virtual viewpoint video is sent to the client device 1002 via communication. Specifically, the server device 1001 generates the virtual viewpoint video. Furthermore, the client device 1002 receives the virtual viewpoint video from the server device 1001 and plays back the virtual viewpoint video. In this embodiment, the client device 1002 does not receive three-dimensional shape data of the subject. The hardware configurations of the server device 1001 and the client device 1002 may be similar to the hardware configuration of the image processing device 100.

[0085] The server device 1001 has a model acquisition unit 201, a space construction unit 202, a time designation unit 203, a viewpoint designation unit 204, a video generation unit 205, a save control unit 209, and a communication unit 1003. The client device 1002 has a video output unit 206, an image save unit 207, an image storage unit 208, a communication unit 1003, and an operation unit 1004. The same functional units as those in the first embodiment are assigned the same reference numerals. Differences from the first embodiment will be described below.

[0086] The communication unit 1003 communicates with the server device 1001 or the client device 1002. The communication unit 1003 allows two-way communication between the server device 1001 and the client device 1002. The configuration of the communication unit 1003 can be selected so as to ensure the transmission bandwidth necessary for communication between the server device 1001 and the client device 1002. The communication unit 1003 may communicate via a network. For example, the communication unit 1003 may communicate via a local network such as Ethernet. In this case, the communication unit 1003 can communicate via, for example, various switching hubs or router devices. Furthermore, at least a portion of the network may be the Internet or a dedicated line. In this case, the server device 1001 and the client device 1002 can be installed in physically separate locations.

[0087] The operation unit 1004 accepts user operations. The operation unit 1004 can also send information indicating the user operations to each unit of the server device 1001, such as the time designation unit 203 and the viewpoint designation unit 204, via communication by the communication unit 1003. For example, the operation unit 1004 can accept a user operation regarding the virtual viewpoint and send information indicating the user operation to the viewpoint designation unit 204. The operation unit 1004 can also accept user operations regarding the playback of volumetric video. For example, the operation unit 1004 can accept user operations for designating a time code, playing, stopping, and performing special playback. The operation unit 1004 can also transmit information indicating these user operations to the time designation unit 203. In this way, the user can operate the displayed virtual viewpoint video. The operation unit 1004 can also accept a user operation for saving a virtual viewpoint image. The image saving unit 207 can save the virtual viewpoint image in the image storage unit 208 according to such user operations.

[0088] In the server device 1001 according to this embodiment, the video generation unit 205 acquires multiple virtual viewpoint images, each of which is an image of a three-dimensional model from a specifiable virtual viewpoint corresponding to a certain time. The communication unit 1003 outputs the video represented by the multiple virtual viewpoint images generated by the video generation unit 205 to the client device 1002 for display. The video data transmitted by the communication unit 1003 may be compressed using any encoding method, such as MPEG or H.264. The storage control unit 209 restricts the image storage unit 207 from storing the displayed virtual viewpoint image in accordance with a user instruction when a predetermined condition is met.

[0089] Furthermore, in the client device 1002 according to this embodiment, the communication unit 1003 acquires a plurality of virtual viewpoint images, each of which is an image of a three-dimensional model from a specifiable virtual viewpoint corresponding to a certain time. Furthermore, the video output unit 206 outputs a video represented by the plurality of virtual viewpoint images to an output device for display. Furthermore, the image saving unit 207 can store, in the image storage unit 208, the virtual viewpoint images being output by the video output unit 206 or the virtual viewpoint images being displayed on the output device, among the virtual viewpoint images constituting the virtual viewpoint video sent from the server device 1001. At this time, the image saving unit 207, under the control of the saving control unit 209, can limit the saving operation of the displayed virtual viewpoint image in accordance with a user instruction to the case where a predetermined condition is satisfied.

[0090] 7A and 7B are flowcharts showing the processing flow of an image processing method performed by the image processing system 1000 according to one embodiment. By repeating the flowcharts shown in FIGS. 7A and 7B, a virtual space image is generated and displayed. As in the first embodiment, the following describes a configuration in which saving of virtual viewpoint images is restricted when volumetric video is stopped. FIG. 7A shows processing performed by the server device 1001. FIG. 7B shows processing performed by the client device 1002.

[0091] First, the flowchart of the processing performed by the server device 1001 shown in FIG. 7A will be described. In S1101, the communication unit 1003 of the server device 1001 acquires information indicating a user operation from the operation unit 1004. For example, the communication unit 1003 can receive information indicating a user operation to change a virtual viewpoint or a user operation related to the playback of volumetric video. Based on the received information, the viewpoint designation unit 204 can set a virtual viewpoint. Furthermore, based on the received information, the time designation unit 203 can perform time code control related to the playback of volumetric video.

[0092] Steps S301, S302, S303, and S304 are performed in the same manner as in the first embodiment. The determination in step S306 is also performed in the same manner as in the first embodiment. That is, the save control unit 209 can determine whether to restrict the operation of saving the virtual viewpoint image in accordance with a predetermined condition. If the operation of saving the virtual viewpoint image is to be restricted, the process proceeds to step S1102. If not, the process proceeds to step S1103. In this example, if it is determined that the volumetric video has stopped, the process proceeds to step S1102. If not, the process proceeds to step S1103.

[0093] In S1102, the saving control unit 209 determines to restrict the saving operation of the virtual viewpoint image. In this example, the saving control unit 209 determines to prohibit saving of the virtual viewpoint image. In S1103, the saving control unit 209 determines not to restrict the saving operation of the virtual viewpoint image. In this example, the saving control unit 209 determines to allow saving of the virtual viewpoint image.

[0094] In S1104, the video generation unit 205 transmits the virtual viewpoint image generated in S304 to the client device 1002 via the communication unit 1003. In addition, the saving control unit 209 transmits information (save permission information) indicating whether to restrict the saving operation of the virtual viewpoint image to the client device 1002 via the communication unit 1003. For example, the saving control unit 209 may transmit the determination results in S1102 to S1103 to the client device 1002. The saving control unit 209 may also transmit information specifying a restriction method for saving the virtual viewpoint image to the client device 1002.

[0095] Next, a flowchart of the process performed by the client device 1002 shown in FIG. 7B will be described.

[0096] In S1105, the operation unit 1004 accepts a user operation. For example, the operation unit 1004 can accept a user operation related to a virtual viewpoint and playback of volumetric video. The operation unit 1004 then transmits operation information indicating the accepted user operation to the server device 1001 via the communication unit 1003. The transmitted information is received in S1101 as described above.

[0097] In S1106, the communication unit 1003 of the client device 1002 acquires the information transmitted from the server device 1001 in S1104 as described above. That is, the communication unit 1003 can acquire the virtual viewpoint image generated by the server device 1001 and the storage permission information. The communication unit 1003 sends the acquired virtual viewpoint image to the video output unit 206. In addition, the communication unit 1003 sends the storage permission information to the image storage unit 207.

[0098] The determination in S305 is performed in the same manner as in the first embodiment. If it is determined that an instruction to save a virtual viewpoint image has been issued, the process proceeds to S1107. If it is determined that an instruction to save a virtual viewpoint image has not been issued, the process shown in FIG. 7B ends.

[0099] In S1107, the image saving unit 207 determines whether or not saving permission information indicating that the saving operation of the virtual viewpoint image is to be restricted has been sent from the server device 1001. If saving permission information indicating that the saving operation of the virtual viewpoint image is to be restricted has been sent from the server device 1001, the process proceeds to S307. If such information has not been sent, the process proceeds to S308. The process of S307 is performed in the same manner as in the first embodiment. In this example, the image saving unit 207 does not save the virtual viewpoint image. Furthermore, in S308, the image saving unit 207 saves the virtual viewpoint image acquired in S1104 in the image storage unit 208.

[0100] In the above embodiment, the communication unit 1003 of the client device 1002 acquires the storage permission information along with the virtual viewpoint image in S1106. On the other hand, the image storage unit 207 may acquire the storage permission information by inquiring of the server device 1001 in S1107. In this way, the server device 1001 may transmit the storage permission information in response to an inquiry from the client device 1002, instead of sending the storage permission information along with the virtual viewpoint image in S1104.

[0101] According to the above embodiment, the server device 1001 generates a virtual viewpoint image. Furthermore, the server device 1001 determines whether to restrict the saving operation of the virtual viewpoint image. The user cannot save the virtual viewpoint image in the client device 1002 when the three-dimensional model of the subject is stationary. This makes it difficult to restore the three-dimensional shape of the subject based on the virtual viewpoint image.

[0102] As in the first embodiment, the predetermined condition is not limited to the volumetric video being stopped. As in the first modification, the save control unit 209 may restrict the saving operation of the virtual viewpoint image by the image saving unit 207 in accordance with the specified virtual viewpoint. Such processing can be realized by determining in S306 whether or not the save control unit 209 restricts the saving operation of the virtual viewpoint image, as in the first modification.

[0103] Furthermore, similar to the above-described second modification, the saving control unit 209 may limit the saving operation of the virtual viewpoint image by the image saving unit 207 depending on the number of times the virtual viewpoint image has been saved. In such an example, in S308, the image saving unit 207 can transmit image recording information to the server device 1001 via the communication unit 1003. The image saving unit 207 can receive the image recording information via the communication unit 1003. The image saving unit 207 may store the received image recording information in an image recording information saving unit (not shown). Furthermore, in S306, the saving control unit 209 can determine whether or not to limit the saving operation of the virtual viewpoint image, similar to the second modification. At this time, the saving control unit 209 can determine whether or not a predetermined condition regarding the number of times the virtual viewpoint image has been saved is satisfied, by referring to the image recording information received from the image saving unit 207.

[0104] Furthermore, similarly to the first embodiment, the conditions for restricting the storage of virtual viewpoint images can be used in combination.

[0105] (Variation 4) The configuration of the image processing system 1000 is not limited to that shown in Fig. 6. For example, the connection between the server device 1001 and the client device 1002 is not limited to a one-to-one connection. For example, a plurality of client devices 1002 may be connected to the server device 1001.

[0106] The server device 1001 may also operate on multiple physical devices. In this case, the server device 1001 may be logically recognized as a single device by the client device 1002. The server device 1001 may also be realized using a virtual service such as cloud technology. In this case, the server device 1001 may have multiple logical function units that perform the above-described processing using software technology.

[0107] In this case, the plurality of client devices 1002 may view volumetric video in a common virtual space. Furthermore, the playback state of the volumetric video (e.g., playback, stop, or time code of the display target) may be common among the plurality of client devices 1002. For example, when an operation to stop the volumetric video is performed by one client device 1002, the volumetric video may also be stopped by the other client devices 1002. Similarly, the plurality of client devices 1002 may view virtual viewpoint video from a commonly set virtual viewpoint. For example, when an operation to specify a virtual viewpoint is performed by one client device 1002, the virtual viewpoint image displayed on the other client devices 1002 may change in accordance with the specified virtual viewpoint.

[0108] Furthermore, when a first user performs an operation to save a virtual viewpoint image, another second user may have performed an operation to stop the volumetric video. In such a case, the saving control unit 209 can determine in S306 that the volumetric video has stopped. Then, the saving control unit 209 can determine not to permit the first user to save the virtual viewpoint image.

[0109] In addition, the save control unit 209 may control the time designation unit 203 so as not to accept an instruction to stop the volumetric video when the parameters of the virtual viewpoint designated by at least one of the multiple client devices 1002 satisfy a first condition.

[0110] The following describes a process for determining whether to restrict the saving operation of a virtual viewpoint image when multiple client devices 1002 connect to the server device 1001, similar to Modification Example 2. In this example, virtual viewpoint images are saved in accordance with user instructions from each of multiple users. Meanwhile, the saving control unit 209 can receive image recording information via the communication unit 1003 as described above and store the received image recording information in an image recording information saving unit (not shown). Here, the saving control unit 209 can store the image recording information for all client devices 1002. Furthermore, the saving control unit 209 can store the image recording information in association with the user of the client device 1002. For example, the saving control unit 209 can store the image recording information together with user information such as a user ID.

[0111] As described above, in S306, the saving control unit 209 can determine whether to limit the saving operation of the virtual viewpoint image, depending on the number of times the virtual viewpoint image has been saved. Here, the number of times the virtual viewpoint image has been saved may be the total number of times the virtual viewpoint image has been saved in accordance with user instructions from each of multiple users. That is, in S306, the saving control unit 209 can determine whether to limit the saving operation of the virtual viewpoint image, depending on the total number of times the virtual viewpoint image has been saved for all users. In other words, the saving control unit 209 can check across all users whether the number of times the virtual viewpoint image has been saved for a specific time code is equal to or greater than a predetermined number.

[0112] For example, assume that the number of times a virtual viewpoint image can be saved for the same time code is 10, and that user A has already saved five virtual viewpoint images for a specific time code, and user B has already saved five virtual viewpoint images. In this case, even if user C attempts to save another virtual viewpoint image for the same time code, user C cannot save the virtual viewpoint image because users A and B have already saved a predetermined number of virtual viewpoint images. In another example, if a virtual viewpoint image can only be saved up to 10 times within 15 minutes in real time, and user A and user B have saved a total of 10 virtual viewpoint images, user C will have to wait 15 minutes before saving a virtual viewpoint image. Note that the time is not limited to 15 minutes and may be longer or shorter. Furthermore, the number of times a virtual viewpoint image can be saved is not limited to 10 and may be shorter or longer.

[0113] Furthermore, the predetermined conditions may include conditions for individual users. Furthermore, the predetermined conditions may include conditions for individual users in addition to conditions for all users. For example, the number of times that can be saved for all users may be set to 10 or less, and the number of times that can be saved for each individual user may be set to 5 or less. In this way, the upper limit of the number of times that can be saved for each individual user may be different from the upper limit of the number of times that can be saved for all users.

[0114] [Embodiment 3] An image processing device according to one embodiment will be described below with reference to the drawings. In the following embodiment, the image processing device 1600 generates and outputs a virtual viewpoint video represented by a plurality of virtual viewpoint images, similar to the image processing device 100 according to embodiment 1. The image processing device 1600 according to this embodiment restricts user operations on video when predetermined conditions related to the method of transmitting the video to the output device are satisfied. For example, the image processing device 1600 can restrict user operations on video when the video transmitted to the output device is not encrypted. First, a configuration for restricting an operation to stop volumetric video when the video signal to be output is not encrypted will be described below.

[0115] 8 is a schematic diagram showing an example of the functional configuration of an image processing device 1600 according to this embodiment. The hardware configuration of the image processing device 1600 may be similar to the hardware configuration of the image processing device 100. The image processing device 1600 includes a model acquisition unit 201, a space construction unit 202, a time designation unit 203, a viewpoint designation unit 204, a video generation unit 205, a video output unit 206, an encryption unit 1601, and an operation restriction unit 1602.

[0116] The same reference numerals are assigned to the same functional units as in the first embodiment. For example, the video generation unit 205 can acquire multiple virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time. Furthermore, the video output unit 206 can output video represented by the multiple virtual viewpoint images for display by an output device. The following describes the differences from the first embodiment.

[0117] The encryption unit 1601 can encrypt the virtual viewpoint video generated by the video generation unit 205. The encryption unit 1601 can encrypt a video signal representing the virtual viewpoint video in accordance with a setting by a user. The encryption method is not particularly limited, but for example, the HDCP method can be used. As the encryption method, a method that makes it impossible for a device that receives a video signal to record the video signal may be used. Furthermore, as the encryption method, a method that makes it impossible for a device that receives a video signal to reproduce the original video signal based on the video signal recorded may be used.

[0118] The operation limiting unit 1602 limits user operations on the video when a predetermined condition regarding the communication method of the video to the output device is satisfied. In this example, the operation limiting unit 1602 limits user operations on the video depending on whether the virtual viewpoint video output by the image processing device is encrypted. For example, the operation limiting unit 1602 can limit user operations on the video depending on whether it is determined that the virtual viewpoint video output by the image processing device is not encrypted. Furthermore, the operation limiting unit 1602 can determine not to limit user operations on the video depending on whether it is determined that the virtual viewpoint video output by the image processing device is encrypted. In this example, the operation limiting unit 1602 rejects an operation to stop the volumetric video if the virtual viewpoint video output by the image processing device is not encrypted. With this configuration, the volumetric video cannot be stopped. However, the operation limiting unit 1602 may also limit user operations on the video depending on whether a communication method that makes it difficult to use the output video is used.

[0119] 9 is a flowchart showing the processing flow of an image processing method performed by the image processing device 1600 according to one embodiment. A virtual space image is generated according to the flowchart shown in FIG. 9. Furthermore, by repeating the flowchart shown in FIG. 9, a virtual space video is generated and output.

[0120] In S1701, the time designation unit 203 determines whether or not a volumetric video stop instruction has been received from the user. If a volumetric video stop instruction has not been received, the process proceeds to S301. If a volumetric video stop instruction has been received, the process proceeds to S1702.

[0121] In S1702, the operation restriction unit 1602 determines whether the video signal has been encrypted by the encryption unit 1601. If the video signal has been encrypted, the process proceeds to S1703. If the video signal has not been encrypted, the process proceeds to S1704.

[0122] In S1703, the operation restriction unit 1602 determines to stop the volumetric video in accordance with a user instruction. That is, in subsequent S301, the time designation unit 203 sets the time code to a fixed value. Note that in subsequent S1701, the time designation unit 203 can resume playback of the volumetric video in response to determining that a volumetric video playback instruction (or pause release instruction) has been received from the user. In this case, in subsequent S301, the time designation unit 203 counts up the time code.

[0123] In S1704, the operation limiting unit 1602 determines not to stop the volumetric video in accordance with the user instruction. That is, despite receiving the instruction to stop the volumetric video, the time designation unit 203 counts up the time code in the subsequent S301. In this case, the operation limiting unit 1602 may notify the user that user operations are restricted. For example, the operation limiting unit 1602 can notify the user who issued the instruction to stop the volumetric video that the volumetric video cannot be stopped. Specifically, the operation limiting unit 1602 can control the video output unit 206 to display a UI indicating that stopping is not possible on the output device that displays the virtual viewpoint video.

[0124] Steps S301 to S304 are performed in the same manner as in the first embodiment.

[0125] Note that the method for restricting user operations on video when a predetermined condition regarding the method of transmitting video to the output device is satisfied is not limited to the above method. For example, the encryption / decryption of the virtual viewpoint video may be dynamically switched. Specifically, the operation restriction unit 1602 may control the encryption unit 1601 to encrypt the output virtual viewpoint video in response to an instruction to stop the volumetric video. With this configuration, encrypted virtual viewpoint video is output when the volumetric video is stopped. The operation restriction unit 1602 may also control the encryption unit 1601 to decrypt the virtual viewpoint video when resuming playback of the volumetric video. With this configuration, the operation to stop the volumetric video can also be restricted when the virtual viewpoint video is not encrypted.

[0126] Furthermore, when an instruction to stop the volumetric video is received while the virtual viewpoint image is not encrypted, the operation restriction unit 1602 can control the image generation unit 205 or the image output unit 206 to stop outputting the virtual viewpoint image. When the operation restriction unit 1602 rejects a user operation in this manner, the operation restriction unit 1602 may control the output device to output a pre-prepared image. For example, in this case, the image output unit 206 can output an image different from the volumetric video, such as a black screen. Furthermore, the image output unit 206 can notify the user who issued the instruction to stop the volumetric video that the volumetric video cannot be stopped. While the output of the volumetric video is stopped, the video output unit 206 can display a UI on the output device indicating that the image cannot be output.

[0127] According to the above embodiment, when the volumetric video is stopped, unencrypted virtual viewpoint video is not output. A user may attempt to output virtual viewpoint video including virtual viewpoint images from various viewpoints of a three-dimensional model corresponding to the same time code by stopping the volumetric video and moving the virtual viewpoint. However, because the virtual viewpoint video output from the image processing device 1600 is encrypted, it is difficult to restore the three-dimensional shape of the subject based on the output virtual viewpoint image.

[0128] (Variation 5) The method by which the operation restriction unit 1602 restricts user operations on video is not limited to the above method. When a predetermined condition related to the method of communicating video to the output device is satisfied, the operation restriction unit 1602 can restrict user operations that comply with the predetermined condition. As the predetermined condition related to user operations, for example, the same condition as in Variation 1 can be used. Specifically, the operation restriction unit 1602 can reject a user operation that specifies a virtual viewpoint that satisfies a first condition when it is determined that the video to be transmitted to the output device is not encrypted. With this configuration, virtual viewpoint video from a virtual viewpoint that satisfies the first condition will not be output.

[0129] For example, the operation restriction unit 1602 can acquire information indicating whether the virtual viewpoint image is encrypted from the encryption unit 1601. Furthermore, the operation restriction unit 1602 can acquire a user instruction for operating the virtual viewpoint. The operation restriction unit 1602 can determine whether to permit operation of the virtual viewpoint based on this information. For example, if the virtual viewpoint image is not encrypted and parameters of the virtual viewpoint according to the user instruction satisfy predetermined conditions, the operation restriction unit 1602 can control the viewpoint specification unit 204 so as not to accept operation of the virtual viewpoint according to the user instruction.

[0130] 10 is a flowchart showing the processing flow of the image processing method performed by the image processing apparatus 1600 according to an embodiment of the present invention in this modified example. Steps S301 and S302 are performed in the same manner as in the first embodiment.

[0131] In S1901, the operation restriction unit 1602 determines whether a user instruction to operate the virtual viewpoint has been input. If a user instruction has been input, the process proceeds to S1702, and if no operation has been performed, the process proceeds to S304.

[0132] In S1702, the operation restriction unit 1602 determines whether the video signal has been encrypted by the encryption unit 1601. If the video signal has been encrypted, the process proceeds to S1903. If the video signal has not been encrypted, the process proceeds to S1902.

[0133] In S1902, the operation restriction unit 1602 determines whether the parameters of the virtual viewpoint satisfy a predetermined condition, as in Modification Example 1. If the parameters of the virtual viewpoint satisfy the predetermined condition, the process proceeds to S1904. If not, the process proceeds to S1903.

[0134] In S1903, the operation restriction unit 1602 permits the operation of the virtual viewpoint in accordance with the user's instruction. Then, the viewpoint designation unit 204 sets the virtual viewpoint in accordance with the user's instruction. In S1904, the operation restriction unit 1602 rejects the operation of the virtual viewpoint in accordance with the user's instruction. In this case, the viewpoint designation unit 204 does not change the virtual viewpoint.

[0135] S304 is performed in the same manner as in embodiment 1. That is, the video generation unit 205 generates a virtual viewpoint image according to the virtual viewpoint set in S1902 or a virtual viewpoint that has been set previously.

[0136] According to the above embodiment, when the virtual viewpoint video is not encrypted, the output of the virtual viewpoint video from a virtual viewpoint that complies with predetermined conditions is restricted. For example, when the virtual viewpoint video is not encrypted, the user cannot set the virtual viewpoint outside the permitted range (e.g., the permitted shooting area 701). In this way, the virtual viewpoint corresponding to the output virtual viewpoint image is limited. On the other hand, when the virtual viewpoint is outside the permitted range, the virtual viewpoint video output from the image processing device 1600 is encrypted, making it difficult to restore the three-dimensional shape of the subject based on the output virtual viewpoint image.

[0137] As in the above embodiment, when a virtual viewpoint image is not encrypted and a predetermined condition is satisfied, the operation restriction unit 1602 may control the video generation unit 205 or the video output unit 206 to stop outputting the virtual viewpoint image. For example, when a virtual viewpoint image is not encrypted and a parameter of the virtual viewpoint satisfies a predetermined condition, the operation restriction unit 1602 can stop outputting the virtual viewpoint image.

[0138] So far, the conditions under which the output of the virtual viewpoint video is restricted have been described. As in Modification 3, these conditions can be used in combination. Furthermore, the manner of restriction by the operation restriction unit 1602 is not limited to the above. For example, as in Modification 3, when the output of the virtual viewpoint image is restricted, the operation restriction unit 1602 may reduce the image quality of the virtual viewpoint image output from the video output unit 206.

[0139] [Fourth Embodiment] An image processing system 2000 according to one embodiment will be described below with reference to Fig. 11 . Fig. 11 shows an example of the functional configuration of the image processing system 2000. As in the second embodiment, the image processing system 2000 includes a server device 2001 and a client device 2002. Also, as in the second embodiment, the server device 2001 performs rendering processing of volumetric video, and a virtual viewpoint video is sent to the client device 2002 via communication. Note that the hardware configurations of the server device 2001 and the client device 2002 may be similar to the hardware configuration of the image processing device 100.

[0140] In this embodiment, similarly to the third embodiment, when predetermined conditions regarding the video communication method between the client device 2002 and the output device are satisfied, user operations on the video are restricted. First, the following describes a configuration for restricting user operations to stop volumetric video when the video signal to be output is not encrypted.

[0141] The server device 2001 has a model acquisition unit 201, a space construction unit 202, a time designation unit 203, a viewpoint designation unit 204, a video generation unit 205, an operation restriction unit 1602, and a communication unit 1003. The client device 2002 has a video output unit 206, a communication unit 1003, an operation unit 1004, and an encryption unit 1601. The same functional units as those in the second and third embodiments are assigned the same reference numerals. The differences from the second and third embodiments will be described below.

[0142] The operation restriction unit 1602 acquires information indicating a method of video communication between the client device 2002 and the output device. In this embodiment, the operation restriction unit 1602 can acquire information indicating whether or not the virtual viewpoint video has been encrypted from the encryption unit 1601 via the communication unit 1003. The operation restriction unit 1602 can also control the time designation unit 203 to restrict a user operation to stop the volumetric video.

[0143] 12A and 12B are flowcharts showing the processing flow of an image processing method performed by an image processing system 2000 according to an embodiment. A virtual space image is generated and output by repeating the flowcharts shown in FIGS. 12A and 12B. FIG. 12A shows the processing performed by a server device 2001. FIG. 12B shows the processing performed by a client device 2002.

[0144] S1101 is performed in the same manner as in the second embodiment. In this embodiment, an instruction to stop the volumetric video can be received from the user in S1101. Furthermore, S1701 to S1704 are performed in the same manner as in the third embodiment. That is, if the virtual viewpoint video has been encrypted, the operation restriction unit 1602 determines in S1703 to stop the volumetric video. Furthermore, if the virtual viewpoint video has not been encrypted, the operation restriction unit 1602 determines in S1704 not to stop the volumetric video.

[0145] Steps S301 to S304 are performed in the same manner as in embodiment 2. In step S2102, the video generation unit 205 transmits the virtual viewpoint image generated in step S304 to the client device 2002 via the communication unit 1003.

[0146] Step S1105 is performed in the same manner as in embodiment 3. In step S2103, the encryption unit 1601 transmits information indicating whether the virtual viewpoint video has been encrypted (encryption status information) to the operation restriction unit 1602 via the communication unit 1003. In step S2104, the communication unit 1003 of the client device 1002 acquires the virtual viewpoint image generated by the server device 1001. Thereafter, the video output unit 206 can output the virtual viewpoint image to the output device for display.

[0147] Even in the above embodiment, when the volumetric video is stopped, unencrypted virtual viewpoint images are not output, which makes it difficult to reconstruct the three-dimensional shape of the subject based on virtual viewpoint images from various viewpoints obtained by stopping the volumetric video and moving the virtual viewpoint.

[0148] (Variation 6) As in the third embodiment, the image processing system 2000 can restrict user operations on video that comply with predetermined conditions when predetermined conditions related to a method of transmitting video to an output device are satisfied. Here, the predetermined conditions are not limited to those described above. For example, as in Variation 5, the predetermined conditions may be conditions related to a virtual viewpoint specified by a user operation.

[0149] 13 is a flowchart showing the processing flow of the image processing method in this modified example, which is performed by the server device 2001. The client device 2002 can operate according to the flowchart in FIG.

[0150] S1101, S301, and S302 are performed in the same manner as in FIG. 12A. That is, in S1101, a user instruction to operate the virtual viewpoint can be received. Furthermore, S1901, S1702, and S1902 to S1904 are performed in the same manner as in Modification 5. That is, if the virtual viewpoint video has been encrypted, in S1903 the viewpoint designation unit 204 sets the virtual viewpoint in accordance with the user instruction. Also, if the virtual viewpoint video has not been encrypted and the virtual viewpoint does not satisfy predetermined conditions, the viewpoint designation unit 204 sets the virtual viewpoint in accordance with the user instruction in S1903. On the other hand, if the virtual viewpoint video has not been encrypted and the virtual viewpoint satisfies predetermined conditions, the viewpoint designation unit 204 does not accept an operation of the virtual viewpoint in accordance with the user instruction in S1904.

[0151] Even with the above-described modification, if the virtual viewpoint video is not encrypted, the output of the virtual viewpoint video from the virtual viewpoint that complies with the predetermined conditions is restricted, making it difficult to reconstruct the three-dimensional shape of the subject based on the output virtual viewpoint image.

[0152] As in the fourth modification, the configuration of the image processing system 2000 is not limited to that shown in Fig. 11. For example, the server device 2001 may operate on multiple physical devices or may operate on the cloud.

[0153] Alternatively, multiple client devices 2002 may connect to the server device 2001. Furthermore, similar to the fourth modification, multiple client devices 1002 may view volumetric video in a common virtual space. In this case, encryption settings may differ for each client device 2002. For example, a first user may encrypt virtual viewpoint video, while a second user may not encrypt the virtual viewpoint video. In such a case, the operation restriction unit 1602 may determine whether to restrict user operations on the video depending on whether all of the multiple client devices 2002 encrypt the virtual viewpoint video they output. For example, if at least one of the multiple client devices 2002 does not encrypt the virtual viewpoint video it outputs, the operation restriction unit 1602 may determine in step S1704 not to stop the volumetric video. The server device 2001 may reject connections from client devices 2002 that are not configured to encrypt the virtual viewpoint video they output.

[0154] Note that, similarly to the third embodiment, the encryption / decryption of the virtual viewpoint video may be dynamically switched. For example, the operation restriction unit 1602 may notify the encryption unit 1601 to encrypt the virtual viewpoint video to be output in response to an instruction to stop the volumetric video. In this case, when the encryption unit 1601 starts encrypting the virtual viewpoint video, it can transmit information indicating that the virtual viewpoint video is being encrypted to the operation restriction unit 1602. Upon receiving such information, the operation restriction unit 1602 can control the time designation unit 203 to stop the volumetric video. Furthermore, the operation restriction unit 1602 may notify the encryption unit 1601 to decrypt the virtual viewpoint video when playback of the volumetric video is resumed.

[0155] Other Embodiments In the third and fourth embodiments, a configuration has been described in which a user operation to stop volumetric video is restricted when a predetermined condition regarding the method of transmitting video to an output device is satisfied. In this manner, the operation restricting unit 1602 can restrict the output of video while the volumetric video is stopped in accordance with the predetermined condition. Here, the predetermined condition is not limited to when the video signal is not encrypted. For example, the operation restricting unit 1602 may restrict the video output operation while the volumetric video is stopped in accordance with the virtual viewpoint. Specifically, as in the first modification, the operation restricting unit 1602 may restrict the video output operation while the volumetric video is stopped when the parameters of the virtual viewpoint satisfy a predetermined condition. Furthermore, the operation restricting unit 1602 may control the viewpoint specifying unit 204 to reject the operation of the virtual viewpoint when the volumetric video is stopped, so that virtual viewpoint videos from various viewpoints are not output while the volumetric video is stopped.

[0156] Furthermore, in Modification Example 1, if the parameters of the virtual viewpoint satisfy a predetermined condition, the time designation unit 203 may not accept a user instruction to stop the volumetric video. Similarly, in Modification Example 4, if the parameters of the virtual viewpoint operated by at least one of the multiple client devices 1002 satisfy a predetermined condition, the time designation unit 203 may not accept a user instruction to stop the volumetric video. Furthermore, in Modification Examples 2 and 4, if a predetermined condition related to the number of times the virtual viewpoint has been saved is satisfied, the time designation unit 203 may not accept a user instruction to stop the volumetric video. In these cases as well, if the predetermined condition is satisfied, the output of images while the volumetric video is stopped is restricted.

[0157] In addition, in the first and second embodiments, a configuration has been described in which the saving operation of a virtual viewpoint image is restricted when a predetermined condition is satisfied. As described above, an image processing device according to an embodiment includes a video generation unit 205 or a communication unit 1003 that acquires multiple virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time. The image processing device also includes a video output unit 206 or a communication unit 1003 that outputs video represented by the multiple virtual viewpoint images. Here, the video output unit 206 or the communication unit 1003 can output video in a first mode in which multiple virtual viewpoint images of a three-dimensional model corresponding to different times are sequentially output so that a volumetric video is played. Furthermore, the video output unit 206 or the communication unit 1003 can output video in a second mode in which multiple virtual viewpoint images of a three-dimensional model corresponding to a specific time are sequentially output so that the volumetric video is stopped. The image processing device has a save control unit 209 or an operation restriction unit 1602 that, when certain conditions are met, allows video output in the first mode while restricting the video output operation or the saving operation of the virtual viewpoint image in the second mode.

[0158] According to this configuration, when a predetermined condition is satisfied, stopping of the volumetric video or saving of the virtual viewpoint image can be restricted. This makes it difficult to reconstruct the three-dimensional shape of a subject based on the stopped volumetric video or the saved virtual viewpoint image. For example, it makes it difficult to reconstruct the three-dimensional shape of a subject based on virtual viewpoint images from multiple directions output while the volumetric video is stopped. It also makes it difficult to reconstruct the three-dimensional shape of a subject based on saved virtual viewpoint images from multiple directions. On the other hand, even when the predetermined condition is satisfied, playback of the volumetric video can be permitted. This improves the user experience when watching the volumetric video.

[0159] The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0160] The present disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present disclosure. Accordingly, the following claims are appended to clarify the scope of the present disclosure.

[0161] This application claims priority based on Japanese Patent Application No. 2023-119300, filed on July 21, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing device comprising: an output means for outputting a plurality of virtual viewpoint images to an external device, each of which is an image including a three-dimensional model corresponding to a certain time; and a control means for limiting a process of saving a virtual viewpoint image displayed on the external device among the plurality of virtual viewpoint images, when a predetermined condition is satisfied.

2. The image processing device according to claim 1, wherein said control means prohibits the saving of said virtual viewpoint image in response to the determination that said predetermined condition is satisfied.

3. The image processing device according to claim 1, wherein said control means reduces the quality of said virtual viewpoint image to be stored in response to the determination that said predetermined condition is satisfied.

4. An image processing device as described in claim 1 or 3, characterized in that the control means performs control to reduce the virtual viewpoint image to be stored in response to a determination that the predetermined condition is satisfied.

5. An image processing device as described in claim 1, 3, or 4, characterized in that the control means controls the resolution of the virtual viewpoint image to be stored in response to determining that the specified condition is satisfied so that a portion corresponding to one component of the three-dimensional model is smaller than one pixel on the virtual viewpoint image.

6. An image processing device according to any one of claims 1 to 5, characterized in that the output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to a specified time, and the predetermined condition includes the same time being continuously specified.

7. The image processing device according to claim 6, wherein said control means performs control to continuously update said designated time in response to a user instruction to save said virtual viewpoint image.

8. An image processing device according to any one of claims 1 to 5, characterized in that the predetermined condition includes satisfying a first condition regarding a virtual viewpoint corresponding to the virtual viewpoint image.

9. The image processing device according to claim 8, wherein the output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to the specified time, and the control means prohibits the same time from being continuously specified in response to determining that the first condition is satisfied.

10. An image processing device according to any one of claims 1 to 5, characterized in that the predetermined condition includes satisfying a second condition related to the number of times the virtual viewpoint image is saved.

11. The image processing device described in claim 10, characterized in that the second condition includes a condition regarding the number of times a virtual viewpoint image of the three-dimensional model is saved at a specific time, or a condition regarding the number of times the virtual viewpoint image is saved within a specified period of real time.

12. An image processing device as described in claim 10 or 11, characterized in that the virtual viewpoint image is saved in accordance with user instructions from each of a plurality of users, and the number of times the virtual viewpoint image is saved is the total number of times the virtual viewpoint image is saved in accordance with the user instructions from each of the plurality of users.

13. The image processing device according to any one of claims 1 to 12, wherein the control means notifies a user that a restriction has been imposed on the saving operation of the virtual viewpoint image.

14. An image processing device comprising: an acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting an image represented by the plurality of virtual viewpoint images for display by an output device; and a control means for restricting user operations on the image when predetermined conditions relating to a method of communicating the image to the output device are satisfied.

15. The image processing device according to claim 14, wherein the control means restricts user operations on the image when it is determined that the image to be transmitted to the output device is not encrypted.

16. The image processing device described in claim 14 or 15, characterized in that the output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to a specified time, and the control means performs control to reject user operations that repeatedly specify the same time when it is determined that the image to be transmitted to the output device is not encrypted.

17. An image processing device as described in any one of claims 14 to 16, characterized in that the control means performs control to reject a user operation specifying the virtual viewpoint that satisfies a first condition in response to determining that the image to be transmitted to the output device is not encrypted.

18. The image processing device according to claim 16 or 17, wherein said control means performs control so that a prepared image is output to said output device when said user operation is rejected.

19. The image processing device according to any one of claims 14 to 18, wherein the control means notifies the user that the user operation is restricted.

20. An image processing device as described in any one of claims 1 to 19, further comprising a time designation means for designating a time, and a viewpoint designation means for designating a virtual viewpoint, wherein the output means outputs an image of the three-dimensional model from the designated virtual viewpoint corresponding to the designated time as the virtual viewpoint image.

21. An image processing device comprising: an acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting video represented by the plurality of virtual viewpoint images; and a control means for, when a predetermined condition is satisfied, permitting video output in a first mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to different times are sequentially output, while restricting video output operation or virtual viewpoint image storage operation in a second mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to a specific time are sequentially output.

22. An image processing method performed by an image processing device, comprising: a step of outputting a plurality of virtual viewpoint images, each of which is an image including a three-dimensional model corresponding to a certain time, to an external device; and a step of limiting the process of saving a virtual viewpoint image displayed on the external device among the plurality of virtual viewpoint images, when a predetermined condition is satisfied.

23. An image processing method performed by an image processing device, comprising the steps of: acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; outputting an image represented by the plurality of virtual viewpoint images for display by an output device; and restricting user operations on the image when predetermined conditions related to the communication method of the image to the output device are satisfied.

24. An image processing method performed by an image processing device, comprising the steps of: acquiring a plurality of virtual viewpoint images, each of which is an image taken from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; outputting an image represented by the plurality of virtual viewpoint images; and, when a predetermined condition is satisfied, permitting image output in a first mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to different times are sequentially output, while restricting the image output operation or the virtual viewpoint image storage operation in a second mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to a specific time are sequentially output.

25. A program for causing a computer to function as an image processing device according to any one of claims 1 to 21.

Images