Image processing apparatus, image processing method, and program

Abstract

To provide a technique for providing virtual viewpoint images, which makes it difficult to restore a three-dimensional shape of a subject based on plural virtual viewpoint images.SOLUTION: An image processing apparatus is configured to: acquire plural virtual viewpoint images in which each is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; output videos shown by the plural virtual viewpoint images for display; and limit storage operations of the displayed virtual viewpoint images that follow user instructions in the case when prescribed conditions are met.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, an image processing method, and a program, and particularly relates to a technique for generating a virtual viewpoint video.

Background Art

[0002] In recent years, a technique for generating a virtual viewpoint image has been known. For example, a technique called volumetric video has attracted attention. In this technique, multiple cameras are installed at different positions to perform synchronous shooting from multiple viewpoints. Then, using the images from multiple viewpoints obtained by such shooting, a virtual viewpoint image from an arbitrary viewpoint can be generated. According to such a technique, for example, highlight scenes of sports such as soccer or basketball can be viewed from an arbitrary angle. Also, it is possible to view a video from inside the field during a game, which was not achievable in the past. In this way, the user can view videos from various positions or angles different from conventional videos, that is, videos with added value.

[0003] Such a virtual viewpoint image can be generated, for example, by a server acquiring images from multiple cameras and performing a series of processes such as foreground / background separation, three-dimensional shape estimation, three-dimensional model generation, and rendering. On the other hand, cloud technology has been developing in recent years. Also, the processing capabilities of information devices such as PCs and tablet terminals have been improving. Furthermore, the communication speed has been increasing due to the development of the network infrastructure. For these reasons, operations of virtual viewpoints and video viewing, which were conventionally performed using dedicated devices, can now be performed using a general user's tablet terminal or PC.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When a user can freely manipulate a virtual viewpoint, the subject can obtain virtual viewpoint images of the subject from multiple aspects. By using these virtual viewpoint images, the user may be able to restore the three-dimensional shape of the subject. For example, such an operation can be performed by using an application having a photogrammetry function. As a result, the rights of the provider of the virtual viewpoint video content may be infringed.

[0006] Patent Document 1 discloses a technique for restricting a subject area for generating a virtual viewpoint image. According to the technique described in Patent Document 1, two images with different shooting directions are displayed on a display device. When two points are specified from the two images, a reconstructable area is specified based on the specified two points, and an image of this area is generated. However, such a method is insufficient for preventing the restoration of the three-dimensional shape data of the subject based on the virtual viewpoint images of the subject from multiple directions.

[0007] An object of the present disclosure is to make it difficult to restore the three-dimensional shape of a subject based on a plurality of virtual viewpoint images in a technique for providing virtual viewpoint images.

Means for Solving the Problems

[0008] An image processing apparatus according to an embodiment includes the following configuration. That is, output means for outputting a plurality of virtual viewpoint images, each of which is an image including a three-dimensional model corresponding to a certain time, to an external device; control means for restricting a process of saving a virtual viewpoint image displayed on the external device among the plurality of virtual viewpoint images when a predetermined condition is satisfied; and 、When it is determined that the video transmitted to the output device is not encrypted, the control means restricts user operations on the video .

Advantages of the Invention

[0009] In the technology of providing virtual viewpoint images, it is possible to make it difficult to restore the three-dimensional shape of a subject based on a plurality of virtual viewpoint images.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] An image processing apparatus according to an embodiment can be realized by a computer including a processor and a memory. FIG. 1 is a block diagram schematically showing a hardware configuration example of an image processing apparatus according to an embodiment. The image processing apparatus 100 includes a CPU 101, a main storage device 102, an auxiliary storage device 103, an external I / F 104, and a GPU 105. These components are connected to be communicable with each other via a bus 106.

[0013] The CPU 101 is a central processing unit. The CPU 101 controls the entire image processing apparatus 100. The CPU 101 can perform various operations and judgments, image processing, and data input / output.

[0014] The main storage device 102 can function as a work area of the CPU 101 or a temporary storage area for data. The main storage device 102 can be implemented using a storage medium such as a Dynamic Random Access Memory (DRAM) or a Static Random Access Memory (SRAM).

[0015] The auxiliary storage device 103 can store information such as various programs, various setting information, various image data, camera parameters, three-dimensional shape data, or two-dimensional maps. The auxiliary storage device 103 can be implemented using a non-volatile memory such as a Read Only Memory (ROM) or a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a storage medium such as a tape medium. The auxiliary storage device 103 may be implemented by a combination of a plurality of storage media for the purpose of increasing the capacity or speed. At this time, these storage media may be logically combined into one by using RAID technology or the like.

[0016] The external I / F 104 is a general-purpose input / output interface. The external I / F 104 can communicate with external devices such as a camera, a PC, or a server. Also, the external I / F 104 can be connected to an external device that accepts user operations such as a joystick, a keyboard, or a mouse. Further, the external I / F 104 can be connected to an external storage for file input / output. The external I / F 104 may be an interface having a connection terminal for a physical cable for InfiniBand, Ethernet (registered trademark), or Universal Serial Bus (USB). Also, the external I / F 104 may be a wireless interface for wireless LAN or Bluetooth.

[0017] The GPU 105 is an arithmetic unit for executing image processing operations at high speed. The GPU 105 may have a function of outputting a video signal to an output device such as a TV that can display external video. Further, the GPU 105 may execute a process of rendering a virtual viewpoint image. Furthermore, the GPU 105 may execute a process of generating shape data or color information necessary for generating a virtual viewpoint image. The GPU 105 may have a video terminal such as DVI, HDMI (registered trademark), Display Port, or SDI for outputting a video signal. Note that the GPU 105 may include a main storage device different from the main storage device 102. Also, the GPU 105 may perform some or all of various operations performed by the CPU 101 other than the processes shown above. Note that the GPU 105 may not have a video signal output function. For example, the GPU 105 may perform only operations such as image processing or generating shape data or color information necessary for generating a virtual viewpoint image. In this case, the external I / F 104 may further have a function of outputting a video output function or data necessary for video output.

[0018] The bus 106 is composed of a general-purpose bus such as PCI Express, and each part of the hardware block can perform two-way communication. In FIG. 1, all the hardware blocks are connected on the same bus 106, but the configuration is not limited to this. For example, the main storage device 102 and the CPU 101 may be connected by a bus different from the bus 106. In this case, communication may be performed between the main storage device 102 and other blocks via the CPU 101. Also, a hardware block such as the auxiliary storage device 103 or the GPU 105 may be connected to the external I / F 104. In this case, communication with the auxiliary storage device 103 or the GPU 105 can be performed via the external I / F 104. Note that the type of the bus 106 is not limited. As the bus 106, any bus that enables each block to perform two-way communication can be used. For example, the bus 106 may be a bus that conforms to a standardized communication standard such as USB. The bus 106 may be a dedicatedly designed communication bus.

[0019] In this way, by a processor such as the CPU 101 executing a program stored in a memory such as the main storage device 102 or the auxiliary storage device 103, the functions of each part shown in FIG. 2 and the like described later can be realized. Note that the configuration shown in FIG. 1 is merely an example of the configuration of the image processing apparatus 100. For example, some or all of the functions of the image processing apparatus 100 may be realized by dedicated hardware. Also, some of the functions of the image processing apparatus 100 may be realized by externally connected hardware. Further, the main storage device 102 and the auxiliary storage device 103 may be realized by one piece of hardware. Furthermore, instead of the GPU 105, other hardware such as an FPGA, a DSP, or a dedicated image processing LSI may be used.

[0020] Also, the image processing apparatus 100 does not necessarily have to be configured by a single device. For example, a plurality of devices having the same configuration may be prepared. At this time, the configurations of the individual devices may be different. Also, the image processing apparatus according to one embodiment may be configured by, for example, a plurality of information processing devices connected via a network. In one embodiment, the functions of the image processing apparatus 100 can be provided as a cloud service.

[0021] [Embodiment 1] Hereinafter, an image processing apparatus according to an embodiment will be described with reference to the drawings. In the following embodiments, the image processing apparatus 100 generates and outputs a virtual viewpoint video represented by a plurality of virtual viewpoint images. However, the image processing apparatus 100 may acquire a virtual viewpoint video from another device and output the acquired virtual viewpoint video. In the following embodiments, the image processing apparatus 100 has a function of storing virtual viewpoint images constituting the virtual viewpoint video. Such a storage function is also called an image capture function. That is, the image processing apparatus 100 can store a virtual viewpoint image displayed on an output device according to a user instruction. In the following example, the stored virtual viewpoint image is a two-dimensional image. Note that the storage method is not particularly limited. In one embodiment, the output virtual viewpoint image is stored. In another embodiment, a screen shot of the display on the output device is stored. In a further embodiment, a virtual viewpoint image based on the same time code and virtual viewpoint as the virtual viewpoint image displayed on the output device is generated by the video generation unit 205, and the generated virtual viewpoint image is stored.

[0022] Furthermore, in the present embodiment, the storage operation of the virtual viewpoint image is restricted according to a predetermined condition. First, a case where the storage operation of the virtual viewpoint image in a state where the reproduction of the virtual viewpoint video is stopped will be described below.

[0023] In the following example, the image processing apparatus 100 can reproduce a volumetric video. The volumetric video is generated based on a three-dimensional model corresponding to each time. More specifically, the volumetric video is generated based on a three-dimensional model associated with a time code representing time. The three-dimensional model can change according to time. The image processing apparatus 100 can generate an image of the three-dimensional model corresponding to a specific time from a virtual viewpoint as a virtual viewpoint image. Note that image processing such as adding a background to the virtual viewpoint image may be further performed.

[0024] As described below, the video output unit 206 of the image processing apparatus 100 sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to the designated time. Here, playing a volumetric video corresponds to continuously updating the designated time. That is, a plurality of virtual viewpoint images are generated based on three-dimensional models corresponding to different times respectively. The virtual viewpoint video composed of these plurality of virtual viewpoint images is played as a volumetric video. Hereinafter, sequentially displaying the virtual viewpoint images generated based on the three-dimensional models corresponding to each of the consecutive times is referred to as playing the volumetric video.

[0025] On the other hand, a plurality of virtual viewpoint images may be generated based on a specific three-dimensional model corresponding to a specific time. In this specification, continuously displaying the virtual viewpoint images generated based on a specific three-dimensional model corresponding to a specific time is referred to as stopping the volumetric video. Stopping the volumetric video corresponds to continuously designating the same time. Stopping the volumetric video includes sequentially displaying a plurality of virtual viewpoint images generated based on a specific three-dimensional model corresponding to a specific time. These plurality of virtual viewpoint images may be generated based on different virtual viewpoints. Thus, when the volumetric video is stopped, the video output unit 206 of the image processing apparatus 100 can output a virtual viewpoint video composed of virtual viewpoint images from various directions for the three-dimensional model corresponding to the same time. On the other hand, stopping the volumetric video may include continuously displaying the same one virtual viewpoint image. Stopping the volumetric video includes the case of temporarily stopping the progress of the time code (i.e., pausing).

[0026] FIG. 2 is a schematic diagram showing a functional configuration example of the image processing apparatus 100 according to an embodiment. The image processing apparatus 100 includes a model acquisition unit 201, a space configuration unit 202, a time designation unit 203, a viewpoint designation unit 204, a video generation unit 205, a video output unit 206, an image storage unit 207, an image storage section 208, and a storage control unit 209.

[0027] The model acquisition unit 201 acquires data of the three-dimensional model corresponding to each time. In the present embodiment, the model acquisition unit 201 acquires the three-dimensional model of the volumetric video associated with the time code instructed by the space configuration unit 202. The three-dimensional model can represent, for example, sports such as soccer, or contents such as concerts or plays. Specifically, the three-dimensional model can represent the three-dimensional shape of an object. Also, the three-dimensional model can represent the texture of the object. The model acquisition unit 201 may acquire three-dimensional models for each of a plurality of objects. For example, the three-dimensional model can represent the three-dimensional shape of a person such as a player or a performer, or an object such as a ball. Also, the three-dimensional model may represent the three-dimensional shape of a structure such as a stadium or stage equipment serving as a background. Note that some objects may not change over time. For example, the background structure may be stationary regardless of the time code. The three-dimensional model corresponding to such an object may be associated with a plurality of time codes. Also, a range of consecutive time codes may be associated with one three-dimensional model.

[0028] The model acquisition unit 201 may acquire these three-dimensional models via a network such as the Internet or a local area network (LAN). The model acquisition unit 201 may read these three-dimensional models from a storage medium such as an optical medium, a USB memory, an HDD, or a magnetic tape medium. The model acquisition unit 201 may acquire the three-dimensional models associated with each of all the time codes at once. Also, the model acquisition unit 201 may sequentially acquire the three-dimensional models necessary for the space configuration unit 202 to configure a virtual space for a specific time code.

[0029] The space configuration unit 202 constructs a three-dimensional virtual space. The space configuration unit 202 can construct a virtual space corresponding to a specific time code specified by the time specifying unit 203. Specifically, the space configuration unit 202 can acquire a three-dimensional model associated with a specific time code from the model acquisition unit 201. Then, the space configuration unit 202 can place the acquired three-dimensional model in the virtual space.

[0030] The time specifying unit 203 specifies a time. The video generation unit 205, which will be described later, generates a virtual viewpoint image of a three-dimensional model corresponding to the time specified by the time specifying unit 203. In the present embodiment, the time specifying unit 203 can specify a time code. Then, the time specifying unit 203 outputs the specified time code to the space configuration unit 202 so as to construct a three-dimensional virtual space corresponding to the time code.

[0031] During normal playback, the time specifying unit 203 advances the time code as the real time elapses. For example, the time specifying unit 203 can advance the time code so as to advance 60 frames per second. On the other hand, the time specifying unit 203 can stop the count-up of the time code. In this case, the volumetric video output or displayed stops. Note that the number of frames per second during normal playback is not limited to 60 frames. The number of frames per second may be, for example, 50 frames, 30 frames, 24 frames, 59.94 frames, 29.97 frames, or 23.976 frames.

[0032] The time specifying unit 203 can control the time code according to an instruction from the user. For example, when receiving a pause instruction from the user, the time specifying unit 203 can stop the count-up of the time code. Also, when receiving a fast-forward or rewind instruction from the user, the time specifying unit 203 can increase the number of frames counted up or down per second. Furthermore, the time specifying unit 203 can also perform time code control for special playback. For example, the time specifying unit 203 can accept an operation of directly specifying the time code.

[0033] The viewpoint specifying unit 204 specifies a virtual viewpoint. The viewpoint specifying unit 204 can set a virtual viewpoint within the virtual space configured by the space configuring unit 202. The virtual viewpoint corresponds to a virtual camera that performs shooting within the virtual space. The virtual camera is a concept for explaining the virtual viewpoint used for generating the virtual viewpoint image. The virtual camera is a virtual camera different from a plurality of imaging devices actually installed around the imaging area for generating a three-dimensional model. That is, the virtual viewpoint image can be regarded as an imaging image from the virtual viewpoint set within the virtual space associated with the imaging area. In other words, it can be said that the virtual viewpoint image is an image imitating the imaging image obtained by the camera when assuming that a camera exists at the position of the virtual viewpoint set within the virtual space. The video generating unit 205 described later generates an image of a three-dimensional model arranged within the virtual space from the virtual viewpoint as the virtual viewpoint image.

[0034] The viewpoint specifying unit 204 can operate the virtual viewpoint based on a user operation. For example, the viewpoint specifying unit 204 can move the virtual viewpoint within the virtual space. Also, the viewpoint specifying unit 204 can set parameters of the virtual viewpoint. Specifically, the viewpoint specifying unit 204 can perform camera operations such as a zoom operation. The position and orientation of the virtual viewpoint thus set correspond to the position and orientation of the virtual camera. In this way, a virtual viewpoint image from a free viewpoint can be generated.

[0035] In this embodiment, the viewpoint specifying unit 204 sets the camera parameters of the virtual camera corresponding to the virtual viewpoint. The camera parameters can include external parameters and internal parameters. The information included in the camera parameters is not particularly limited. The camera parameters can include, for example, parameters representing the three-dimensional position of the camera and information representing the three-dimensional orientation of the camera. The orientation of the camera can be represented by information representing rotations in the pan, tilt, and roll directions. Also, the camera parameters may include information such as the size of the field of view (angle of view), the number of pixels of the virtual sensor, or the focal length.

[0036] Note that the method for setting the virtual viewpoint is not particularly limited. For example, the virtual viewpoint may be associated with a character or object in the virtual space such as an avatar operated by the user. For example, the virtual camera may correspond to the field of view of the character. Also, the virtual viewpoint may be set so that the relative position with respect to these characters or objects is fixed. Even in this case, the virtual viewpoint can move following the movement of the character or object.

[0037] The image generation unit 205 can acquire a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of the three-dimensional model corresponding to a certain time. As described above, the image generation unit 205 can generate, as a virtual viewpoint image, an image of the three-dimensional model corresponding to the time specified by the time specification unit 203 from the virtual viewpoint specified by the viewpoint specification unit 204. For example, as described above, the image generation unit 205 can generate, as a virtual viewpoint image, an image of the three-dimensional model from the specified virtual viewpoint in the virtual space corresponding to a specific time code. The image generation unit 205 can generate the virtual viewpoint image as a two-dimensional image in a format that can be viewed by the user. Also, the image generation unit 205 can sequentially generate virtual viewpoint images of the three-dimensional model corresponding to the specified time. For example, the image generation unit 205 can generate a plurality of virtual viewpoint images for virtual spaces corresponding to different time codes. Also, the image generation unit 205 can generate a plurality of virtual viewpoint images for the virtual space corresponding to the same time code. Furthermore, the image generation unit 205 can generate a plurality of virtual viewpoint images corresponding to different virtual viewpoints. In this way, the image generation unit 205 can generate a virtual viewpoint video composed of a plurality of virtual viewpoint images.

[0038] The video output unit 206 outputs, for display, the video represented by a plurality of virtual viewpoint images. The video output unit 206 can sequentially output the virtual viewpoint images of the three-dimensional model corresponding to the specified time generated by the image generation unit 205. For example, the video output unit 206 can output a virtual viewpoint video composed of a plurality of virtual viewpoint images generated by the image generation unit 205 to an output device such as a display. At this time, the video output unit 206 may output the virtual viewpoint video via a video signal output terminal such as HDMI (registered trademark), DisplayPort, or SDI.

[0039] The image storage unit 207 stores the virtual viewpoint image. The image storage unit 207 can store the virtual viewpoint image according to a user instruction. Also, the image storage unit 207 can store the displayed virtual viewpoint image. Specifically, the image storage unit 207 can store, among the virtual viewpoint images constituting the virtual viewpoint video, the virtual viewpoint image output by the video output unit 206 or the virtual viewpoint image displayed on the output device.

[0040] The image storage unit 207 can store the virtual viewpoint image generated by the video generation unit 205 in the image storage unit 208. The image storage unit 207 can store the virtual viewpoint image as a two-dimensional image file conforming to a format such as JPEG, BMP, PNG, or MPEG. Here, the operation of storing the virtual viewpoint image by the image storage unit 207 is restricted by the storage control unit 209. In this example, the image storage unit 207 stores the virtual viewpoint image in the image storage unit 208 in response to the storage control unit 209 determining that storage is permitted.

[0041] The storage control unit 209 restricts the operation of storing the virtual viewpoint image when a predetermined condition is satisfied. In this example, the storage control unit 209 determines whether to permit the storage of the virtual viewpoint image. The storage control unit 209 can prohibit the storage of the virtual viewpoint image in response to being determined that a predetermined condition is satisfied. Thus, the storage control unit 209 can determine whether the virtual viewpoint image can be stored as a two-dimensional image file.

[0042] In this example, the predetermined condition is that the volumetric video is stopped. That is, the storage control unit 209 determines whether the virtual viewpoint image can be stored according to whether the volumetric video is stopped. The storage control unit 209 determines that the virtual viewpoint image can be stored in response to determining that the volumetric video is being played. On the other hand, the storage control unit 209 determines not to permit the storage of the virtual viewpoint image in response to determining that the volumetric video is stopped.

[0043] Note that the storage control unit 209 may notify the user that the storage operation of the virtual viewpoint image is restricted. For example, when image storage is not permitted, the storage control unit 209 can notify that the image cannot be stored. The storage control unit 209 can notify the user via an output device that is displaying the virtual viewpoint video.

[0044] FIG. 3 is a flowchart showing the processing flow of the image processing method performed by the image processing apparatus 100 according to an embodiment. According to the flowchart shown in FIG. 3, a virtual space image is generated. Also, by repeating the flowchart shown in FIG. 3, a plurality of virtual space images are generated, that is, a virtual space video is generated.

[0045] In S301, the time specifying unit 203 sets a time code. When the volumetric video is stopped, the time specifying unit 203 sets the time code to a fixed value. In this case, the configuration of the virtual space constructed in S302 becomes fixed. For this reason, stopping of the volumetric video is realized. On the other hand, when the volumetric video is being played back, the time specifying unit 203 counts up the time code.

[0046] In S302, the space configuration unit 202 acquires, from the model acquisition unit 201, a three-dimensional model of the volumetric video corresponding to the time code specified by the time specifying unit 203. Then, the space configuration unit 202 configures the virtual space as described above.

[0047] In S303, the viewpoint specifying unit 204 sets parameters of the virtual viewpoint according to the user's operation as described above. In this way, the virtual viewpoint of the volumetric video is determined.

[0048] In S304, the video generation unit 205 generates a virtual viewpoint image that the user can view according to the parameters of the virtual viewpoint set in S303. The virtual viewpoint image generated in this way may be displayed on an output device.

[0049] FIG. 4 is a diagram for explaining the generation of a virtual viewpoint image based on volumetric video data created by photographing a soccer game. FIG. 4 schematically shows a virtual space at a certain time and the state of shooting using a virtual camera within this virtual space. FIG. 4(A) is a schematic diagram showing the state of the virtual space from an aerial view above the stadium. In the soccer field 401, a soccer game is being played. The spectator seats 402 are the spectator seats in the soccer stadium. Seats are installed in the spectator seats 402 in the same way as in the real-world soccer stadium.

[0050] The player 403 is a soccer player. The player 403 is represented as a three-dimensional model in the volumetric video. Hereinafter, for the sake of explanation, one player 403 on the soccer field 401 will be described. However, there may be two or more players in the virtual space. Usually, in a soccer game, there are 11 players per team, that is, a total of 22 players for the two competing teams, and in addition, referees and the like are present on the soccer field 401.

[0051] The virtual cameras 404a and 404b are virtual cameras operated by the viewpoint designating unit 204. The virtual cameras 404a and 404b virtually photograph the player 403. Note that these virtual cameras 404a and 404b may be invisible within the virtual space. For example, when the virtual camera 404b facing the virtual camera 404a performs imaging, the virtual camera 404a may not appear in the captured image of the virtual camera 404b.

[0052] The virtual cameras 404a and 404b can be moved to any location, including the field where a soccer game is being played. Also, the virtual cameras 404a and 404b can move freely three-dimensionally in the vertical, horizontal, and height directions. However, the positions of the virtual cameras 404a and 404b may be restricted. For example, the virtual cameras 404a and 404b may be able to move only within a predetermined area such as the spectator seats 402. Also, an area where the virtual cameras 404a and 404b cannot move may be set in the virtual space. Also, as described above, the virtual viewpoint may be the viewpoint of an avatar arranged in the virtual space. In this case, the movable range of the avatar may be restricted to a predetermined area such as within the spectator seats 402. Further, the movement of the virtual viewpoint in the height direction may be restricted within a certain range. The certain range in this case may be set according to the height of the avatar.

[0053] FIG. 4(B) schematically shows an example of a virtual viewpoint image obtained by the virtual camera 404a. FIG. 4(C) schematically shows an example of a virtual viewpoint image obtained by the virtual camera 404b. The video generation unit 205 can render these two-dimensional images so that the user can view them.

[0054] In S305, the image storage unit 207 determines whether there is an instruction from the user to save the virtual viewpoint image. If it is determined that there is an instruction to save the virtual viewpoint image, the process proceeds to S306. If it is determined that there is no instruction to save the virtual viewpoint image, the process shown in FIG. 3 ends.

[0055] In S306, the save control unit 209 determines whether to restrict the save operation of the virtual viewpoint image. In this example, the save control unit 209 determines whether the volumetric video has stopped. The save control unit 209 can inquire of the time specifying unit 203 whether the volumetric video has stopped. If it is determined that the volumetric video has stopped, the process proceeds to S307. Otherwise, the process proceeds to S308.

[0056] In S307, the virtual viewpoint image is not saved by the image storage unit 207. In this case, the image storage unit 207 can notify the user who gave the save instruction of the virtual viewpoint image that the save cannot be performed. For example, the image storage unit 207 can display a UI indicating that the save cannot be performed on the output device that displays the virtual viewpoint video.

[0057] In S308, the image storage unit 207 saves the virtual viewpoint image generated in S304 in the image storage unit 208.

[0058] According to the above embodiment, the virtual viewpoint image cannot be saved when the volumetric video is stopped. For example, the user may attempt to save a virtual viewpoint image of the three-dimensional model corresponding to a specific time code from the position of the virtual camera 404a as shown in FIG. 4(B) while the volumetric video is stopped. Further, the user may attempt to save a virtual viewpoint image of the three-dimensional model corresponding to the same time code from the position of the virtual camera 404b as shown in FIG. 4(C) by moving the virtual viewpoint. However, in the above embodiment, the virtual viewpoint image cannot be saved when the three-dimensional model of the subject is stationary. Therefore, it becomes difficult to save a plurality of virtual viewpoint images of the three-dimensional model corresponding to the same time code. Thus, it can be made difficult to restore the three-dimensional shape of the subject based on the virtual viewpoint image.

[0059] (Modification Example 1) As described above, the storage control unit 209 restricts the storage operation of the virtual viewpoint images by the image storage unit 207 according to a predetermined condition. In the above example, the storage control unit 209 prohibited the storage of the virtual viewpoint images by the image storage unit 207 when the volumetric video was stopped. On the other hand, the predetermined condition is not limited to the volumetric video being stopped. Hereinafter, a case where the predetermined condition includes satisfying a first condition regarding the virtual viewpoint will be described. In the following example, the storage control unit 209 can restrict the storage operation of the virtual viewpoint images by the image storage unit 207 according to the designated virtual viewpoint.

[0060] For example, the storage control unit 209 can restrict the storage operation of the virtual viewpoint images by the image storage unit 207 based on whether the virtual viewpoint satisfies the first condition. More specifically, the storage control unit 209 can determine whether to restrict the storage operation of the virtual viewpoint images by the image storage unit 207 based on whether the parameters of the virtual viewpoint satisfy the first condition. For example, the storage control unit 209 can determine whether to restrict the storage operation of the virtual viewpoint images by the image storage unit 207 according to the position of the virtual viewpoint designated by the viewpoint designation unit 204. In one embodiment, the first condition is that the position of the virtual viewpoint is not within a specific range. For example, the storage control unit 209 can determine to restrict the storage operation of the virtual viewpoint images in response to determining that the position of the virtual viewpoint is not within the specific range.

[0061] This process will be described with reference to FIG. 5(A). The shooting permission area 701 indicates the range in which shooting by the virtual camera (i.e., storage of virtual viewpoint images) is permitted. Since the virtual camera 404a is within the shooting permission area 701, the storage control unit 209 permits the storage of the virtual viewpoint images obtained by the virtual camera 404a. On the other hand, since the virtual camera 404b is not within the shooting permission area 701, the storage control unit 209 does not permit the storage of the virtual viewpoint images obtained by the virtual camera 404a.

[0062] In addition, the storage control unit 209 can determine whether to restrict the operation of storing the virtual viewpoint image by the image storage unit 207 according to the orientation of the virtual viewpoint specified by the viewpoint specifying unit 204, or a combination of the position and orientation. For example, the storage control unit 209 can determine to restrict the operation of storing the virtual viewpoint image according to the determination that the orientation of the virtual viewpoint is not within a specific range.

[0063] This process will be described with reference to FIGS. 5(B) and 5(C). The shooting permission direction range 703 indicates the range of the orientation of the virtual camera in which shooting by the virtual camera is permitted. As shown in FIG. 5(B), since the orientation 702 of the virtual camera 404a is within the shooting permission direction range 703, the storage control unit 209 permits the storage of the virtual viewpoint image obtained by the virtual camera 404a. On the other hand, as shown in FIG. 5(C), since the orientation 704 of the virtual camera 404b is not within the shooting permission direction range 703, the storage control unit 209 does not permit the storage of the virtual viewpoint image obtained by the virtual camera 404a.

[0064] The conditions regarding the position of the virtual viewpoint and the conditions regarding the orientation of the virtual viewpoint as described above may be combined. For example, when the orientation 702 of the virtual camera 404a is within the shooting permission direction range 703 and the virtual camera 404a is within the shooting permission area 701, the storage control unit 209 may permit the storage of the virtual viewpoint image obtained by the virtual camera 404a. That is, when the virtual camera 404a is facing the orientation 704, even if the virtual camera 404a is within the shooting permission area 701, the storage control unit 209 can restrict the operation of storing the virtual viewpoint image obtained by the virtual camera 404a. In this way, conditions regarding the camera parameters of the virtual camera, such as the shooting permission area 701 and the shooting permission direction range 703, can be combined as appropriate.

[0065] Furthermore, the storage control unit 209 can determine whether to limit the storage operation of the virtual viewpoint image by the image storage unit 207 according to the resolution of the three-dimensional model of the subject on the virtual viewpoint image. Such a resolution can be determined based on the parameters of the virtual viewpoint. For example, the resolution can be represented by the number of pixels of the virtual viewpoint image of the part corresponding to one three-dimensional shape component such as a voxel or a texture of the three-dimensional model corresponding to the subject. Note that the three-dimensional shape component corresponding to the texture can be, for example, one pixel constituting the texture, or the entire texture represented by vertex coordinates. In such an example, the first condition may be that the resolution is smaller than one pixel. Such a resolution can be determined based on the position, angle of view, and number of pixels of the virtual camera. Therefore, the condition regarding such a resolution is also included in the first condition regarding the virtual viewpoint.

[0066] In such a modification, in S306, the storage control unit 209 determines whether to limit the storage operation of the virtual viewpoint image according to the virtual viewpoint as described above. When it is determined to limit the storage operation of the virtual viewpoint image by the image storage unit 207 according to the above conditions, the process proceeds to S307. Otherwise, the process proceeds to S308.

[0067] Note that when the parameters of the virtual viewpoint satisfy the first condition, the storage control unit 209 may further control the time specifying unit 203 so as to prohibit the stop of the volumetric video.

[0068] According to this modification, the virtual viewpoints corresponding to the stored virtual viewpoint images are limited. Therefore, it is possible to suppress storing virtual viewpoint images of the three-dimensional model from various directions in order to restore the three-dimensional shape of the subject. Further, by bringing the virtual viewpoint closer to the three-dimensional model, it is possible to suppress storing virtual viewpoint images of the three-dimensional model having a high resolution. Thus, according to this modification, it is possible to make it difficult to restore the three-dimensional shape of the subject.

[0069] (Modification 2) Hereinafter, a case will be described in which a predetermined condition includes satisfying a second condition regarding the number of times of storing a virtual viewpoint image. For example, the storage control unit 209 can limit the storage operation of the virtual viewpoint image by the image storage unit 207 according to the number of times of storing the virtual viewpoint image. The second condition may include a condition regarding the number of times of storing the virtual viewpoint image of the three-dimensional model at a specific time. For example, the second condition may be that the number of times of storing the virtual viewpoint image at a specific time code is equal to or more than a predetermined number of times. In such an example, a limit can be set on the number of virtual viewpoint images that can be stored at the same time code of the volumetric video.

[0070] In this modification example, the image storage unit 207 stores, in association with the virtual viewpoint image, image recording information corresponding to the virtual viewpoint image in the image storage unit 208. This image recording information may include time information related to the virtual viewpoint image. This time information may indicate the time code corresponding to the virtual viewpoint image. For example, the image storage unit 207 can store the time code of the volumetric video being played back, which is acquired from the time specifying unit 203, as the image recording information. On the other hand, the time information may indicate the time at the actual time when the virtual viewpoint image is stored. The recorded time may be a time based on Coordinated Universal Time. Further, the image recording information may include information on the virtual viewpoint corresponding to the virtual viewpoint image. For example, the image recording information may include camera parameters such as external parameters and internal parameters set by the viewpoint specifying unit 204 when the image storage unit 207 stores the virtual viewpoint image.

[0071] Further, the storage control unit 209 determines whether to limit the storage operation of the virtual viewpoint image according to the number of times the virtual viewpoint image is stored. For example, the storage control unit 209 can obtain the current time code specified by the time specifying unit 203. Then, the storage control unit 209 refers to the image storage unit 208 to determine the number of times the past virtual viewpoint image has been stored at the current time code. Note that the number of times of storage at the current time code may be the number of times of storage at the same time code as the current time code. On the other hand, the number of times of storage at the current time code may be the number of times of storage at a time code in the vicinity of the current time code. For example, the storage control unit 209 may determine the number of times the virtual viewpoint image is stored within a predetermined period (for example, within 1 second before and after) in the vicinity of the current time code. Also, the volumetric video may be divided into a plurality of scenes in advance. For example, the volumetric video may be divided in units such as every 1 second according to the time code. In this case, the storage control unit 209 can determine the number of times of storage in the same scene as the current time code.

[0072] In this example, the second condition is that this number of times of storage is equal to or greater than a predetermined number of times. That is, in response to determining that the user has stored the virtual viewpoint image for the current time code more than a specified number of times in the past, the storage control unit 209 limits the storage operation of the virtual viewpoint image. On the other hand, in response to determining that the user has not stored the virtual viewpoint image for the current time code more than a specified number of times in the past, the storage control unit 209 permits the storage of the virtual viewpoint image. Note that the specified number of times is not particularly limited and may be 1 time, or may be 2 times, 3 times, or 4 times or more.

[0073] In another embodiment, the storage control unit 209 may determine the number of times the virtual viewpoint image is stored for the entire volumetric video. In this example, the second condition is that the number of times of storage for the entire volumetric video is equal to or greater than a predetermined number of times. In such an example, the number of times the virtual viewpoint image is stored for the entire volumetric video can be limited.

[0074] In another embodiment, the storage control unit 209 may determine the number of times (i.e., storage frequency) of storing the virtual viewpoint images within a predetermined period of real time. In this example, the second condition is that this number of times of storage is equal to or greater than a threshold value. For example, in response to determining that the virtual viewpoint images have already been stored 10 times within 15 minutes in real time, the storage control unit 209 can limit the storage operation of the virtual viewpoint images. Note that the length of the predetermined time is not particularly limited and may be longer than 15 minutes, such as 1 hour, 1 day, or 1 week, or may be shorter than 15 minutes, such as 5 minutes or 1 minute. Also, the threshold value is not particularly limited and may be, for example, more or less than 10 times.

[0075] Further, the storage control unit 209 may determine whether to limit the storage operation of the virtual viewpoint images using the above combinations of conditions.

[0076] In such a modification, in S306, the storage control unit 209 determines whether to limit the storage operation of the virtual viewpoint images according to the number of times of storing the virtual viewpoint images as described above. If it is determined to limit the storage operation of the virtual viewpoint images by the image storage unit 207 according to the above conditions, the process proceeds to S307. Otherwise, the process proceeds to S308.

[0077] According to this modification, the number of times of storing the virtual viewpoint images is limited. Therefore, it becomes difficult for the user to store more virtual viewpoint images than a predetermined number. That is, in order to restore the three-dimensional shape of the subject, it is possible to suppress the user from storing a large number of virtual viewpoint images of the three-dimensional model from various directions. Thus, according to this modification, it is possible to make it difficult to restore the three-dimensional shape of the subject.

[0078] (Modification 3) So far, examples of restricting the saving operation of virtual viewpoint images when the volumetric video is stopped, examples of restricting the saving operation of virtual viewpoint images according to the virtual viewpoint, and examples of restricting the saving operation of virtual viewpoint images according to the number of saves have been described. These conditions can be used in combination. The combination method of the conditions is not particularly limited. For example, when all of a plurality of conditions are satisfied, it may be determined that a predetermined condition is satisfied. Also, when at least one of a plurality of conditions is satisfied, it may be determined that a predetermined condition is satisfied.

[0079] For example, according to a determination that either one or both of the condition that the volumetric video is stopped and the condition regarding the parameters of the virtual viewpoint are satisfied, the saving control unit 209 may restrict the saving operation of the virtual viewpoint image. Also, according to a determination that either one or both of the condition that the volumetric video is stopped and the condition regarding the number of saves of the virtual viewpoint image are satisfied, the saving control unit 209 may restrict the saving operation of the virtual viewpoint image. Further, according to a determination that either one or both of the condition regarding the parameters of the virtual viewpoint and the condition regarding the number of saves of the virtual viewpoint image are satisfied, the saving control unit 209 may restrict the saving operation of the virtual viewpoint image.

[0080] In the above example, when the saving control unit 209 restricts the saving operation of the virtual viewpoint image, the image saving unit 207 does not save the virtual viewpoint image in the image storage unit 208. However, the mode of restriction by the saving control unit 209 is not limited to the above. For example, the saving control unit 209 may reduce the quality of the saved virtual viewpoint image according to a determination that a predetermined condition is satisfied. For example, the saving control unit 209 can control the image saving unit 207 to reduce the saved virtual viewpoint image according to a determination that a predetermined condition is satisfied. In this case, the image saving unit 207 can save the reduced virtual viewpoint image in the image storage unit 208. In this way, the image saving unit 207 can lower the resolution of the three-dimensional model in the virtual viewpoint image saved in the image storage unit 208.

[0081] As described above, the resolution of the three-dimensional model in the virtual viewpoint image can be represented by the number of pixels on the virtual viewpoint image corresponding to one three-dimensional shape component of the three-dimensional model corresponding to the subject. In this case, the storage control unit 209 can control the resolution of the three-dimensional model in the virtual viewpoint image stored by the image storage unit 207 in the image storage unit 208 to be less than one pixel.

[0082] As another example, when the storage control unit 209 restricts the storage operation of the virtual viewpoint image, the playback state of the volumetric video or the parameters of the virtual viewpoint may be changed so that the storage of the virtual viewpoint image is permitted. For example, the storage control unit 209 can control the time specifying unit 203 to release the stop state of the volumetric video and play the volumetric video in response to a user instruction to store the virtual viewpoint image. Even in such a configuration, the storage of the virtual viewpoint image when the volumetric video is stopped is not permitted, while the playback of the volumetric video is restarted in response to a user instruction, so that it becomes possible to store the virtual viewpoint image.

[0083] [Embodiment 2] Hereinafter, the image processing system 1000 according to an embodiment will be described with reference to FIG. 6. FIG. 6 shows a functional configuration example of the image processing system 1000 according to the present embodiment. The image processing system 1000 includes a server device 1001 and a client device 1002. Also in the present embodiment, similar to Embodiment 1, the saving operation of the virtual viewpoint image is restricted according to a predetermined condition. On the other hand, it is different from Embodiment 1 in that the server device 1001 performs rendering processing of the volumetric video, and the virtual viewpoint video is sent to the client device 1002 by communication. Specifically, the server device 1001 generates a virtual viewpoint video. Further, the client device 1002 receives the virtual viewpoint video from the server device 1001 and plays back the virtual viewpoint video. In the present embodiment, the client device 1002 does not receive the three-dimensional shape data of the subject. Note that the hardware configurations of the server device 1001 and the client device 1002 may be the same as the hardware configuration of the image processing device 100.

[0084] The server device 1001 includes a model acquisition unit 201, a space configuration unit 202, a time specification unit 203, a viewpoint specification unit 204, a video generation unit 205, a saving control unit 209, and a communication unit 1003. Also, the client device 1002 includes a video output unit 206, an image saving unit 207, an image storage unit 208, a communication unit 1003, and an operation unit 1004. The same reference numerals are assigned to the functional units similar to those in Embodiment 1. Hereinafter, the differences from Embodiment 1 will be described.

[0085] The communication unit 1003 communicates with the server device 1001 or the client device 1002. Using the communication unit 1003, the server device 1001 and the client device 1002 can perform two-way communication. The configuration of the communication unit 1003 can be selected so that the transmission bandwidth required for communication between the server device 1001 and the client device 1002 is ensured. The communication unit 1003 may perform communication via a network. For example, the communication unit 1003 may perform communication via a local network such as Ethernet. In this case, the communication unit 1003 can perform communication via, for example, various switching hubs or router devices. Also, at least a part of the network may be the Internet or a dedicated line. In this case, the server device 1001 and the client device 1002 can be installed at physically separated locations.

[0086] The operation unit 1004 receives user operations. Also, the operation unit 1004 can send information indicating a user operation to each part of the server device 1001, such as the time specifying unit 203 and the viewpoint specifying unit 204, via communication by the communication unit 1003. For example, the operation unit 1004 can receive a user operation for a virtual viewpoint and send information indicating the user operation to the viewpoint specifying unit 204. Also, the operation unit 1004 can receive a user operation related to the playback of a volumetric video. The operation unit 1004 can receive, for example, user operations for specifying, playing, stopping, and performing special playback of time codes. The operation unit 1004 can send information indicating these user operations to the time specifying unit 203. In this way, the user can operate the displayed virtual viewpoint video. Furthermore, the operation unit 1004 can receive a user operation for saving a virtual viewpoint image. The image storage unit 207 can save the virtual viewpoint image in the image storage unit 208 according to such a user operation.

[0087] In the server device 1001 according to the present embodiment, the video generation unit 205 acquires a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time. Further, the communication unit 1003 outputs, for display, the video represented by the plurality of virtual viewpoint images generated by the video generation unit 205 to the client device 1002. The video data transmitted by the communication unit 1003 may be compressed by an arbitrary encoding method such as MPEG or H.264. Then, the storage control unit 209 restricts the storage operation of the virtual viewpoint image being displayed according to the user instruction, which is performed by the image storage unit 207, when a predetermined condition is satisfied.

[0088] Also, in the client device 1002 according to the present embodiment, the communication unit 1003 acquires a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time. Further, the video output unit 206 outputs, for display, the video represented by the plurality of virtual viewpoint images to an output device. Further, the image storage unit 207 can store, in the image storage unit 208, the virtual viewpoint image output by the video output unit 206 or the virtual viewpoint image displayed on the output device among the virtual viewpoint images constituting the virtual viewpoint video sent from the server device 1001. At this time, the image storage unit 207 can restrict the storage operation of the virtual viewpoint image being displayed according to the user instruction in accordance with the control by the storage control unit 209 when a predetermined condition is satisfied.

[0089] FIGS. 7(A) and (B) are flowcharts showing the processing flow of the image processing method performed by the image processing system 1000 according to an embodiment. By repeating the flowcharts shown in FIGS. 7(A) and (B), a virtual space video is generated and displayed. Hereinafter, as in the first embodiment, a configuration in which the storage of virtual viewpoint images is restricted when the volumetric video is stopped will be described. FIG. 7(A) shows the processing performed by the server device 1001. Further, FIG. 7(B) shows the processing performed by the client device 1002.

[0090] First, a flowchart of the processing performed by the server device 1001 shown in FIG. 7(A) will be described. In S1101, the communication unit 1003 of the server device 1001 acquires information indicating a user operation from the operation unit 1004. For example, the communication unit 1003 can receive information indicating a user operation to change the virtual viewpoint or a user operation related to the playback of a volumetric video. Based on the received information, the viewpoint specifying unit 204 can set the virtual viewpoint. Also, based on the received information, the time specifying unit 203 can perform time code control related to the playback of the volumetric video.

[0091] S301, S302, S303, and S304 are performed in the same manner as in the first embodiment. The determination in S306 is also performed in the same manner as in the first embodiment. That is, the storage control unit 209 can determine whether to limit the storage operation of the virtual viewpoint image according to a predetermined condition. When limiting the storage operation of the virtual viewpoint image, the process proceeds to S1102. Otherwise, the process proceeds to S1103. In this example, when it is determined that the volumetric video is stopped, the process proceeds to S1102. Otherwise, the process proceeds to S1103.

[0092] In S1102, the storage control unit 209 determines to limit the storage operation of the virtual viewpoint image. In this example, the storage control unit 209 determines to prohibit the storage of the virtual viewpoint image. In S1103, the storage control unit 209 determines not to limit the storage operation of the virtual viewpoint image. In this example, the storage control unit 209 determines to permit the storage of the virtual viewpoint image.

[0093] In S1104, the video generation unit 205 transmits the virtual viewpoint image generated in S304 to the client device 1002 via the communication unit 1003. Also, the storage control unit 209 transmits information (storage permission information) indicating whether to restrict the storage operation of the virtual viewpoint image to the client device 1002 via the communication unit 1003. For example, the storage control unit 209 may transmit the determination results in S1102 to S1103 to the client device 1002. The storage control unit 209 may also transmit information specifying a restriction method for storing the virtual viewpoint image to the client device 1002.

[0094] Next, a flowchart of the processing performed by the client device 1002 shown in FIG. 7(B) will be described.

[0095] In S1105, the operation unit 1004 receives a user operation. For example, the operation unit 1004 can receive user operations related to the playback of the virtual viewpoint and the volumetric video. Then, the operation unit 1004 transmits operation information indicating the received user operation to the server device 1001 via the communication unit 1003. The information thus transmitted is received in S1101 as described above.

[0096] In S1106, the communication unit 1003 of the client device 1002 acquires the information transmitted from the server device 1001 in S1104 as described above. That is, the communication unit 1003 can acquire the virtual viewpoint image generated by the server device 1001 and the storage permission information. The communication unit 1003 sends the acquired virtual viewpoint image to the video output unit 206. Also, the communication unit 1003 sends the storage permission information to the image storage unit 207.

[0097] The determination in S305 is performed in the same manner as in the first embodiment. If it is determined that there is an instruction to store the virtual viewpoint image, the process proceeds to S1107. If it is determined that there is no instruction to store the virtual viewpoint image, the process shown in FIG. 7(B) ends.

[0098] In S1107, the image storage unit 207 determines whether storage permission information indicating that the storage operation of the virtual viewpoint image is restricted is sent from the server device 1001. When the storage permission information indicating that the storage operation of the virtual viewpoint image is restricted is sent from the server device 1001, the process proceeds to S307. When such information is not sent, the process proceeds to S308. The process of S307 is performed in the same manner as in Embodiment 1. In this example, the virtual viewpoint image is not stored by the image storage unit 207. Also, in S308, the image storage unit 207 stores the virtual viewpoint image acquired in S1104 in the image storage unit 208.

[0099] In the above embodiment, the communication unit 1003 of the client device 1002 acquired the storage permission information together with the virtual viewpoint image in S1106. On the other hand, the image storage unit 207 may acquire the storage permission information by querying the server device 1001 in S1107. In this way, instead of sending the storage permission information together with the virtual viewpoint image in S1104, the server device 1001 may transmit this information in response to an inquiry from the client device 1002.

[0100] According to the above embodiment, the generation process of the virtual viewpoint image is performed by the server device 1001. Further, the server device 1001 determines whether to restrict the storage operation of the virtual viewpoint image. When the three-dimensional model of the subject is stationary, the user cannot store the virtual viewpoint image in the client device 1002. Therefore, it is possible to make it difficult to restore the three-dimensional shape of the subject based on the virtual viewpoint image.

[0101] Similar to Embodiment 1, the predetermined condition is not limited to the fact that the volumetric video is stopped. Similar to the above Modification 1, the storage control unit 209 may restrict the storage operation of the virtual viewpoint image by the image storage unit 207 according to the specified virtual viewpoint. Such processing can be realized by the storage control unit 209 determining whether to restrict the storage operation of the virtual viewpoint image in the same manner as in Modification 1 in S306.

[0102] Also, similar to the above-described Modification 2, the storage control unit 209 may limit the storage operation of the virtual viewpoint image by the image storage unit 207 according to the number of times the virtual viewpoint image is stored. In such an example, the image storage unit 207 can transmit the image recording information to the server device 1001 via the communication unit 1003 in S308. The image storage unit 207 can receive the image recording information via the communication unit 1003. The image storage unit 207 may store the received image recording information in an image recording information storage unit (not shown). Also, in S306, the storage control unit 209 can determine whether to limit the storage operation of the virtual viewpoint image in the same manner as in Modification 2. At this time, the storage control unit 209 can refer to the image recording information received from the image storage unit 207 and determine whether a predetermined condition regarding the number of times the virtual viewpoint image is stored is satisfied.

[0103] Also, similar to Embodiment 1, the conditions under which the storage of the virtual viewpoint image is restricted can be used in combination.

[0104] (Modification 4) The configuration of the image processing system 1000 is not limited to that shown in FIG. 6. For example, the connection between the server device 1001 and the client device 1002 is not limited to a one-to-one connection. For example, a plurality of client devices 1002 may be connected to the server device 1001.

[0105] Also, the server device 1001 may operate on a plurality of physical devices. In this case, the server device 1001 may be recognized as one logical device from the client device 1002. Also, the server device 1001 may be realized using a virtual service such as cloud technology. In this case, the server device 1001 may have a plurality of logical functional units that perform the above-described processing by software technology.

[0106] At this time, a plurality of client devices 1002 may view volumetric videos on a common virtual space. Also, the playback state of the volumetric video (e.g., playback, stop, or time code of the display target) may be common among the plurality of client devices 1002. For example, when an operation to stop the volumetric video is performed by one client device 1002, the volumetric video may also stop on other client devices 1002. Similarly, a plurality of client devices 1002 may view virtual viewpoint images from a commonly set virtual viewpoint. For example, when an operation to specify a virtual viewpoint is performed by one client device 1002, the virtual viewpoint image displayed on other client devices 1002 may change according to the specified virtual viewpoint.

[0107] Also, when the first user performs a save operation on the virtual viewpoint image, an operation to stop the volumetric video may be performed by another second user. In such a case, in S306, the save control unit 209 can determine that the volumetric video is stopped. Then, the save control unit 209 can determine not to permit the first user to save the virtual viewpoint image.

[0108] Also, when the parameters of the virtual viewpoint specified by at least one of the plurality of client devices 1002 satisfy the first condition, the save control unit 209 may control the time specifying unit 203 so as not to accept an instruction to stop the volumetric video.

[0109] The following describes the process when determining whether to limit the saving operation of the virtual viewpoint image when a plurality of client devices 1002 are connected to the server device 1001, in the same manner as in Modification Example 2. In this example, the virtual viewpoint image is saved according to the user instructions of each of the plurality of users. On the other hand, the saving control unit 209 can receive the image recording information via the communication unit 1003 as described above, and can store the received image recording information in an image recording information storage unit (not shown). Here, the saving control unit 209 can store the image recording information for all the client devices 1002. Also, the saving control unit 209 can store the image recording information in association with the users of the client devices 1002. For example, the saving control unit 209 can store the image recording information together with user information such as user IDs.

[0110] As described above, in S306, the saving control unit 209 can determine whether to limit the saving operation of the virtual viewpoint image according to the number of times the virtual viewpoint image is saved. Here, the number of times the virtual viewpoint image is saved may be the total number of times the virtual viewpoint image is saved according to the user instructions of each of the plurality of users. That is, in S306, the saving control unit 209 can determine whether to limit the saving operation of the virtual viewpoint image according to the total number of times the virtual viewpoint image is saved for all users. In other words, the saving control unit 209 can cross-check for all users whether the number of times the virtual viewpoint image is saved for a specific time code is equal to or greater than a predetermined number of times.

[0111] For example, assume that the number of times a virtual viewpoint image can be saved for the same time code is 10 times, and user A has already saved the virtual viewpoint image 5 times for a specific time code, and user B has already saved the virtual viewpoint image 5 times. At this time, even if user C performs an operation to save the virtual viewpoint image for the same time code, since the virtual viewpoint image has already been saved the predetermined number of times by user A and user B, user C cannot save the virtual viewpoint image. In another example, only up to 10 virtual viewpoint images can be saved within 15 minutes of real time. If user A and user B have saved a total of 10 virtual viewpoint images, user C will have to wait for 15 minutes until the virtual viewpoint image can be saved. Note that the time is not limited to 15 minutes and may be longer or shorter. Also, the number of times that can be saved is not limited to 10 times and may be less or more.

[0112] Furthermore, the predetermined conditions may include conditions for individual users. Also, the predetermined conditions may include conditions for individual users in addition to the conditions for all users. For example, in addition to setting the number of times that can be saved for all users to 10 or less, the number of times that can be saved for individual users may be set to 5 or less. In this way, the upper limit value of the number of saves for individual users may be different from the upper limit value of the number of saves for all users.

[0113] [Embodiment 3] Hereinafter, an image processing apparatus according to an embodiment will be described with reference to the drawings. In the following embodiment, the image processing apparatus 1600 generates and outputs a virtual viewpoint video represented by a plurality of virtual viewpoint images in the same manner as the image processing apparatus 100 according to Embodiment 1. The image processing apparatus 1600 according to the present embodiment restricts user operations on the video when a predetermined condition regarding the communication method of the video to the output device is satisfied. For example, the image processing apparatus 1600 can restrict user operations on the video when the video transmitted to the output device is not encrypted. First, a configuration for restricting the operation of stopping the volumetric video when the output video signal is not encrypted will be described below.

[0114] FIG. 8 is a schematic diagram showing a functional configuration example of the image processing apparatus 1600 according to the present embodiment. Note that the hardware configuration of the image processing apparatus 1600 may be the same as that of the image processing apparatus 100. The image processing apparatus 1600 includes a model acquisition unit 201, a spatial configuration unit 202, a time specifying unit 203, a viewpoint specifying unit 204, a video generation unit 205, a video output unit 206, an encryption unit 1601, and an operation restriction unit 1602.

[0115] The same reference numerals are given to the functional units similar to those in the first embodiment. For example, the video generation unit 205 can acquire a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time. Further, the video output unit 206 can output a video represented by the plurality of virtual viewpoint images for display by an output device. Hereinafter, the differences from the first embodiment will be described.

[0116] The encryption unit 1601 can encrypt the virtual viewpoint video generated by the video generation unit 205. The encryption unit 1601 can encrypt a video signal indicating the virtual viewpoint video according to settings by the user. The encryption method is not particularly limited, and for example, the HDCP method can be used. As the encryption method, a method that makes it impossible to record the video signal in a device that has received the video signal may be used. Also, as the encryption method, a method that makes it impossible to reproduce the original video signal based on the video signal recorded by a device that has received the video signal may be used.

[0117] The operation restriction unit 1602 restricts user operations on the video when a predetermined condition regarding the communication method of the video to the output device is satisfied. In this example, the operation restriction unit 1602 restricts user operations on the video according to whether the virtual viewpoint video output by the image processing device is encrypted or not. For example, the operation restriction unit 1602 can restrict user operations on the video in response to a determination that the virtual viewpoint video output by the image processing device is not encrypted. Also, the operation restriction unit 1602 can determine not to restrict user operations on the video in response to a determination that the virtual viewpoint video output by the image processing device is encrypted. In this example, when the virtual viewpoint video output by the image processing device is not encrypted, the operation restriction unit 1602 rejects an operation to stop the volumetric video. According to such a configuration, it becomes impossible to stop the volumetric video. However, the operation restriction unit 1602 may restrict user operations on the video according to whether any communication method that makes it difficult to use the output video is being used or not.

[0118] FIG. 9 is a flowchart showing a processing flow of an image processing method performed by the image processing device 1600 according to an embodiment. According to the flowchart shown in FIG. 9, a virtual space image is generated. Also, by repeating the flowchart shown in FIG. 9, a virtual space video is generated and output.

[0119] In S1701, the time specifying unit 203 determines whether it has received a stop instruction for the volumetric video from the user. If it has not received the volumetric video stop instruction, the process proceeds to S301. If it has received the stop instruction for the volumetric video, the process proceeds to S1702.

[0120] In S1702, the operation restriction unit 1602 determines whether the encryption unit 1601 is encrypting the video signal or not. If the video signal is encrypted, the process proceeds to S1703. If the video signal is not encrypted, the process proceeds to S1704.

[0121] In S1703, the operation restriction unit 1602 determines to stop the volumetric video according to the user instruction. That is, in subsequent S301, the time specifying unit 203 sets the time code to a fixed value. In subsequent S1701, the time specifying unit 203 can resume the playback of the volumetric video in response to determining that a playback instruction (or stop cancellation instruction) of the volumetric video has been received from the user. In this case, in subsequent S301, the time specifying unit 203 counts up the time code.

[0122] In S1704, the operation restriction unit 1602 determines not to stop the volumetric video according to the user instruction. That is, even though a stop instruction for the volumetric video has been received, in subsequent S301, the time specifying unit 203 counts up the time code. In this case, the operation restriction unit 1602 may notify the user that the user operation is restricted. For example, the operation restriction unit 1602 can notify the user who issued the stop instruction for the volumetric video that the volumetric video cannot be stopped. Specifically, the operation restriction unit 1602 can control the video output unit 206 to display a UI indicating that the stop cannot be performed on the output device that displays the virtual viewpoint video.

[0123] S301 to S304 are performed in the same manner as in the first embodiment.

[0124] Note that the method of restricting user operations on video when predetermined conditions regarding the method of communicating video to the output device are satisfied is not limited to the above method. For example, it may be possible to dynamically switch between encrypting / decrypting virtual viewpoint video. Specifically, the operation restriction unit 1602 may control the encryption unit 1601 to encrypt the virtual viewpoint video to be output in response to a stop instruction for volumetric video. According to such a configuration, encrypted virtual viewpoint video is output when the volumetric video is stopped. The operation restriction unit 1602 may control the encryption unit 1601 to decrypt the encryption of the virtual viewpoint video when restarting the playback of the volumetric video. Even with such a configuration, it is possible to restrict the stop operation of the volumetric video when the virtual viewpoint video is not encrypted.

[0125] Also, when a stop instruction for volumetric video is received while the virtual viewpoint image is not encrypted, the operation restriction unit 1602 can control the video generation unit 205 or the video output unit 206 to stop the output of the virtual viewpoint image. When rejecting the user operation in this way, the operation restriction unit 1602 may perform control to output a pre-prepared video to the output device. For example, in this case, the video output unit 206 can output a video different from the volumetric video, such as a black screen. Further, the video output unit 206 can notify the user who gave the stop instruction for the volumetric video that the volumetric video cannot be stopped. While the output of the volumetric video is stopped, the video output unit 206 can cause the output device to display a UI indicating that video output is not possible.

[0126] According to the above embodiment, when the volumetric video is stopped, an unencrypted virtual viewpoint video is not output. The user may attempt to output a virtual viewpoint video including virtual viewpoint images from various viewpoints of the three-dimensional model corresponding to the same time code by stopping the volumetric video and moving the virtual viewpoint. However, since the virtual viewpoint video output from the image processing apparatus 1600 is encrypted, it is difficult to restore the three-dimensional shape of the subject based on the output virtual viewpoint image.

[0127] (Modification Example 5) The method by which the operation restriction unit 1602 restricts user operations on the video is not limited to the above method. The operation restriction unit 1602 can restrict user operations according to predetermined conditions when predetermined conditions regarding the communication method of the video to the output device are satisfied. As the predetermined conditions regarding the user operations, for example, those similar to Modification Example 1 can be used. Specifically, the operation restriction unit 1602 can reject a user operation for designating a virtual viewpoint that satisfies the first condition in response to determining that the video transmitted to the output device is not encrypted. According to such a configuration, the virtual viewpoint video from the virtual viewpoint that satisfies the first condition is not output.

[0128] For example, the operation restriction unit 1602 can acquire information indicating whether the virtual viewpoint image is encrypted from the encryption unit 1601. Also, the operation restriction unit 1602 can acquire a user instruction for operating the virtual viewpoint. The operation restriction unit 1602 can determine whether to permit the operation of the virtual viewpoint according to this information. For example, when the virtual viewpoint image is not encrypted and the parameters of the virtual viewpoint according to the user instruction satisfy predetermined conditions, the operation restriction unit 1602 can control the viewpoint designation unit 204 so as not to accept the operation of the virtual viewpoint according to the user instruction.

[0129] FIG. 10 is a flowchart showing the processing flow of an image processing method performed by an image processing apparatus 1600 according to an embodiment in this modified example. S301 and S302 are performed in the same manner as in the first embodiment.

[0130] In S1901, the operation restriction unit 1602 determines whether a user instruction for operating the virtual viewpoint has been input. If a user instruction has been input, the process proceeds to S1702; if there is no operation, the process proceeds to S304.

[0131] In S1702, the operation restriction unit 1602 determines whether the encryption unit 1601 is encrypting the video signal. If the video signal is encrypted, the process proceeds to S1903. If the video signal is not encrypted, the process proceeds to S1902.

[0132] In S1902, the operation restriction unit 1602 determines whether the parameters of the virtual viewpoint satisfy a predetermined condition, in the same manner as in the first modified example. If the parameters of the virtual viewpoint satisfy the predetermined condition, the process proceeds to S1904. Otherwise, the process proceeds to S1903.

[0133] In S1903, the operation restriction unit 1602 permits the operation of the virtual viewpoint according to the user instruction. Then, the viewpoint designation unit 204 sets the virtual viewpoint according to the user instruction. In S1904, the operation restriction unit 1602 rejects the operation of the virtual viewpoint according to the user instruction. In this case, the viewpoint designation unit 204 does not change the virtual viewpoint.

[0134] S304 is performed in the same manner as in the first embodiment. That is, the video generation unit 205 generates a virtual viewpoint image according to the virtual viewpoint set in S1902 or the virtual viewpoint set previously.

[0135] According to the above embodiment, when the virtual viewpoint video is not encrypted, the output of the virtual viewpoint video from a virtual viewpoint according to a predetermined condition is restricted. For example, when the virtual viewpoint video is not encrypted, the user cannot set the virtual viewpoint outside the permitted range (e.g., the shooting permission area 701). Thus, the virtual viewpoint corresponding to the output virtual viewpoint image is limited. On the other hand, when the virtual viewpoint is outside the permitted range, since the virtual viewpoint video output from the image processing apparatus 1600 is encrypted, it is difficult to restore the three-dimensional shape of the subject based on the output virtual viewpoint image.

[0136] Note that, similar to the above embodiment, when a predetermined condition is satisfied in a state where the virtual viewpoint image is not encrypted, the operation restriction unit 1602 may control the video generation unit 205 or the video output unit 206 to stop the output of the virtual viewpoint image. For example, when the virtual viewpoint image is not encrypted and the parameters of the virtual viewpoint satisfy a predetermined condition, the operation restriction unit 1602 can stop the output of the virtual viewpoint image.

[0137] So far, the conditions under which the output of the virtual viewpoint video is restricted have been described. Similar to Modification 3, these conditions can be used in combination. Also, the mode of restriction by the operation restriction unit 1602 is not limited to the above. For example, similar to Modification 3, when the output of the virtual viewpoint image is restricted, the operation restriction unit 1602 may reduce the image quality of the virtual viewpoint image output from the video output unit 206.

[0138] [Embodiment 4] Hereinafter, an image processing system 2000 according to an embodiment will be described with reference to FIG. 11. FIG. 11 shows a functional configuration example of the image processing system 2000. Similar to Embodiment 2, the image processing system 2000 includes a server device 2001 and a client device 2002. Also, similar to Embodiment 2, on the one hand, the server device 2001 performs rendering processing of volumetric video, and a virtual viewpoint video is sent to the client device 2002 by communication. Note that the hardware configurations of the server device 2001 and the client device 2002 can be the same as the hardware configuration of the image processing device 100.

[0139] In the present embodiment, similar to Embodiment 3, when a predetermined condition regarding the video communication method between the client device 2002 and the output device is satisfied, the user operation on the video is restricted. First, hereinafter, a configuration for restricting the user operation to stop the volumetric video when the output video signal is not encrypted will be described.

[0140] The server device 2001 includes a model acquisition unit 201, a space configuration unit 202, a time specification unit 203, a viewpoint specification unit 204, a video generation unit 205, an operation restriction unit 1602, and a communication unit 1003. The client device 2002 includes a video output unit 206, a communication unit 1003, an operation unit 1004, and an encryption unit 1601. The same reference numerals are assigned to the functional units similar to those in Embodiments 2 and 3. Hereinafter, the differences from Embodiments 2 and 3 will be described.

[0141] The operation restriction unit 1602 acquires information indicating the video communication method between the client device 2002 and the output device. In the present embodiment, the operation restriction unit 1602 can acquire, via the communication unit 1003, information indicating whether the virtual viewpoint video is encrypted from the encryption unit 1601. Also, the operation restriction unit 1602 can control the time specification unit 203 so as to restrict the user operation to stop the volumetric video.

[0142] FIG. 12(A) and (B) are flowcharts showing the processing flow of the image processing method performed by the image processing system 2000 according to one embodiment. By repeating the flowcharts shown in FIGS. 12(A) and (B), a virtual space video is generated and output. FIG. 12(A) shows the processing performed by the server device 2001. FIG. 12(B) shows the processing performed by the client device 2002.

[0143] S1101 is performed in the same manner as in Embodiment 2. In the case of this embodiment, a stop instruction for the volumetric video from the user can be received in S1101. Also, S1701 to S1704 are performed in the same manner as in Embodiment 3. That is, when the virtual viewpoint video is encrypted, in S1703, the operation restriction unit 1602 determines to stop the volumetric video. Also, when the virtual viewpoint video is not encrypted, in S1704, the operation restriction unit 1602 determines not to stop the volumetric video.

[0144] S301 to S304 are performed in the same manner as in Embodiment 2. In S2102, the video generation unit 205 transmits the virtual viewpoint image generated in S304 to the client device 2002 via the communication unit 1003.

[0145] S1105 is performed in the same manner as in Embodiment 3. In S2103, the encryption unit 1601 transmits information (encryption state information) indicating whether the virtual viewpoint video is encrypted to the operation restriction unit 1602 via the communication unit 1003. In S2104, the communication unit 1003 of the client device 1002 acquires the virtual viewpoint image generated by the server device 1001. Thereafter, the video output unit 206 can output the virtual viewpoint image to the output device for display.

[0146] Also according to the above embodiment, in a state where the volumetric video is stopped, an unencrypted virtual viewpoint video is not output. For this reason, it is difficult to restore the three-dimensional shape of the subject based on the virtual viewpoint images from various viewpoints obtained by stopping the volumetric video and moving the virtual viewpoint.

[0147] (Modification Example 6) Similar to Embodiment 3, when a predetermined condition regarding the method of communicating the video to the output device is satisfied, the image processing system 2000 can restrict a user operation according to a predetermined condition for the video. Here, the predetermined condition is not limited to those described above. For example, similar to Modification Example 5, the predetermined condition may be a condition regarding the virtual viewpoint designated by the user operation.

[0148] FIG. 13 is a flowchart showing the processing flow of the image processing method performed by the server device 2001 in this modification example. Note that the client device 2002 can operate according to the flowchart of FIG. 12(B).

[0149] S1101, S301, and S302 are performed in the same manner as in FIG. 12(A). That is, in S1101, a user instruction for operating the virtual viewpoint can be received. Also, S1901, S1702, and S1902 to S1904 are performed in the same manner as in Modification Example 5. That is, when the virtual viewpoint video is encrypted, in S1903, the viewpoint designation unit 204 sets the virtual viewpoint according to the user instruction. Also, when the virtual viewpoint video is not encrypted and the virtual viewpoint does not satisfy a predetermined condition, in S1903, the viewpoint designation unit 204 sets the virtual viewpoint according to the user instruction. On the other hand, when the virtual viewpoint video is not encrypted and the virtual viewpoint satisfies a predetermined condition, in S1904, the viewpoint designation unit 204 does not accept an operation of the virtual viewpoint according to the user instruction.

[0150] Also according to the above modification example, when the virtual viewpoint video is not encrypted, the output of the virtual viewpoint video from the virtual viewpoint according to a predetermined condition is restricted. For this reason, it is difficult to restore the three-dimensional shape of the subject based on the output virtual viewpoint image.

[0151] Note that, similar to Modification Example 4, the configuration of the image processing system 2000 is not limited to that shown in FIG. 11. For example, the server device 2001 may operate on a plurality of physical devices or may operate on the cloud.

[0152] Also, a plurality of client devices 2002 may be connected to the server device 2001. Further, similar to Modification Example 4, a plurality of client devices 1002 may view a volumetric video in a common virtual space. At this time, the encryption settings may be different for each client device 2002. For example, the first user may encrypt the virtual viewpoint video, while the second user may not encrypt the virtual viewpoint video. In such a case, the operation restriction unit 1602 may determine whether to restrict the user operation on the video according to whether the virtual viewpoint videos output by all of the plurality of client devices 2002 are encrypted. For example, when the virtual viewpoint video output by at least one of the plurality of client devices 2002 is not encrypted, the operation restriction unit 1602 may determine not to stop the volumetric video in S1704. Note that the server device 2001 may reject the connection from the client device 2002 for which the setting to encrypt the output virtual viewpoint video is not made.

[0153] Note that, similar to Embodiment 3, it may be possible to dynamically switch between encryption / non-encryption of the virtual viewpoint video. For example, the operation restriction unit 1602 may notify the encryption unit 1601 to encrypt the output virtual viewpoint video in response to an instruction to stop the volumetric video. In this case, when starting the encryption of the virtual viewpoint video, the encryption unit 1601 can transmit information indicating that the virtual viewpoint video is being encrypted to the operation restriction unit 1602. The operation restriction unit 1602 that has received such information notification can control the time specifying unit 203 to stop the volumetric video. Also, the operation restriction unit 1602 may notify the encryption unit 1601 to cancel the encryption of the virtual viewpoint video when restarting the playback of the volumetric video.

[0154] (Other Embodiments) In Embodiments 3 and 4, a configuration was described in which a user operation to stop volumetric video is restricted when a predetermined condition regarding the method of communicating video to an output device is satisfied. In this way, the operation restriction unit 1602 can restrict the output of video in a state where the volumetric video is stopped according to a predetermined condition. Here, the predetermined condition is not limited to the case where the video signal is not encrypted. For example, the operation restriction unit 1602 may restrict the video output operation in a state where the volumetric video is stopped according to the virtual viewpoint. Specifically, similar to Modification 1, the operation restriction unit 1602 may restrict the video output operation in a state where the volumetric video is stopped when the parameters of the virtual viewpoint satisfy a predetermined condition. Furthermore, the operation restriction unit 1602 may control the viewpoint designation unit 204 so as to reject the operation of the virtual viewpoint when the volumetric video is stopped, so that virtual viewpoint videos from various viewpoints are not output in a state where the volumetric video is stopped.

[0155] Also, in Modification 1, when the parameters of the virtual viewpoint satisfy a predetermined condition, the time designation unit 203 may not accept a user instruction to stop the volumetric video. Similarly, in Modification 4, when the parameters of the virtual viewpoint operated by at least one of the plurality of client devices 1002 satisfy a predetermined condition, the time designation unit 203 may not accept a user instruction to stop the volumetric video. Also, in Modifications 2 and 4, when a predetermined condition regarding the number of times the virtual viewpoint is saved is satisfied, the time designation unit 203 may not accept a user instruction to stop the volumetric video. Also in these cases, when a predetermined condition is satisfied, the output of video in a state where the volumetric video is stopped is restricted.

[0156] Also, in Embodiments 1 and 2, a configuration for restricting the saving operation of virtual viewpoint images when predetermined conditions are satisfied was described. Thus, an image processing apparatus according to one embodiment includes a video generation unit 205 or a communication unit 1003 that acquires a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time. Further, the image processing apparatus includes a video output unit 206 or a communication unit 1003 that outputs a video represented by the plurality of virtual viewpoint images. Here, the video output unit 206 or the communication unit 1003 can output the video in a first mode in which a plurality of virtual viewpoint images of a three-dimensional model corresponding to different times are sequentially output so that a volumetric video is played. Also, the video output unit 206 or the communication unit 1003 can output the video in a second mode in which a plurality of virtual viewpoint images of a three-dimensional model corresponding to a specific time are sequentially output so that the volumetric video stops. And the image processing apparatus includes a saving control unit 209 or an operation restriction unit 1602 that permits video output in the first mode while restricting the video output operation or the saving operation of virtual viewpoint images in the second mode when predetermined conditions are satisfied.

[0157] According to such a configuration, when predetermined conditions are satisfied, it is possible to restrict the stop of the volumetric video or the saving of virtual viewpoint images. For this reason, it can be made difficult to restore the three-dimensional shape of the subject based on the stopped volumetric video or the saved virtual viewpoint images. For example, it can be made difficult to restore the three-dimensional shape of the subject based on virtual viewpoint images from a plurality of directions output during the stop of the volumetric video. Also, it can be made difficult to restore the three-dimensional shape of the subject based on the saved virtual viewpoint images from a plurality of directions. On the other hand, even when predetermined conditions are satisfied, it is possible to permit the playback of the volumetric video. For this reason, it is possible to improve the user experience when viewing the volumetric video.

[0158] The present disclosure can also be realized by supplying a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that implements one or more functions.

[0159] The disclosure herein includes the following image processing apparatus, image processing method, and program. (Item 1) An acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; An output means for outputting, for display, a video represented by the plurality of virtual viewpoint images; A control means for restricting a saving operation of a displayed virtual viewpoint image in accordance with a user instruction when a predetermined condition is satisfied; An image processing apparatus, comprising the above. (Item 2) The image processing apparatus according to Item 1, wherein the control means prohibits saving of the virtual viewpoint image in response to a determination that the predetermined condition is satisfied. (Item 3) The image processing apparatus according to Item 1, wherein the control means reduces the quality of the virtual viewpoint image to be saved in response to a determination that the predetermined condition is satisfied. (Item 4) The image processing apparatus according to Item 1 or 3, wherein the control means performs control to reduce the virtual viewpoint image to be saved in response to a determination that the predetermined condition is satisfied. (Item 5) The image processing apparatus according to Item 1, 3, or 4, wherein the control means controls the resolution of the virtual viewpoint image to be saved such that a portion corresponding to one component of the three-dimensional model becomes smaller than one pixel on the virtual viewpoint image in response to a determination that the predetermined condition is satisfied. (Item 6) The output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to the specified time, The image processing apparatus according to any one of items 1 to 5, wherein the predetermined condition includes that the same time is continuously specified. (Item 7) The image processing apparatus according to item 6, wherein the control means performs control to continuously update the specified time in response to a user instruction to save the virtual viewpoint image. (Item 8) The image processing apparatus according to any one of items 1 to 5, wherein the predetermined condition includes satisfying a first condition regarding the virtual viewpoint. (Item 9) The output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to the specified time, The image processing apparatus according to item 8, wherein the control means prohibits the same time from being continuously specified in response to a determination that the first condition is satisfied. (Item 10) The image processing apparatus according to any one of items 1 to 5, wherein the predetermined condition includes satisfying a second condition regarding the number of times the virtual viewpoint image is saved. (Item 11) The image processing apparatus according to item 10, wherein the second condition includes a condition regarding the number of times the virtual viewpoint image of the three-dimensional model is saved at a specific time, or a condition regarding the number of times the virtual viewpoint image is saved within a predetermined period of real time. (Item 12) The virtual viewpoint images are saved according to user instructions by each of a plurality of users, The image processing apparatus according to item 10 or 11, wherein the number of times the virtual viewpoint image is saved is the total number of times the virtual viewpoint image is saved according to the user instructions by each of the plurality of users. (Item 13) The image processing apparatus according to any one of items 1 to 12, wherein the control means notifies the user that a restriction has been imposed on the saving operation of the virtual viewpoint image. (Item 14) An acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; An output means for outputting, for display by an output device, a video represented by the plurality of virtual viewpoint images; A control means for restricting a user operation on the video when a predetermined condition regarding a communication method of the video to the output device is satisfied; An image processing apparatus, characterized by comprising: (Item 15) The image processing apparatus according to item 14, wherein the control means restricts a user operation on the video in response to a determination that the video transmitted to the output device is not encrypted. (Item 16) The output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to a specified time, The image processing apparatus according to any one of items 14 to 15, wherein the control means controls to reject a user operation of continuously specifying the same time in response to a determination that the video transmitted to the output device is not encrypted. (Item 17) The image processing apparatus according to any one of items 14 to 16, wherein the control means controls to reject a user operation of specifying a virtual viewpoint that satisfies a first condition in response to a determination that the video transmitted to the output device is not encrypted. (Item 18) The image processing apparatus according to any one of items 16 to 17, wherein the control means controls to output a pre-prepared video to the output device when rejecting the user operation. (Item 19) The image processing apparatus according to any one of items 14 to 18, wherein the control means notifies the user that the user operation is restricted. (Item 20) time specifying means for specifying a time, and viewpoint specifying means for specifying a virtual viewpoint, and the acquisition means generates, as the virtual viewpoint image, an image of the three-dimensional model corresponding to the specified time from the specified virtual viewpoint, the image processing apparatus according to any one of items 1 to 19. (Item 21) acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time, output means for outputting a video represented by the plurality of virtual viewpoint images, control means for permitting video output in a first mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to different times are sequentially output when a predetermined condition is satisfied, while restricting video output operation in a second mode in which a plurality of virtual viewpoint images of the three-dimensional model corresponding to a specific time are sequentially output or a storage operation of the virtual viewpoint image, An image processing apparatus, comprising: (Item 22) An image processing method performed by an image processing apparatus, a step of acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time, a step of outputting, for display, a video represented by the plurality of virtual viewpoint images, a step of restricting, when a predetermined condition is satisfied, a storage operation of the displayed virtual viewpoint image according to a user instruction, An image processing method, comprising: (Item 23) An image processing method performed by an image processing apparatus, a step of acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time, Outputting the video represented by the plurality of virtual viewpoint images for display by an output device; Restricting a user operation on the video when a predetermined condition regarding a communication method of the video to the output device is satisfied; An image processing method, characterized by including the above. (Item 24) An image processing method performed by an image processing apparatus, acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; outputting the video represented by the plurality of virtual viewpoint images; permitting video output in a first mode of sequentially outputting a plurality of virtual viewpoint images of the three-dimensional model corresponding to different times when a predetermined condition is satisfied, while restricting a video output operation in a second mode of sequentially outputting a plurality of virtual viewpoint images of the three-dimensional model corresponding to a specific time, or an operation of saving a virtual viewpoint image; An image processing method, characterized by including the above. (Item 25) A program for causing a computer to function as the image processing apparatus according to any one of Items 1 to 21.

[0160] The present disclosure is not limited to the above embodiments, and various changes and modifications can be made without departing from its spirit and scope. Therefore, claims are attached to disclose its scope.

Description of Reference Numerals

[0161] 201: Model acquisition unit, 202: Spatial configuration unit, 203: Time specification unit, 204: Viewpoint specification unit, 205: Video generation unit, 206: Video output unit, 207: Image storage unit, 208: Image storage section, 209: Storage control unit

Claims

Claim 1: An acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting, for display by an output device, a video represented by the plurality of virtual viewpoint images; a control means for restricting a user operation on the video when a predetermined condition regarding a communication method of the video to the output device is satisfied; An image processing apparatus comprising the control means for restricting a user operation on the video in response to determining that the video transmitted to the output device is not encrypted. Claim 2: An acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting, for display by an output device, a video represented by the plurality of virtual viewpoint images; a control means for restricting a user operation on the video when a predetermined condition regarding a communication method of the video to the output device is satisfied; Comprising: the output means sequentially outputs virtual viewpoint images of the three-dimensional model corresponding to a designated time; the control means controls to reject a user operation of continuously designating the same time in response to determining that the video transmitted to the output device is not encrypted. An image processing apparatus characterized by this. Claim 3: An acquisition means for acquiring a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; an output means for outputting, for display by an output device, a video represented by the plurality of virtual viewpoint images; a control means for restricting a user operation on the video when a predetermined condition regarding a communication method of the video to the output device is satisfied; Comprising: the control means controls to reject a user operation of designating the virtual viewpoint that satisfies the first condition in response to determining that the video transmitted to the output device is not encrypted. An image processing apparatus characterized by this. Claim 4 The image processing apparatus according to claim 2, wherein the control means controls to output a pre-prepared video to the output device when rejecting the user operation. Claim 5 The image processing apparatus according to claim 1, wherein the control means notifies the user that the user operation is restricted. Claim 6 An image processing method performed by an image processing apparatus, obtaining a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; outputting, for display by an output device, a video represented by the plurality of virtual viewpoint images; restricting a user operation on the video when a predetermined condition regarding a communication method of the video to the output device is satisfied; including, and characterized by restricting a user operation on the video in response to a determination that the video transmitted to the output device is not encrypted.

7. An image processing method performed by an image processing apparatus, obtaining a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; outputting, for display by an output device, a video represented by the plurality of virtual viewpoint images; restricting a user operation on the video when a predetermined condition regarding a communication method of the video to the output device is satisfied; including, in the outputting step, sequentially outputting virtual viewpoint images of the three-dimensional model corresponding to a specified time, and performing control to reject a user operation of continuously specifying the same time in response to a determination that the video transmitted to the output device is not encrypted.

8. An image processing method performed by an image processing apparatus, obtaining a plurality of virtual viewpoint images, each of which is an image from a specifiable virtual viewpoint of a three-dimensional model corresponding to a certain time; outputting, for display by an output device, a video represented by the plurality of virtual viewpoint images; restricting a user operation on the video when a predetermined condition regarding a communication method of the video to the output device is satisfied; including, and performing control to reject a user operation of specifying the virtual viewpoint that satisfies a first condition in response to a determination that the video transmitted to the output device is not encrypted.

9. A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 5.

Images