Image generation apparatus, image generation method, and program

Abstract

An image generation device according to an aspect of the present technology generates multi-viewpoint image data for displaying a multi-viewpoint image at a predetermined frame rate, the image generation device comprising a first generation unit, a generation control unit, and a second generation unit. The first generation unit can generate a plurality of viewpoint images corresponding to a plurality of viewpoint positions. For each frame, the generation control unit sets one or more target viewpoint positions which are parts among the plurality of viewpoint positions, and causes the first generation unit to generate one or more target viewpoint images corresponding to the one or more set viewpoint positions. The second generation unit generates the multi-viewpoint image data by using the one or more target viewpoint images which are generated for each frame.

Description

Technical Field

[0001] The present technology relates to an image generation apparatus, an image generation method, and a program applicable to the display of multi-viewpoint images.

Background Art

[0002] Patent Document 1 discloses a technique aimed at enabling the background for generating a virtual viewpoint image to be updated accurately and with a low processing load. Specifically, in order to reproduce the background in a three-dimensional space, the background shape is divided into partial regions. Then, the input image captured by the camera is divided for each of the partial regions. The input image divided for each partial region is compared with the input image of the corresponding partial region in the previous frame, and the importance level is determined for each partial region. Based on the determined importance level, it is determined whether to update the input image for each partial region (paragraphs

[0012]

[0020] ~[0023 etc. of the specification of Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Thus, there is a need for a technique that enables the display of multi-viewpoint images with a low processing load.

[0005] In view of the above circumstances, an object of the present technology is to provide an image generation apparatus, an image generation method, and a program capable of reducing the processing load involved in generating a viewpoint image.

Means for Solving the Problems

[0006] To achieve the above object, an image generation device according to one embodiment of the present technology is an image generation device that generates multi-viewpoint image data for displaying a multi-viewpoint image at a predetermined frame rate, and includes a first generation unit, a generation control unit, and a second generation unit. The first generation unit is capable of generating a plurality of viewpoint images corresponding to a plurality of viewpoint positions. For each frame, the generation control unit sets one or more target viewpoint positions that are part of the plurality of viewpoint positions, and causes the first generation unit to generate one or more target viewpoint images corresponding to the set one or more target viewpoint positions. The second generation unit generates the multi-viewpoint image data using the one or more target viewpoint images generated for each frame.

[0007] In this image generation device, for each frame, one or more target viewpoint images corresponding to one or more target viewpoint positions that are part of the plurality of viewpoint positions are generated. Therefore, for each frame, one or more target viewpoint images that are part of the plurality of viewpoint images are generated. As a result, it is possible to reduce the processing load associated with generating the viewpoint images.

[0008] The second generation unit may generate the multi-viewpoint image data for the predetermined frame using the one or more target viewpoint images generated in the predetermined frame and the one or more target viewpoint images generated in a frame earlier than the predetermined frame.

[0009] The generation control unit may cause one or more first target viewpoint images corresponding to one or more first target viewpoint positions to be generated in a first frame, and cause one or more second target viewpoint images corresponding to one or more second target viewpoint positions that are different from any of the one or more first target viewpoint positions to be generated in a second frame consecutive to the first frame.

[0010] The second generation unit may generate the multi-viewpoint image data for the second frame using the one or more first target viewpoint images and the one or more second target viewpoint images.

[0011] The generation control unit sets the number of update frames, divides the plurality of viewpoint positions into a plurality of target viewpoint position groups having the same number as the number of update frames and with no overlap of the target viewpoint positions with each other, and assigns each of the plurality of target viewpoint position groups to each of the consecutive frames of the number of update frames, and may generate a target viewpoint image group corresponding to the assigned target viewpoint position group in each of the plurality of frames.

[0012] The generation control unit sets the number of update frames to 2, divides the plurality of viewpoint positions into a first target viewpoint position group and a second target viewpoint position group with no overlap of the target viewpoint positions with each other, assigns the first target viewpoint position group and the second target viewpoint position group to two consecutive frames, and generates a first target viewpoint image group corresponding to the assigned first target viewpoint position group and a second target viewpoint image group corresponding to the assigned second target viewpoint position group in each of the two frames.

[0013] The generation control unit may be able to change the number of update frames.

[0014] The generation control unit may change the number of update frames based on the movement of the object to be displayed or the mode set for the multi-viewpoint image display.

[0015] The generation control unit may set the one or more target viewpoint positions based on the interpupillary distance for each frame and generate the one or more target viewpoint images.

[0016] The generation control unit may divide the plurality of viewpoint positions into the plurality of target viewpoint position groups based on the interpupillary distance.

[0017] The second generation unit uses the one or more target viewpoint images generated in a predetermined frame and the multi-viewpoint image data generated in a frame earlier than the predetermined frame. The multi-view image data of the predetermined frame may be generated.

[0018] The first generation unit may generate a virtual image as the viewpoint image.

[0019] The first generation unit may acquire, as the plurality of viewpoint images, a plurality of captured images from a plurality of imaging devices arranged at the plurality of viewpoint positions. In this case, the generation control unit may cause the first generation unit to output, for each frame, a captured image corresponding to the one or more target viewpoint images and discard other captured images.

[0020] The generation control unit may set the one or more target viewpoint positions based on position information of a user for each frame, and generate the one or more target viewpoint images.

[0021] The second generating unit may generate the multi-viewpoint image data as data for multi-viewpoint display on a multi-viewpoint display device.

[0022] The multi-viewpoint display device may include a plurality of projectors. In this case, the second generator may generate, as the multi-viewpoint image data, a plurality of corresponding multi-viewpoint image data corresponding to each of the plurality of projectors.

[0023] The multi-viewpoint display device may include a multi-viewpoint display, and the second generator may generate the multi-viewpoint image data corresponding to the multi-viewpoint display.

[0024] An image generation method according to one embodiment of the present technology is an image generation method that is executed by a computer system and generates multi-viewpoint image data for displaying a multi-viewpoint image at a predetermined frame rate, and includes setting, for each frame, one or more target viewpoint positions that are part of a plurality of viewpoint positions, and generating one or more target viewpoint images that correspond to the one or more set target viewpoint positions. The one or more target viewpoint images generated for each frame are used to generate the multi-viewpoint image data.

[0025] A program according to one aspect of the present technology causes a computer system to execute the image generation method.

Brief Description of Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Embodiments for Carrying Out the Invention

[0027] Hereinafter, embodiments according to the present technology will be described with reference to the drawings.

[0028] [Basic Configuration of Image Display System] FIG. 1 is a schematic diagram showing a basic configuration example of an image display system according to an embodiment of the present technology. FIG. 2 is a chart diagram showing a basic operation example of the image display system.

[0029] As shown in FIG. 1, the image display system 100 includes a multi-viewpoint display device 5 and an image generation device 6. The multi-viewpoint display device 5 and the image generation device 6 are communicably connected via wire or wirelessly. The connection form between each device is not limited, and for example, wireless LAN communication such as WiFi or short-range wireless communication such as Bluetooth (registered trademark) can be used.

[0030] The multi-viewpoint display device 5 can display a multi-viewpoint image. The multi-viewpoint image is an image capable of displaying images 3 corresponding to each of a plurality of viewpoint positions 8. A user (observer) can observe different images by changing the position to observe (i.e., the viewpoint position 8). For example, as shown in FIG. 1, an image 3b when the character 2 is viewed from the front is displayed corresponding to the viewpoint position 8b which is the position in front of the multi-viewpoint display device 5. An image 3a when the character 2 is viewed from the left side and wrapped around is displayed corresponding to the viewpoint position 8a which is the position moved to the left from the front viewpoint position 8b with respect to the multi-viewpoint display device 5. For the multi-viewpoint display device 5, corresponding to the viewpoint position 8c which is a position moved to the right from the frontal viewpoint position 8b, an image 3c when the character 2 is viewed as it wraps around from the right side is displayed. Of course, it is not limited to such multi-viewpoint images, and arbitrary images may be displayed for each viewpoint position 8. In this image display system 100, the multi-viewpoint display device 5 displays a multi-viewpoint image based on multi-viewpoint image data generated by the image generation device 6. A specific configuration example of the multi-viewpoint display device 5 will be described later.

[0031] The image generation device 6 generates multi-viewpoint image data for displaying a multi-viewpoint image at a predetermined frame rate. In the present disclosure, an image includes both a still image and a moving image (video). Also, generating multi-viewpoint image data at a predetermined frame rate corresponds to generating image data of a multi-viewpoint image displayed at a predetermined frame rate. That is, the multi-viewpoint image data includes the image data of the multi-viewpoint image displayed in each frame. In the following description, the image data of the multi-viewpoint image displayed in each frame is referred to as the multi-viewpoint image data of each frame. Also, in the present disclosure, generating an image at a predetermined frame rate is not necessarily limited to generating a moving image (video). Even when an image with a relatively high frame rate is generated and an image display is executed such that a still image appears to be frame-by-frame to the viewer, this technology is applicable. Of course, such an image display can also be regarded as a display of a moving image (video).

[0032] The image generation device 6 has hardware necessary for the configuration of a computer, such as a processor such as a CPU, GPU, DSP, a memory such as a ROM and a RAM, and a storage device such as an HDD. Of course, hardware such as an FPGA or an ASIC may be used (see FIG. 17). For example, by the processor loading and executing a program according to the present technology pre-recorded in a ROM or the like into a RAM, an image generation method according to the present technology is executed. For example, the image generating device 6 can be realized by any computer such as a PC (Personal Computer). Of course, hardware such as FPGA and ASIC may also be used. In this embodiment, the processor executes a predetermined program to configure functional blocks including a first generating unit 10, a generation control unit 11, and a second generating unit 12. Of course, dedicated hardware such as an IC (integrated circuit) may be used to realize the functional blocks. The program is installed in the image generation device 6 via, for example, various recording media. Alternatively, the program may be installed via the Internet or the like. The type of recording medium on which the program is recorded is not limited, and any computer-readable recording medium may be used. For example, any computer-readable non-transitory storage medium may be used.

[0033] The first generating unit 10 is capable of generating a plurality of viewpoint images 13 corresponding to a plurality of viewpoint positions 8 . As shown in FIG. 1, a plurality of viewpoint images 13a to 13c correspond to images 3a to 3c displayed by the multi-viewpoint display device 5 and observed from each viewpoint position 8. The images 3a to 3c observed by the user from each viewpoint position 8 are also called viewpoint images. In the present disclosure, for ease of understanding, images (image data) generated mainly by the first generator 10 will be described as viewpoint images. In this embodiment, the viewpoint image 13 is generated by CG (computer graphics). Specifically, a virtual camera 14 is placed around an object to be displayed (character 2 shown in FIG. 1) so as to surround the object. Then, the virtual images captured by each virtual camera 14 are generated as viewpoint images 13. This makes it possible to obtain images of an object viewed from different angles. Note that the position where the virtual camera 14 is arranged is set corresponding to the viewpoint position 8 defined for the multi-viewpoint display device 5. Conversely, based on the position where the virtual camera 14 is arranged, the viewpoint position 8 from which the user can observe the object at different angles is defined. The generation of the viewpoint image 13 by the first generation unit 10 can also be said to be the rendering of the viewpoint image 13. Also, the viewpoint image 13 can be called a rendering image. Of course, the number of viewpoint positions 8 is not limited and can be set arbitrarily. By setting many viewpoint positions 8, it becomes possible to observe the character 2 from various angles, and it becomes possible to provide a high-quality viewing experience.

[0034] The generation control unit 11 controls the generation of a plurality of viewpoint images 13 by the first generation unit 10. In the present embodiment, the generation control unit 11 sets one or more target viewpoint positions that are part of a plurality of viewpoint positions 8 for each frame. Also, the generation control unit 11 causes the first generation unit 10 to generate one or more target viewpoint images corresponding to the set one or more target viewpoint positions. That is, the generation control unit 11 determines which viewpoint image 13 to generate for each frame. Based on the determination, the viewpoint image 13 generated in each frame becomes the target viewpoint image.

[0035] In the example shown in FIG. 2, in frames (♯m) and (♯m + 2), the viewpoint positions 8a and 8c that are part of the three viewpoint positions 8a to 8c are set as the target viewpoint positions for generating the viewpoint image 13. Then, the viewpoint images 13a and 13c corresponding to the viewpoint positions 8a and 8c are generated as the target viewpoint images. In frames (♯m + 1) and (♯m + 3), the viewpoint position 8b that is part of the three viewpoint positions 8a to 8c is set as the target viewpoint position for generating the viewpoint image 13. Then, the viewpoint image 13b corresponding to the viewpoint position 8b is generated as the target viewpoint image. Thus, in this embodiment, for each frame, the viewpoint images 13 corresponding to all viewpoint positions 8 are not generated, but one or more target viewpoint images corresponding to one or more target viewpoint positions are generated. That is, for each frame, one or more target viewpoint images are generated instead of all the viewpoint images 13. For each frame, which viewpoint position 8 is set as one or more target viewpoint positions is not limited and may be arbitrarily set.

[0036] The second generation unit 12 generates multi-viewpoint image data using one or more target viewpoint images generated for each frame. The second generation unit 12 generates multi-viewpoint image data for each frame. The multi-viewpoint image data is generated as data for multi-viewpoint display of the multi-viewpoint display device 5. Therefore, the multi-viewpoint image data is generated in accordance with the configuration of the multi-viewpoint display device 5 and the multi-viewpoint display method. For example, in accordance with the configuration of the multi-viewpoint display device 5 and the multi-viewpoint display method, a plurality of viewpoint images 13 generated by the first generation unit 10 are appropriately converted to generate multi-viewpoint image data. In this embodiment, the second generation unit 12 generates multi-viewpoint image data using one or more target viewpoint images generated for each frame. That is, one or more target viewpoint images generated for each frame are used to generate multi-viewpoint image data for each frame. Note that in the present disclosure, the processing using the image data is not limited to the processing using only the image data. It includes at least any processing using the image data.

[0037] [Multi-viewpoint image display] With reference to FIGS. 3 to 5, a specific example of the multi-viewpoint display device 5 will be described. The multi-viewpoint display device 5 shown in FIG. 3 includes a plurality of projectors 16 and a light control element 17. Each of the plurality of projectors 16 can project an image and is configured as a projector array. In the multi-viewpoint display device 5 illustrated in FIG. 3, five projectors 16 and light control elements 17 are capable of displaying images 3a to 3c corresponding to three viewpoint positions 8a to 8c, respectively. The specific configuration of the projector 16 is not limited, and any configuration may be adopted.

[0038] As the light control element 17, for example, a transmission type anisotropic diffusion screen is used. A transmissive anisotropic diffusion screen has anisotropic diffusion characteristics with different diffusion degrees in the horizontal and vertical directions, for example, the diffusion degree in the horizontal direction is set smaller than that in the vertical direction, and the screen is configured to have narrow diffusion characteristics in the horizontal direction. By disposing the anisotropic diffusion screen, it becomes possible to display the image 3 with an appropriate width corresponding to each viewpoint position 8. The specific configuration of the anisotropic diffusion screen is not limited, and for example, a lens diffuser plate configured with a microlens array or the like, or a transmissive HOE (Holographic Optical Element) or the like can be used as the anisotropic diffusion screen.

[0039] As illustrated in FIG. 4, in this embodiment, viewpoint images 13a to 13b corresponding to three viewpoint positions 8a to 8c, respectively, are divided into a plurality of strip-shaped regions along the horizontal direction of the image (hereinafter, the images of the divided regions will be referred to as strip images 18). The divided strip images 18 are rearranged as appropriate to generate image data of the projection image 19 to be projected from each projector 16. Each projector 16 projects the projection image 19 based on the image data. Therefore, a certain projector 16 projects a projection image 19 including respective strip images 18 of mutually different viewpoint images 13. Also, another projector 16 may project a projection image 19 including only one strip image 18. At each viewpoint position 8, the user views an image 3 in which projection images 19 projected from different projectors 16 are combined with strip images 18 corresponding to the viewpoint position 8. This achieves multi-viewpoint image display. Hereinafter, image data and an image displayed based on the image data may be described using the same drawing. For example, the projected image 19 shown in FIG. 4 may be described as image data of the projected image 19.

[0040] For the multi-viewpoint display device 5 illustrated in FIG. 3, the second generation unit 12 rearranges the strip images 18 of the respective viewpoint images 13, and generates multi-viewpoint image data. Specifically, the strip images 18 are rearranged to generate image data of the projection images 19 corresponding to each of the plurality of projectors 16. The plurality of image data corresponding to each of the plurality of projectors 16 are generated as multi-viewpoint image data. The plurality of image data corresponding to each of the plurality of projectors 16 corresponds to the plurality of corresponding multi-viewpoint image data corresponding to each of the plurality of projectors 16. The rearrangement of the strip images 18 can be realized using well-known techniques based on, for example, the number of projectors 16 and the number of viewpoint positions 8.

[0041] The multi-viewpoint display device 5 shown in FIG. The multi-viewpoint display 21 makes it possible to simultaneously display images 3 corresponding to a plurality of viewpoint positions 8 toward each of the viewpoint positions 8. The multi-viewpoint display 21 can be configured using any one of a lenticular lens system, a lens array system, and a parallax barrier system, for example, but is not limited to these systems.

[0042] The multi-viewpoint display 21 shown in FIG. 5 includes a flat display panel 22 and a lenticular lens 23. The flat display panel 22 has a plurality of pixels arranged in the horizontal and vertical directions, and the lenticular lenses 23 are arranged along the vertical direction. In the example shown in FIG. 5, viewpoint images 13a to 13c corresponding to three viewpoint positions 8a to 8c, respectively, are divided into a plurality of strip images 18 along the horizontal direction of the images. The divided strip images 18 are rearranged as appropriate to generate image data for a display image to be displayed on the flat display panel 22. The display image is displayed on the flat display panel 22 based on the image data. 5, for example, the pixel area facing the four convex portions 23a of the lenticular lens 23 is divided into three areas 24a to 24c along the horizontal direction. Then, the strip images 18 of the three viewpoint images 13 are allocated to the three divided areas. Of course, the rearrangement is not limited to this. At each viewpoint position 8, the user views a strip image 18 in which the direction of light rays is controlled by the lenticular lens 23 toward each viewpoint position 8. This realizes a multi-viewpoint image display. 5, second generation unit 12 rearranges strip images 18 of each viewpoint image 13 to generate multi-viewpoint image data. The multi-viewpoint image data becomes multi-viewpoint image data corresponding to multi-viewpoint display 21. The rearrangement of the strip images 18 can be realized using well-known technology based on the configuration of the multi-viewpoint display 21, for example. Compared to a configuration using a projector array, a configuration using multi-viewpoint display 21 allows the entire device to be designed more compactly.

[0043] [Displaying 3D images] FIG. 6 is a diagram for explaining the display of a stereoscopic image. For each viewpoint position 8, the width (range) within which the same image 3 (same viewpoint image) can be observed is defined as the viewpoint width. As shown in Figure 6A, when the viewpoint width is larger than the inter-pupillary distance (IPD), the user will often view the same image with both eyes, and therefore will view a flat image (2D image). As shown in FIG. 6B, when the viewing width is smaller than the interpupillary distance (IPD), it becomes possible to display a disparity image as the image 3 (viewpoint image) corresponding to the viewpoint position 8. Therefore, the user can observe different disparity images (right-eye image, left-eye image) with both eyes, and the observation of a stereoscopic image (3D image) is realized. This technology can be applied to both the display of a planar image and the display of a stereoscopic image.

[0044] FIG. 7 is a schematic diagram showing a functional configuration example of the image generation device 6. The image generation device 6 includes a plurality of viewpoint image generation units 26, a plurality of viewpoint image storage units 27, a viewpoint image generation control unit 28, a display image generation unit 29, and a display image output unit 30. The plurality of viewpoint image generation units 26, the viewpoint image generation control unit 28, the display image generation unit 29, and the display image output unit 30 are configured, for example, by a processor executing a predetermined program. Of course, dedicated hardware such as an IC (integrated circuit) may be used to realize these functional blocks. The plurality of viewpoint image storage units 27 are realized, for example, by an HDD, a flash memory, or other solid-state memories. It is not limited to this, and any storage device may be used.

[0045] The plurality of viewpoint image generation units 26 are configured corresponding to a plurality of viewpoint positions 8. That is, for one viewpoint position 8, one viewpoint image generation unit 26 is configured. Therefore, the plurality of viewpoint image generation units 26 are configured in the number of the viewpoint positions 8. In this embodiment, it is assumed that n viewpoint positions 8 from one viewpoint to n viewpoints are defined as the viewpoint positions 8. Therefore, n viewpoint image generation units 26 are configured. The n viewpoint image generation units 26 generate viewpoint images 13 corresponding to each of the n viewpoint positions 8.

[0046] The plurality of viewpoint image storage units 27 are configured corresponding to the plurality of viewpoint image generation units 26. That is, for one viewpoint image generation unit 26, one viewpoint image storage unit 27 is configured. Therefore, n viewpoint image storage units 27 are configured, which is the same number as the number of viewpoint positions 8. As surrounded by the dashed line in FIG. 7, it can also be said that a pair of a viewpoint image generation unit 26 and a viewpoint image storage unit 27 is configured for one viewpoint position 8. The n viewpoint image storage units 27 store the viewpoint images 13 generated by the corresponding paired viewpoint image generation units 26.

[0047] The viewpoint image generation control unit 28 controls the generation of the viewpoint images 13 by each viewpoint image generation unit 26 for each frame. Specifically, for each frame, the viewpoint image generation control unit 28 sets one or more target viewpoint positions that are part of the plurality of viewpoint positions 8. Further, the viewpoint image generation control unit 28 causes the viewpoint image generation unit 26 to generate one or more target viewpoint images corresponding to the set one or more target viewpoint positions. Therefore, the viewpoint image generation control unit 28 determines which viewpoint image 13 is to be generated for each frame. Based on this determination, the viewpoint images 13 generated in each frame become the target viewpoint images.

[0048] FIG. 8 is a schematic diagram for explaining an example of setting one or more target viewpoint positions. The viewpoint image generation control unit 28 sets the update frame number. The update frame number is the number of frames required for updating all the viewpoint images 13. Conversely, the update frame number is the number of frames allocated for updating all the viewpoint images 13. In the present embodiment, all the viewpoint images 13 are updated over consecutive frames of the update frame number. In FIG. 8, the update frame number = l is set. And all the viewpoint images 13 are updated by l consecutive frames (♯m + 1) to (♯m + l).

[0049] The viewpoint image generation control unit 28 divides the plurality of viewpoint positions 8 into a plurality of target viewpoint position groups, the number of which is the same as the number of update frames, and in which the target viewpoint positions do not overlap with each other. In Fig. 8, l target viewpoint position groups (#1) to (#l) are set. The target viewpoint positions included in each group of target viewpoint positions are set so as not to overlap with each other. Therefore, the number of update frames is equal to or less than the total number of viewpoint positions 8.

[0050] 8, a plurality of target viewpoint position groups (#1) to (#1) are assigned to each of the consecutive frames (#m+1) to (#m+1) of the number of updated frames (=l), respectively. Then, a target viewpoint image group (#1) to (#1) corresponding to the assigned target viewpoint position group (#1) to (#1) is generated for each of the plurality of frames (#m+1) to (#m+1). For example, the target viewpoint image group (#3) is a group of target viewpoint images corresponding to each target viewpoint position included in the target viewpoint position group (#3) assigned to the frame (#m+3). 8, all of the viewpoint images 13 are updated in l consecutive frames (#m+1) to (#m+1). Furthermore, by repeating the processing of the consecutive frames (#m+1) to (#m+1), all of the viewpoint images 13 are also repeatedly updated. There are no limitations on how the number of update frames and the group of target viewpoint positions are set, and they may be set arbitrarily.

[0051] For example, the number of update frames is set to 2 (l=2). For example, an odd-numbered frame and the even-numbered frame that follows it are defined as frames (#m+1) and (#m+2). When P viewpoint positions 8 are set, they are assigned indices 1 to P in order from the end. That is, each viewpoint position is identified by calling it the first viewpoint position 8, the Pth viewpoint position, etc. Of course, there are no limitations on the method of identifying the viewpoint positions 8, such as the method of assigning indexes. The P viewpoint positions 8 are divided into a target viewpoint position group (♯1) consisting of odd-numbered viewpoint positions 8 and a target viewpoint position group (♯2) consisting of even-numbered viewpoint positions 8. That is, they are set so that every other viewpoint is included in a different target viewpoint position group. For two consecutive frames (♯m + 1) and (♯m + 2), the target viewpoint position group (♯1) and the target viewpoint position group (♯2) are assigned. Then, for each of the two frames (♯m + 1) and (♯m + 2), a target viewpoint image group (♯1) corresponding to the assigned target viewpoint position group (♯1) and a target viewpoint image group (♯2) corresponding to the assigned target viewpoint position group (♯2) are generated. In this way, the update of all the viewpoint images 13 may be executed every two frames.

[0052] Note that in this example, the target viewpoint position groups (♯1) and (♯2) are an embodiment of a first target viewpoint position group and a second target viewpoint position group in which the target viewpoint positions do not overlap with each other. Also, the target viewpoint image group (♯1) and the target viewpoint image group (♯2) are an embodiment of a first target viewpoint image group corresponding to the assigned first target viewpoint position group and a second target viewpoint image group corresponding to the assigned second target viewpoint position group. Of course, it is not limited to this, and for three consecutive frames, three target viewpoint position groups may be assigned and three target viewpoint image groups may be repeatedly generated.

[0053] The display image generation unit 29 reads out the viewpoint images 13 corresponding to each viewpoint position 8 from the n viewpoint image storage units 27 and generates multi-viewpoint image data. For example, as described with reference to FIG. 4, FIG. 5, etc., rearrangement of the strip images 18 obtained by dividing the viewpoint image 13 is executed, and multi-viewpoint image data is generated. Note that, as shown in FIG. 7, the multi-viewpoint image data may also be referred to as a display image (display image data).

[0054] For example, in the example shown in Figure 8, a group of target viewpoint images (#l) generated in frame (#m+l) and a group of target viewpoint images (#1 to #l-1) generated in each of frames (#m+1 to #m+l-1) earlier than frame (#m+l) are used to generate multi-viewpoint image data for frame (#m+l). In this case, the frame (#m+l) is one embodiment of a predetermined frame according to the present technology. Furthermore, each target viewpoint image group corresponds to one or more target viewpoints. Similarly, for frames other than frame (#m+l), a group of target viewpoint images (one or more target viewpoint images) generated for the frame and a group of target viewpoint images (one or more target viewpoint images) generated for a frame earlier than the frame are used to generate multi-viewpoint image data for the frame.

[0055] Display image output unit 30 outputs the multi-viewpoint image data generated for each frame to multi-viewpoint display device 5.

[0056] In the example shown in FIG. 7, the first generator 10 shown in FIG. 1 is realized by a plurality of viewpoint image generators 26. The viewpoint image generation control unit 28 realizes the generation control unit 11 shown in FIG. The display image generating unit 29 realizes the second generating unit 12 shown in FIG. Alternatively, the viewpoint image generation control unit 28 may not be configured, and each viewpoint image generation unit 26 may determine whether or not to generate a viewpoint image 13 for each frame. In this case, the plurality of viewpoint image generation units 26 also function as the generation control unit 11 shown in FIG. 1.

[0057] 8, two consecutive frames are arbitrarily selected from consecutive frames (#m+1) to (#m+1). The earlier frame of the two consecutive frames is designated as the first frame, and the later frame is designated as the second frame. The group of target viewpoint positions allocated to the first frame is defined as one or more first target viewpoint positions, and the group of target viewpoint positions allocated to the second frame is defined as one or more second target viewpoint positions. Also, the target viewpoint images corresponding to one or more first target viewpoint positions are defined as one or more first target viewpoint images. The target viewpoint images corresponding to one or more second target viewpoint positions are defined as one or more second target viewpoint images. In this case, the viewpoint image generation control unit 28 causes one or more first target viewpoint images corresponding to one or more first target viewpoint positions to be generated in the first frame, and causes one or more second target viewpoint images corresponding to one or more second target viewpoint positions different from any of the one or more first target viewpoint positions to be generated in the second frame consecutive to the first frame. Also, the display image generation unit 29 generates multi-viewpoint image data for the second frame using one or more first target viewpoint images and one or more second target viewpoint images.

[0058] Note that duplication of the target viewpoint positions may be allowed for each of the plurality of target viewpoint position groups shown in FIG. 8. For example, duplication of the target viewpoint positions may be allowed between one or more first target viewpoint positions assigned to the first frame and one or more second target viewpoint positions assigned to the second frame.

[0059] The image generation method executed by the image generation device according to the present technology is not limited to the case where the processing of frames (♯m + 1) to (♯m + l) is repeated as shown in FIG. 8, for example. For example, for a certain two consecutive frames, the above-described processing in which these frames are the first frame and the second frame is executed at least once. Then, multi-viewpoint image data for the second frame is generated using one or more first target viewpoint images and one or more second target viewpoint images. Such processing is also included in one embodiment of the image generation method executed by the image generation device according to the present technology. Furthermore, if the process of generating one or more target viewpoint images, which are some of the viewpoint images 13 rather than all of the viewpoint images 13, is executed in at least one frame and multi-viewpoint image data is generated based on the generated one or more target viewpoint images, it is included in one embodiment of the image generation method executed by the image generation device according to the present technology.

[0060] [Generation of Multi-Viewpoint Image Data] FIG. 9 is a chart showing an example of generating multi-viewpoint image data. FIG. 10 is a schematic diagram for explaining the generation of multi-viewpoint image data.

[0061] As shown in FIG. 9, it is assumed that n viewpoint positions 8 from the 1st viewpoint to the nth viewpoint are set. Corresponding to the n viewpoint positions 8, n viewpoint image generation units 26 and n viewpoint image storage units 27 are configured. Note that in FIG. 10, a diagram in the case where n = 7 is illustrated.

[0062] Also, in this example, the number of update frames is set to 2 (L = 2). Then, at odd frames, the odd-numbered viewpoint positions 8 are set as the target viewpoint position group (one or more target viewpoint positions). At even frames, the even-numbered viewpoint positions 8 are set as the target viewpoint position group (one or more target viewpoint positions). Note that m in FIG. 10 is an odd number.

[0063] The viewpoint image generation unit 26 generates one or more target viewpoint images corresponding to one or more target viewpoint positions that are the odd-numbered viewpoint positions 8 at a frame (♯m) that is an odd frame, and stores them in the viewpoint image storage unit 27. The viewpoint image 13 corresponding to the even-numbered viewpoint position 8 is not generated. In the example shown in FIG. 10A, the 1st, 3rd, 5th, and 7th viewpoint positions 8 are set as the target viewpoint positions. Virtual images (viewpoint images 13) of the character 2 captured by the virtual cameras 14 arranged at positions corresponding to these viewpoint positions 8 are generated as the target viewpoint images.

[0064] The display image generation unit 29 reads out the viewpoint images 13 corresponding to all the viewpoint positions 8 from all the viewpoint image storage units 27. For the odd-numbered viewpoint positions 8, one or more target viewpoint images generated at the frame (♯m) are read out. For the even-numbered viewpoint positions 8, one or more target viewpoint images generated at the immediately preceding even frame (♯m - 1), which is a frame past the frame (♯m), are read out. Multi-viewpoint image data is generated based on all the read-out viewpoint images 13. The generated multi-viewpoint image data is output to the multi-viewpoint display device 5 by the display image output unit 30.

[0065] In frame (#m+1), which is an even-numbered frame, one or more target viewpoint images corresponding to one or more target viewpoint positions that are even-numbered viewpoint positions 8 are generated and stored in viewpoint image storage unit 27. No viewpoint images 13 corresponding to odd-numbered viewpoint positions 8 are generated. 10B, the second, fourth, and sixth viewpoint positions 8 are the target viewpoint positions. Virtual images (viewpoint images 13) of the character 2 captured by virtual cameras 14 placed at positions corresponding to these viewpoint positions 8 are generated as target viewpoint images.

[0066] The display image generating unit 29 reads out the viewpoint images 13 corresponding to all the viewpoint positions 8 from all the viewpoint image storage units 27 . For even-numbered viewpoint positions 8, one or more target viewpoint images generated in frame (#m+1) are read out. For odd-numbered viewpoint positions 8, one or more target viewpoint images generated in the odd-numbered frame (#m) immediately preceding frame (#m), which is a frame earlier than frame (#m+1), are read out. Multi-viewpoint image data is generated based on all the read-out viewpoint images 13. The generated multi-viewpoint image data is output to the multi-viewpoint display device 5 by the display image output unit 30.

[0067] As described above, in the image generating device 6 according to this embodiment, one or more target viewpoint images corresponding to one or more target viewpoint positions that are part of the plurality of viewpoint positions 8 are generated for each frame. Therefore, one or more target viewpoint images that are part of the plurality of viewpoint images 13 are generated for each frame. This makes it possible to reduce the processing load required for generating the viewpoint images 13.

[0068] FIG. 11 is a chart showing the generation of multi-viewpoint image data as a comparative example. In the comparative example shown in FIG. 11, for each frame, the viewpoint images 13 corresponding to all the viewpoint positions 8 are generated. The generated viewpoint images 13 are temporarily stored in a buffer and used for generating multi-viewpoint image data by the display image generation unit. In the comparative example shown in FIG. 11, since all the viewpoint images 13 are generated for each frame, the load (rendering load) for generating the viewpoint images 13 increases. Also, the amount of data to be handled for each frame increases, and the display frame rate of the multi-viewpoint image decreases. In the comparative example shown in FIG. 11, consider the case where the number of viewpoint positions 8 is set to be large. In this case, the rendering load further increases and the display frame rate further decreases. Therefore, it becomes necessary to lower the quality of the video such as reducing the image resolution, and it is difficult to realize high-quality multi-viewpoint display.

[0069] In the image generation apparatus 6 according to the present embodiment, the number of viewpoint images 13 acquired for each frame is suppressed, and all the viewpoint images 13 are sequentially updated for a plurality of frames. Thereby, it becomes possible to sufficiently suppress the cost (rendering load) for generating the viewpoint images 13 per frame, and it becomes possible to sufficiently suppress the amount of data to be handled. As a result, it becomes possible to improve the display frame rate of the multi-viewpoint image without degrading the quality of the video, and it becomes possible to realize high-quality multi-viewpoint display. Also, it becomes possible to reduce the required specifications of the image generation apparatus 6.

[0070] <Other Embodiments> The present technology is not limited to the embodiments described above, and various other embodiments can be realized.

[0071] The value of the number of update frames shown in FIG. 8 is not limited and may be set arbitrarily. For example, FIG. 12 is a schematic diagram showing the case where the number of update frames is set to 3 (l = 3). The update frame number is a parameter that determines the interval (period) at which all viewpoint images 13 are updated. Therefore, setting the update frame number corresponds to setting the update period.

[0072] The update frame number may be arbitrarily changeable by the viewpoint image generation control unit 28. When the update frame number increases, the processing load per frame decreases, but since the acquisition timings of the individual viewpoint images 13 are shifted, there may be a sense of discomfort with moving objects. For example, the update frame number can be arbitrarily changed by the viewpoint image generation control unit 28 based on the movement of the object to be displayed. This makes it possible to balance the processing load and movement deviation, which are in the above-mentioned trade-off relationship. For example, when the movement of the object is fast or large, the update frame number is decreased, and all the viewpoint images 13 are updated with a short update period. When the movement of the display object is slow or the change is small, the update frame number is increased to lengthen the update period for updating all the viewpoint images 13. In this way, by making the update period (i.e., the update frame number) variable according to the movement of the object, it is possible to reduce the perception of object deviation while reducing the processing load.

[0073] The update period of the viewpoint image 13 may be determined according to the mode set by the user. The mode is a mode related to multi-viewpoint image display. For example, any mode such as a mode that prioritizes the performance of multi-viewpoint image display (high-quality display mode, etc.) or a mode that prioritizes low load (low power consumption mode, etc.) can be adopted. For example, when the user selects a mode that prioritizes performance, the update interval can be shortened to reduce the probability that the deviation of the movement of the object is perceived. Also, when a mode that prioritizes low load is selected, the update interval can be lengthened to reduce the processing load.

[0074] Alternatively, the update period (number of update frames) may be set based on the state of the user obtained by camera tracking or the like. Of course, the number of update frames may be fixed to a preset constant.

[0075] For each frame, the interpupillary distance (IPD) may be used to determine which viewpoint position 8 is set as one or more target viewpoint positions. That is, for each frame, one or more target viewpoint positions may be set based on the interpupillary distance (IPD), and one or more target viewpoint images may be generated. In the example shown in FIG. 8, based on the interpupillary distance (IPD), a plurality of viewpoint positions may be divided into a plurality of groups of target viewpoint positions. For example, as illustrated in FIG. 13, a group of virtual cameras 14 of the viewpoint images 13 acquired within a single frame, that is, a group of target viewpoint positions, may be determined by the assumed interpupillary distance (IPD) of the user at the observation position. The interpupillary distance (IPD) may be set to a predetermined value, or a value acquired using camera tracking or the like may be used. By taking the IPD into account, when the user is stationary, the same-timing updated viewpoint images 13 can be seen by the user's left and right eyes, making it difficult to perceive the movement deviation of the object. For example, this process is applied when displaying the stereoscopic image shown in FIG. 6B. As a result, the right-eye image and the left-eye image are updated at the same timing, realizing high-quality stereoscopic display.

[0076] FIG. 14 is a schematic diagram showing a functional configuration example of an image generation device according to another embodiment. FIG. 15 is a chart diagram showing an example of generation of multi-viewpoint image data. In the image generation device 206 shown in FIG. 14, instead of the plurality of viewpoint image storage units 27, a plurality of buffers 32 are configured. Also, a display image storage unit 33 is configured.

[0077] In this embodiment, one or more target viewpoint images corresponding to one or more target viewpoint positions are generated for each frame and temporarily stored in the buffer 32. In this embodiment, the multi-view image data (display image) generated by the display image generating unit 29 is stored in the display image storage unit 33 for each frame.

[0078] For each frame, the display image generation unit 29 reads out one or more target viewpoint images temporarily stored in the buffer 32. The display image generation unit 29 also reads out, from the display image storage unit 33, multi-viewpoint image data generated in a past frame. Then, the display image generating unit 29 generates multi-viewpoint image data using one or more target viewpoint images of the current frame and multi-viewpoint image data of the past frame. In this manner, in this embodiment, multi-viewpoint image data for a specified frame is generated using one or more target viewpoint images generated for a specified frame and multi-viewpoint image data generated for a frame earlier than the specified frame.

[0079] As shown in Figure 15, when generating a series of multi-viewpoint image data (display images), instead of storing and retaining the viewpoint images 13, the display images after the strip images 18 have been rearranged may be stored and retained. The display image generation unit 29 loads the rearranged display image of the previous frame and rearranges and updates only the relevant portion of the acquired viewpoint image 13. The display image generated by this update is generated as the final multi-viewpoint image data. In cases where the load of the rearrangement process of the viewpoint images 13 for generating a display image is large, the present embodiment, which stores the multi-viewpoint image data after rearrangement, makes it possible to reduce the overall load.

[0080] FIG. 16 is a schematic diagram illustrating an example of the functional configuration of an image generating device according to another embodiment. In the image generating device 306 shown in FIG. 16, a plurality of display image output units 35 are configured. For example, when a projector array such as that shown in FIG. 3 is used as the multi-viewpoint display device 5, the image generating device 306 is connected to a plurality of display devices . A plurality of display image output units 35 are configured in accordance with the number of the plurality of display devices 36. Then, a plurality of display image output units 35 are connected to each of the plurality of display devices 36. When the multi-viewpoint display device 5 has multiple output destination display devices 36, such as a configuration using a projector array, the efficiency of multi-viewpoint display is improved by corresponding the display image output unit 35 to the display device 36 on a one-to-one basis.

[0081] Instead of the camera 14, a camera array may be used as the real object to capture the real image. When generating the viewpoint image 13, a camera array may be installed in real space to generate the viewpoint image 13 from an object in the real space, instead of using a virtual camera 14 on CG. Even when generating the viewpoint image 13 from an object in the real space, the method for suppressing generation of the viewpoint image 13 according to the present technology is similarly applicable. For example, the first generating unit 10 acquires a plurality of captured images as a plurality of viewpoint images 13 from a plurality of imaging devices arranged at a plurality of viewpoint positions 8 . The generation control unit 11 causes the first generation unit 10 to output, for each frame, captured images corresponding to one or more target viewpoint images and discard the other captured images. That is, in this example, the output of captured images by the first generation unit 10 corresponds to the generation of target viewpoint images by the first generation unit 10. This embodiment makes it possible to display not only CG but also real objects on the multi-viewpoint display device 5. For example, it becomes possible to reduce the cost of copying the target viewpoint image (captured image) output from the first generation unit 10 to a buffer. Of course, the effects are not limited to this.

[0082] For each frame, user position information may be used to determine which viewpoint positions 8 are to be set as one or more target viewpoint positions. That is, one or more target viewpoint positions may be set based on the position information of the user, and one or more target viewpoint images may be generated. The method for acquiring the user's location information is not limited, and any method such as camera tracking may be used. Furthermore, the user's location information may be estimated by machine learning. If the position at which the viewpoint images are generated is limited to the observer's position, the image cannot be observed from other positions, but the number of viewpoint images 13 generated can be significantly reduced, thereby reducing the processing load and improving the display frame rate. Instead of / in addition to the user's position information, the user's line of sight information or the like may be used.

[0083] FIG. 17 is a block diagram showing an example of the hardware configuration of the image generating device 6. The image generating device 6 includes a CPU 61, a ROM (Read Only Memory) 62, a RAM 63, an input / output interface 65, and a bus 64 that interconnects these components. The input / output interface 65 is connected to a display unit 66, an input unit 67, a storage unit 68, a communication unit 69, a drive unit 70, and the like. The display unit 66 is a display device using, for example, a liquid crystal display, an electroluminescent display, etc. The input unit 67 is, for example, a keyboard, a pointing device, a touch panel, or other operating device. When the input unit 67 includes a touch panel, the touch panel can be integrated with the display unit 66. The storage unit 68 is a non-volatile storage device such as a HDD, flash memory, or other solid-state memory. The drive unit 70 is a device capable of driving a removable storage medium 71 such as an optical storage medium or magnetic recording tape. The communication unit 69 is a modem, router, or other communication device that can be connected to a LAN, WAN, etc. and that communicates with other devices. The communication unit 69 may communicate using either a wired or wireless method. The communication unit 69 is often used separately from the image generation device 6. The information processing (image generation) by the image generation apparatus 6 having the above-described hardware configuration is realized by the cooperation of software stored in the storage unit 68 or the ROM 62 or the like and the hardware resources of the image generation apparatus 6. Specifically, by loading and executing a program constituting the software, which is stored in the ROM 62 or the like, into the RAM 63, the information processing method (image generation method) according to the present technology is realized. The program is installed in the image generation apparatus 6, for example, via the recording medium 61. Alternatively, the program may be installed in the image generation apparatus 6 via a global network or the like. In addition, any computer-readable non-transitory storage medium may be used.

[0084] The image generation method and program according to the present technology may be executed, and the image generation apparatus according to the present technology may be constructed, by the cooperation of a plurality of computers communicably connected via a network or the like. That is, the image generation method and program according to the present technology are executable not only in a computer system constituted by a single computer but also in a computer system in which a plurality of computers operate in conjunction with each other. In the present disclosure, a system means a collection of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, both a plurality of devices housed in separate housings and connected via a network and a single device in which a plurality of modules are housed in one housing are systems. The execution of the image generation method and program according to the present technology by a computer system includes, for example, both cases where the generation of a viewpoint image, the setting of a target viewpoint position, the generation of multi-viewpoint image data, etc. are executed by a single computer and cases where each process is executed by different computers. In addition, the execution of each process by a predetermined computer includes causing another computer to execute a part or all of the process and obtaining the result. That is, the image generation method and program according to the present technology can also be applied to a cloud computing configuration in which one function is shared and jointly processed by a plurality of devices via a network.

[0085] The configurations such as the image generation system, the multi-viewpoint display device, and the image generation device, and the processing flows, etc. described with reference to the respective drawings are merely one embodiment, and can be arbitrarily modified without departing from the gist of the present technology. That is, any other arbitrary configurations, algorithms, etc. for implementing the present technology may be adopted.

[0086] In the present disclosure, when the term "substantially" is used, this is only for facilitating the understanding of the explanation, and there is no special meaning in the use / non-use of the term "substantially". That is, in the present disclosure, concepts that define shapes, sizes, positional relationships, states, etc., such as "center", "center", "uniform", "equal", "same", "orthogonal", "parallel", "symmetric", "extending", "axial direction", "cylindrical shape", "cylindrical shape", "ring shape", "annular shape", etc., are concepts including "substantially center", "substantially center", "substantially uniform", "substantially equal", "substantially same", "substantially orthogonal", "substantially parallel", "substantially symmetric", "substantially extending", "substantially axial direction", "substantially cylindrical shape", "substantially cylindrical shape", "substantially ring shape", "substantially annular shape", etc. For example, states included in a predetermined range (for example, a range of ±10%) based on "completely center", "completely center", "completely uniform", "completely equal", "completely same", "completely orthogonal", "completely parallel", "completely symmetric", "completely extending", "completely axial direction", "completely cylindrical shape", "completely cylindrical shape", "completely ring shape", "completely annular shape", etc. are also included. Therefore, even when the term "substantially" is not added, a concept expressed by adding so-called "substantially" may be included. Conversely, a complete state is not excluded for a state expressed by adding "substantially".

[0087] In the present disclosure, expressions using "greater than" such as "greater than A" and "less than" such as "less than A" are expressions comprehensively including both concepts including the case of being equal to A and concepts not including the case of being equal to A. For example, "greater than A" is not limited to the case of not including equality with A, and also includes "greater than or equal to A". Further, "less than A" is not limited to "less than A", and also includes "less than or equal to A". When implementing the present technology, specific settings and the like may be appropriately adopted from the concepts included in "greater than A" and "less than A" so that the effects described above are exhibited.

[0088] Among the characteristic parts related to the present technology described above, it is also possible to combine at least two characteristic parts. That is, the various characteristic parts described in each embodiment may be arbitrarily combined without distinction between the embodiments. Further, the various effects described above are merely examples and are not limiting, and other effects may be exhibited.

[0089] Note that the present technology can also adopt the following configuration. (1) An image generation device that generates multi-viewpoint image data for displaying a multi-viewpoint image at a predetermined frame rate, a first generation unit capable of generating a plurality of viewpoint images corresponding to a plurality of viewpoint positions, a generation control unit that, for each frame, sets one or more target viewpoint positions that are part of the plurality of viewpoint positions, and causes the first generation unit to generate one or more target viewpoint images corresponding to the set one or more target viewpoint positions, a second generation unit that generates the multi-viewpoint image data using the one or more target viewpoint images generated for each frame, and an image generation device comprising the same. (2) The image generation device according to (1), wherein the second generation unit generates the multi-viewpoint image data for the predetermined frame using the one or more target viewpoint images generated in the predetermined frame and the one or more target viewpoint images generated in a frame earlier than the predetermined frame. Image generation device. (3) The image generation device according to (1) or (2), wherein the generation control unit generates one or more first target viewpoint images corresponding to one or more first target viewpoint positions in a first frame, and generates one or more second target viewpoint images corresponding to one or more second target viewpoint positions different from any of the one or more first target viewpoint positions in a second frame consecutive to the first frame Image generation device. (4) The image generation device according to (3), wherein the second generation unit generates the multi-viewpoint image data of the second frame using the one or more first target viewpoint images and the one or more second target viewpoint images. Image generation device. (5) The image generation device according to any one of (1) to (4), wherein the generation control unit sets the number of update frames, divides the plurality of viewpoint positions into a plurality of groups of target viewpoint positions having the same number as the number of update frames and not overlapping with each other in the target viewpoint positions, for each of the consecutive frames of the number of update frames, assigns each of the plurality of groups of target viewpoint positions, and generates a group of target viewpoint images corresponding to the assigned group of target viewpoint positions in each of the plurality of frames. Image generation device. (6) The image generation device according to (5), wherein the generation control unit sets the number of update frames to 2, divides the plurality of viewpoint positions into a first group of target viewpoint positions and a second group of target viewpoint positions that do not overlap with each other in the target viewpoint positions, assigns the first group of target viewpoint positions and the second group of target viewpoint positions to two consecutive frames, and generates a first group of target viewpoint images corresponding to the assigned first group of target viewpoint positions and a second group of target viewpoint images corresponding to the assigned second group of target viewpoint positions in each of the two frames. Image generating device. (7) The image generating device according to (5) or (6), The generation control unit is capable of changing the number of update frames. Image generating device. (8) The image generating device according to (5) or (6), 6. The image generating device according to claim 5, The generation control unit changes the number of update frames based on a movement of an object to be displayed or a mode set for the multi-viewpoint image display. Image generating device. (9) The image generating device according to any one of (1) to (8), The generation control unit sets the one or more target viewpoint positions based on an interpupillary distance for each frame, and generates the one or more target viewpoint images. Image generating device. (10) The image generating device according to (5), The generation control unit classifies the plurality of viewpoint positions into the plurality of target viewpoint position groups based on an interpupillary distance. Image display device. (11) The image generating device according to (1), The second generation unit uses the one or more target viewpoint images generated in a predetermined frame and the multi-viewpoint image data generated in a frame earlier than the predetermined frame, Generate the multi-viewpoint image data of the predetermined frame Image generating device. (12) The image generating device according to any one of (1) to (11), The first generation unit generates a virtual image as the viewpoint image. Image generating device. (13) The image generating device according to any one of (1) to (11), the first generation unit acquires, as the plurality of viewpoint images, a plurality of captured images from a plurality of image capturing devices arranged at the plurality of viewpoint positions; The generation control unit causes the first generation unit to output, for each frame, a captured image corresponding to the one or more target viewpoint images and discard other captured images. Image generating device. (14) The image generating device according to (1), The generation control unit sets the one or more target viewpoint positions based on user position information for each frame, and generates the one or more target viewpoint images. Image generating device. (15) The image generating device according to any one of (1) to (14), The second generation unit generates the multi-viewpoint image data as data for multi-viewpoint display on a multi-viewpoint display device. Image generating device. (16) The image generating device according to (15), the multi-viewpoint display device includes a plurality of projectors; The second generation unit generates a plurality of corresponding multi-viewpoint image data corresponding to each of the plurality of projectors as the multi-viewpoint image data. Image generating device. (17) The image generating device according to (15), the multi-viewpoint display device includes a multi-viewpoint display; The second generation unit generates the multi-viewpoint image data corresponding to the multi-viewpoint display. Image generating device. (18) An image generation method executed by a computer system for generating multi-viewpoint image data for displaying a multi-viewpoint image at a predetermined frame rate, comprising: For each frame, one or more target viewpoint positions that are part of the plurality of viewpoint positions are set, and one or more target viewpoint images corresponding to the one or more set target viewpoint positions are generated; The multi-viewpoint image data is generated using the one or more target viewpoint images generated for each frame. Image generation method. (19) A program for causing a computer system to execute an image generation method for generating multi-viewpoint image data for displaying a multi-viewpoint image at a predetermined frame rate, The image generation method includes: For each frame, one or more target viewpoint positions that are part of the plurality of viewpoint positions are set, and one or more target viewpoint images corresponding to the one or more set target viewpoint positions are generated; The multi-viewpoint image data is generated using the one or more target viewpoint images generated for each frame. program. [Explanation of symbols]

[0090] 2. Characters 3...Images displayed corresponding to each viewpoint 5...Multi-viewpoint display device 6, 206, 306...Image generating device 8...Viewpoint position 10...First generation unit 11...Generation control unit 12...Second generation section 13...Perspective image 14...Virtual Camera 16...Projector 17...Light control element 18...Tanzaku image 19...Projected image 21...Multi-view display 22...Flat display panel 23...Lenticular lens 100...Image generation system

Claims

1. An image generation device that generates multi-viewpoint image data for displaying a multi-viewpoint image at a predetermined frame rate, comprising: a first generation unit capable of generating a plurality of viewpoint images corresponding to a plurality of viewpoint positions; a generation control unit that, for each frame, sets one or more target viewpoint positions, the number of which is less than the number of the plurality of viewpoint positions, from among the plurality of viewpoint positions, and causes the first generation unit to generate one or more target viewpoint images corresponding to the set one or more target viewpoint positions; a second generation unit that generates the multi-viewpoint image data for the predetermined frame using the one or more target viewpoint images generated in the predetermined frame and the one or more target viewpoint images generated in a frame prior to the predetermined frame or the multi-viewpoint image data generated in a frame prior to the predetermined frame; and the generation control unit dynamically sets the number of update frames for updating all of the plurality of viewpoint images based on at least one of the movement of an object to be displayed, a mode set with respect to the quality of the multi-viewpoint image to be displayed, and a mode set with respect to the processing load of the image generation device, and sets the one or more target viewpoint positions for each of a plurality of consecutive frames having the same number as the number of update frames by allocating the plurality of viewpoint positions to each of the plurality of consecutive frames without overlap. An image generation device.

2. The image generation device according to claim 1, wherein the generation control unit causes one or more first target viewpoint images corresponding to one or more first target viewpoint positions to be generated in a first frame, and causes one or more second target viewpoint images corresponding to one or more second target viewpoint positions, all of which are different from any of the one or more first target viewpoint positions, to be generated in a second frame consecutive to the first frame. An image generation device.

3. The image generation device according to claim 2, wherein the second generation unit generates the multi-viewpoint image data for the second frame using the one or more first target viewpoint images and the one or more second target viewpoint images. An image generation device.

4. The image generation device according to claim 1, wherein the generation control unit sets the number of update frames to 2, and sets the one or more target viewpoint positions for each of two consecutive frames by allocating the plurality of viewpoint positions to each of the two consecutive frames without overlap. An image generation device.

5. The image generation device according to claim 1, wherein the generation control unit allocates the plurality of viewpoint positions to each of a plurality of consecutive frames equal in number to the update frame number without duplication based on the interpupillary distance Image generation device.

6. The image generation device according to claim 1, wherein the first generation unit generates a virtual image as the viewpoint image Image generation device.

7. The image generation device according to claim 1, wherein the first generation unit acquires a plurality of captured images captured by a plurality of imaging devices arranged at the plurality of viewpoint positions as the plurality of viewpoint images, and the generation control unit causes the first generation unit to output, for each frame, a captured image corresponding to the one or more target viewpoint images and discard other captured images Image generation device.

8. The image generation device according to claim 1, wherein the generation control unit sets the one or more target viewpoint positions based on the position information of the user for each frame and generates the one or more target viewpoint images Image generation device.

9. The image generation device according to claim 1, wherein the second generation unit generates the multi-viewpoint image data as data for multi-viewpoint display of a multi-viewpoint display device Image generation device.

10. The image generation device according to claim 9, wherein the multi-viewpoint display device includes a plurality of projectors, and the second generation unit generates, as the multi-viewpoint image data, a plurality of corresponding multi-viewpoint image data corresponding to each of the plurality of projectors Image generation device.

11. The image generation device according to claim 9, wherein the multi-viewpoint display device includes a multi-viewpoint display, and the second generation unit generates the multi-viewpoint image data corresponding to the multi-viewpoint display Image generation device.

12. An image generation method for generating multi-viewpoint image data for displaying a multi-viewpoint image at a predetermined frame rate, which is executed by an image generation device, comprising: for each frame, setting one or more target viewpoint positions from among the plurality of viewpoint positions, the number of which is less than the number of the plurality of viewpoint positions, and generating one or more target viewpoint images corresponding to the set one or more target viewpoint positions A step of generating the multi-viewpoint image data of the predetermined frame by using the one or more target viewpoint images generated in a predetermined frame and the one or more target viewpoint images generated in a frame prior to the predetermined frame or the multi-viewpoint image data generated in a frame prior to the predetermined frame including The step of generating the one or more target viewpoint images dynamically sets the number of update frames for updating all of the plurality of viewpoint images corresponding to the plurality of viewpoint positions based on at least one of the movement of an object to be displayed, a mode set with respect to the quality of the multi-viewpoint image to be displayed, and a mode set with respect to the processing load of the image generation device sets the one or more target viewpoint positions for each of a number of consecutive frames equal to the number of update frames by assigning the plurality of viewpoint positions to each of the number of consecutive frames equal to the number of update frames without duplication Image generation method A program for causing an image generation device to execute an image generation method for generating multi-viewpoint image data for displaying a multi-viewpoint image at a predetermined frame rate, wherein the image generation method for each frame, sets one or more target viewpoint positions, which are fewer than the number of the plurality of viewpoint positions, from among the plurality of viewpoint positions, and generates one or more target viewpoint images corresponding to the set one or more target viewpoint positions a step of generating the multi-viewpoint image data of the predetermined frame by using the one or more target viewpoint images generated in a predetermined frame and the one or more target viewpoint images generated in a frame prior to the predetermined frame or the multi-viewpoint image data generated in a frame prior to the predetermined frame including The step of generating the one or more target viewpoint images dynamically sets the number of update frames for updating all of the plurality of viewpoint images corresponding to the plurality of viewpoint positions based on at least one of the movement of an object to be displayed, a mode set with respect to the quality of the multi-viewpoint image to be displayed, and a mode set with respect to the processing load of the image generation device sets the one or more target viewpoint positions for each of a number of consecutive frames equal to the number of update frames by assigning the plurality of viewpoint positions to each of the number of consecutive frames equal to the number of update frames without duplication Program

Images