Information processing apparatus, information processing system, and program

JP2024096896A5Active Publication Date: 2024-12-05CANON KK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024065889
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-04-16
Publication Date
2024-12-05
Estimated Expiration
2041-11-29

AI Technical Summary

Technical Problem

Existing virtual viewpoint image generation technologies face increased processing load when multiple virtual viewpoints are specified and a specific virtual viewpoint is selected, and there is a demand for generating selected virtual viewpoint images with appropriate quality.

Method used

An information processing device that includes a viewpoint information acquisition means, a selection means, and two image generation units to generate virtual viewpoint images based on quality conditions, one unit generating multiple images at lower quality and another unit generating higher quality images based on selected viewpoint information.

Benefits of technology

The solution effectively suppresses processing load while ensuring the generation of high-quality virtual viewpoint images corresponding to the selected virtual viewpoint.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To reduce a processing load when selecting a specific virtual viewpoint from a plurality of virtual viewpoints, and appropriately create a virtual viewpoint image corresponding to the selected virtual viewpoint.SOLUTION: An information processing apparatus 6 has: a viewpoint information acquisition unit 61 that acquires viewpoint information representing the position of a virtual viewpoint used for creation of a virtual viewpoint image created on the basis of a plurality of picked-up images obtained by a plurality of imaging apparatuses picking up images, and a visual line direction from the virtual viewpoint; and a viewpoint information output unit 63 that outputs a plurality of pieces of viewpoint information acquired on the basis of a plurality of pieces of operation to first creation means that creates a plurality of virtual viewpoint images based on a condition related to first quality on the basis of the viewpoint information, and outputs one or more pieces of viewpoint information selected from the plurality of pieces of viewpoint information to second creation means that creates a virtual viewpoint image based on a condition related to second quality higher than the first quality on the basis of the viewpoint information.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a technology for generating a virtual viewpoint image. [Background technology]

[0002] In recent years, a technology that generates a virtual viewpoint image using the multiple captured images obtained by installing multiple imaging devices (cameras) at different positions and capturing images from different directions has been attracting attention. In the technology for generating a virtual viewpoint image, a user can view an image seen from a specified viewpoint (virtual viewpoint). According to the technology for generating a virtual viewpoint image as described above, for example, a highlight scene of a soccer or basketball game can be viewed from various angles, so that a user can feel a high sense of realism compared to a normal image. Patent Document 1 describes a technology for generating a virtual viewpoint image corresponding to a virtual viewpoint specified based on an operation by a user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-050593 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, the case where multiple virtual viewpoints are specified by multiple operations is not taken into consideration. In addition, when multiple virtual viewpoints are specified and a specific virtual viewpoint is selected from among them, a process is required to generate multiple virtual viewpoint images and present them to the user, which increases the processing load related to the generation of the virtual viewpoint images. In addition, there is a demand for the selected virtual viewpoint image to be generated with appropriate quality.

[0005] The present disclosure aims to reduce the processing load when selecting a specific virtual viewpoint from among a plurality of virtual viewpoints, while appropriately generating a virtual viewpoint image corresponding to the selected virtual viewpoint. [Means for solving the problem]

[0006] The information processing device according to the present disclosure includes an acquisition means for acquiring viewpoint information representing the position of a virtual viewpoint and the line of sight direction from the virtual viewpoint used to generate a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images using a plurality of imaging devices, a first output means for outputting the plurality of viewpoint information acquired by the acquisition means based on a plurality of operations to a first generation means for generating a plurality of virtual viewpoint images based on a first quality condition based on the viewpoint information, and a second output means for outputting one or more viewpoint information selected from the plurality of viewpoint information to a second generation means for generating a virtual viewpoint image based on a second quality condition higher than the first quality based on the viewpoint information. Effect of the Invention

[0007] According to the present disclosure, a virtual viewpoint image corresponding to a selected virtual viewpoint is appropriately generated while suppressing the processing load when a specific virtual viewpoint is selected from a plurality of virtual viewpoints. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system. [Diagram 2] 10 is a flowchart for explaining a process related to display of a virtual viewpoint image. [Diagram 3] 11A and 11B are diagrams illustrating an example of display on a multiple image display unit and an output image display unit. [Figure 4] FIG. 13 is a diagram illustrating a configuration of an information processing system according to a modified example. [Diagram 5] FIG. 2 is a diagram illustrating a hardware configuration of an information processing device. [Figure 6]FIG. 2 is a diagram illustrating a functional configuration of an information processing device. [Figure 7] FIG. 2 is a diagram showing an example of an arrangement of a plurality of imaging units. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are examples of the embodiments of the present disclosure, and the scope of the present disclosure is not limited to these.

[0010] (First embodiment) The hardware configuration of the information processing device in this embodiment will be described below. Fig. 5 is a block diagram showing the hardware configuration of the information processing device 6. The information processing device 6 has a CPU 501, a RAM 502, a ROM 503, an operation unit 504, an output unit 505, an auxiliary storage device 506, an I / F 507, and a bus 508.

[0011] The CPU 501 controls the entire computer using computer programs and data stored in the RAM 502 and the ROM 503, and executes each process performed by the information processing device 6 according to this embodiment. That is, the CPU 501 functions as each processing unit of the information processing device 6. Each processing unit will be described later.

[0012] The RAM 502 has an area for temporarily storing computer programs and data loaded from the auxiliary storage device 506, data acquired from the outside via an I / F (interface) 507, etc. Furthermore, the RAM 502 has a work area used when the CPU 501 executes various processes. That is, the RAM 502 can be allocated as a frame memory, for example, or can provide various other areas as appropriate.

[0013] The ROM 503 stores setting data for the computer, a boot program, etc. The operation unit 504 is composed of a keyboard, a mouse, a joystick, a touch panel, etc., and can be operated by a user of the computer to input various instructions to the CPU 501. The output unit 505 displays the results of processing by the CPU 501. The output unit 505 is composed of, for example, a liquid crystal display, a touch panel, etc.

[0014] The auxiliary storage device 506 is a large-capacity information storage device, such as a hard disk drive. The auxiliary storage device 506 stores an OS (operating system) and computer programs for causing the CPU 501 to realize the functions of each unit shown in Fig. 1. Furthermore, the auxiliary storage device 506 may store image data to be processed. The computer programs and data stored in the auxiliary storage device 506 are loaded into the RAM 502 as appropriate under the control of the CPU 501, and become the subject of processing by the CPU 501.

[0015] The I / F 507 can connect to a network such as a LAN or the Internet, and other devices such as a projector or display device, and the computer can obtain and send various information via the I / F 507. For example, when the information processing device 6 is connected to an external device by wire, a communication cable is connected to the I / F 507. When the information processing device 6 has a function of wirelessly communicating with an external device, the I / F 507 is equipped with an antenna. In this embodiment, a plurality of imaging devices are connected to the I / F 507, and the captured images are obtained and each imaging device is controlled. The bus 508 is a bus that connects the above-mentioned components.

[0016] In this embodiment, the operation unit 504, the output unit 505, and the auxiliary storage device 506 are assumed to exist inside the information processing device 200, but are not limited to this. The information processing device 6 may be configured such that at least one of the operation unit 504, the output unit 505, and the auxiliary storage device 506 is connected to the outside of the information processing device 6 as a separate device. Note that, although the above is a description of the hardware configuration of the information processing device 6, the hardware configuration shown in FIG. 5 is also applicable to other devices included in the information processing system 100 described later.

[0017] Next, a configuration example of an image processing system for generating a virtual viewpoint image according to this embodiment is shown in Fig. 1. The information processing system 100 in this embodiment includes a plurality of imaging units 1, a synchronization unit 2, a three-dimensional shape estimation unit 3, a storage unit 4, a viewpoint generation unit 5, an information processing device 6, a selection unit 7, an image generation unit 8, a display unit 9, a time control unit 10, and a video distribution device 11. In this embodiment, the generated virtual viewpoint image is a moving image, but is not limited to this. This embodiment can also be applied to a virtual viewpoint image of a still image. Each processing unit will be described below.

[0018] The imaging unit 1 is an imaging device such as a camera. The synchronization unit 2 is a device such as a time server, and outputs a synchronization signal to the multiple imaging units 1. The multiple imaging units 1 capture images of the imaging area in high-precision synchronization with each other based on the synchronization signal transmitted from the synchronization unit 2. The multiple imaging units 1 can be installed to surround the imaging area, for example, as shown in FIG. 7. Note that the arrangement and number of the imaging units 1 are not limited to the example of FIG. 7. The imaging area can be, for example, outdoors such as a sports stadium, field, or park, or indoors such as a gymnasium, concert hall, stage, or studio. Each imaging unit 1 outputs a captured image obtained by imaging to the three-dimensional shape estimation unit 3.

[0019] The three-dimensional shape estimation unit 3 generates a three-dimensional model representing the three-dimensional shape of the subject using multiple captured images acquired from multiple imaging units 1. An example of a process for generating a three-dimensional model by the three-dimensional shape estimation unit 3 will be described below. The three-dimensional shape estimation unit 3 generates a foreground image and a background image using the multiple captured images acquired.

[0020] Here, a foreground image is an image in which an object area (foreground area) is extracted from an image captured by an imaging device. An object extracted as a foreground area is a dynamic object (moving body) that moves (its absolute position or shape can change) when images are captured from the same direction in time series. An object is, for example, a person such as a player or a referee on the field where a sport is being played, such as a ball in a ball game, or a singer, musician, performer, or presenter in a concert or entertainment event. An object extracted as a foreground area can also be, for example, an item used by a person.

[0021] Moreover, the background image is an image of at least a region (background region) different from the foreground object. Specifically, the background image is an image in a state in which the foreground object is removed from the captured image. Moreover, the background refers to an imaged object that remains stationary or nearly stationary when images are captured from the same direction in time series. Such imaged objects are, for example, a stage for a concert or the like, a stadium for an event such as a competition, a structure such as a goal used in a ball game, and a field. However, the background is at least a region different from the foreground object, and the imaged object may include other objects in addition to the object and the background. Moreover, the three-dimensional model data is data that represents the above-mentioned object in a three-dimensional shape.

[0022] An example of a method for generating a foreground image and a background image by the three-dimensional shape estimation unit 3 will be described. The three-dimensional shape estimation unit 3 detects an area where pixel values ​​do not change by comparing multiple captured images obtained by capturing images at consecutive times by multiple imaging units 1. The three-dimensional shape estimation unit 3 determines that the detected area is a background area, and generates a background image based on the detected area. The three-dimensional shape estimation unit 3 also compares the generated background image with the captured image, determines that an area where the difference in image values ​​is equal to or greater than a predetermined threshold is a foreground area, and generates a foreground image based on that area.

[0023] Next, an example of a generation method in which the three-dimensional shape estimation unit 3 generates a three-dimensional model of an object corresponding to a foreground region will be described. Based on the foreground region and background region determined in the above-mentioned process, the three-dimensional shape estimation unit 3 generates a silhouette image of the object in which each region is expressed in binary. At this time, the three-dimensional shape estimation unit 3 generates silhouette images of the object viewed from multiple directions by using captured images captured from multiple directions. The three-dimensional shape estimation unit 3 generates a three-dimensional model by using the multiple silhouette images, for example, a shape-from-silhouette method, which is a well-known technique.

[0024] In the above example, the configuration in which the three-dimensional shape estimation unit 3 generates the foreground image, background image, and silhouette image has been described, but the present invention is not limited to this. For example, the imaging unit 1 or another device connected to the imaging unit 1 may have a function of generating at least one of the foreground image, background image, and silhouette image. In this case, the imaging unit 1 or the other device outputs the generated image to the three-dimensional shape estimation unit 3, and the three-dimensional shape estimation unit 3 generates a three-dimensional model using the acquired image. The three-dimensional shape estimation unit 3 outputs the generated three-dimensional model to the storage unit 4.

[0025] The storage unit 4 is, for example, a device such as a database server, and saves and accumulates the following data group as data for generating a virtual viewpoint image. Specifically, the three-dimensional model input from the three-dimensional shape estimation unit 3. Camera parameters representing the position, attitude, and optical characteristics of each imaging unit 1. The storage unit 4 may also record a background model and a background texture image in advance in order to render the background of the virtual viewpoint image. The storage unit 4 may also acquire captured images from the imaging unit 1 and accumulate them.

[0026] The viewpoint generating units 5a to 5d are composed of an operation unit, which is an input device such as a joystick, and a display unit for displaying a virtual viewpoint image corresponding to the viewpoint being operated. The viewpoint generating units 5a to 5d generate viewpoint information representing the position of the virtual viewpoint and the line of sight from the virtual viewpoint based on the input from the operation unit, and input the generated viewpoint information to the information processing device 6. Note that the viewpoint information may include information equivalent to internal parameters of the camera, such as focal length and angle of view, in addition to the information representing the position of the virtual viewpoint and the line of sight from the virtual viewpoint. In the example of FIG. 4, the viewpoint generating units are four, 5a to 5d, but may be three or less or five or more. In the following description, unless there is a particular need to distinguish between them, they will be referred to as the viewpoint generation unit 5.

[0027] The information processing device 6 acquires viewpoint information from the viewpoint generation unit 5, and outputs it to a first image generation unit 8a and a second image generation unit 8b, which will be described later. At this time, the information processing device 6 also receives an instruction to select viewpoint information from a selection unit 7, which will be described later. The configuration of the information processing device 6 will be described later.

[0028] The selection unit 7 is an input device such as a mouse, a keyboard, or a switch, and transmits an instruction to the information processing device 6 to select one or more pieces of viewpoint information from the viewpoint information output by the viewpoint generation unit 5.

[0029] The first image generating unit 8a and the second image generating unit 8b are devices such as a rendering server, and generate a virtual viewpoint image based on viewpoint information acquired from the information processing device 6. When generating the virtual viewpoint image, the first image generating unit 8a and the second image generating unit 8b acquire data required for generating the virtual viewpoint image from the storage unit 4 based on time information included in the viewpoint information. Here, the time information is information related to the time related to the virtual viewpoint image corresponding to the viewpoint information, and specifically, may be a time code indicating the time when the imaging unit 1 performed imaging. In this embodiment, similar time information is also given to data such as the three-dimensional model and the foreground image accumulated in the storage unit 4, and by specifying the time information, it is possible to acquire the required data from the storage unit 4. Note that the time information is not limited to the imaging time of the imaging unit 1, and may be expressed, for example, as a relative time from a predetermined time or a frame number of a video.

[0030] The first image generating unit 8a and the second image generating unit 8b perform rendering processing using a three-dimensional model and a foreground image to generate a virtual viewpoint image of an object that represents the appearance from a virtual viewpoint. The first image generating unit 8a and the second image generating unit 8b also generate a background image using a background model and a background texture, and generate a virtual viewpoint image with a background by synthesizing the background image with the virtual viewpoint image of the object.

[0031] The first image generating unit 8a generates a plurality of virtual viewpoint images based on a predetermined quality condition, based on a plurality of viewpoint information generated by operating the plurality of viewpoint generating units 5. Here, the quality condition of the virtual viewpoint image in this embodiment is a condition related to the resolution of the generated image and the frame rate of the video. The higher the resolution and the higher the frame rate, the higher the quality of the virtual viewpoint image. The first image generating unit 8a outputs the generated virtual viewpoint image to the plurality of image display unit 9a. Note that the first image generating unit 8a and the second image generating unit 8b in this embodiment are different devices.

[0032] The second image generating unit 8b acquires one or more pieces of viewpoint information selected by the selecting unit 7 from the plurality of pieces of viewpoint information, and generates a virtual viewpoint image. At this time, the generated virtual viewpoint image has a higher quality than the virtual viewpoint image generated by the first image generating unit 8a. The second image generating unit 8b outputs the generated virtual viewpoint image to the output image display unit 9b and the video distribution device 11. That is, the first image generating unit 8a generates a plurality of virtual viewpoint images so as to satisfy the conditions for generating a virtual viewpoint image having at least one of a predetermined resolution and a predetermined frame rate. Also, the second image generating unit 8b generates a virtual viewpoint image so as to satisfy the conditions for generating a virtual viewpoint image having at least one of a resolution higher than the predetermined resolution and a frame rate higher than the predetermined frame rate.

[0033] The multiple image display unit 9a and the output image display unit 9b display the virtual viewpoint images acquired from the first image generation unit 8a and the second image generation unit 8b, respectively. The time control unit 10 controls the first image generation unit 8a and the second image generation unit 8b to synchronize the time information used when generating the virtual viewpoint images. The video distribution device 11 distributes the virtual viewpoint images transmitted from the second image generation unit 8b, for example, to television broadcasting, the Internet, etc.

[0034] Next, the functional configuration of the information processing device 6 will be described with reference to Fig. 6. The information processing device 6 has a viewpoint information acquisition unit 61, a reservation acceptance unit 62, and a viewpoint information output unit 63. The viewpoint information acquisition unit 61 acquires viewpoint information transmitted from the viewpoint generation unit 5 and outputs it to the viewpoint information output unit 63. When there are multiple viewpoint generation units 5, the viewpoint information acquisition unit 61 acquires multiple pieces of viewpoint information and outputs it to the viewpoint information output unit 63. The selection acceptance unit 62 accepts an instruction to select viewpoint information from the selection unit 7, and outputs information representing the accepted instruction to the viewpoint information output unit 63.

[0035] The viewpoint information output unit 63 outputs viewpoint information to the first image generation unit 8a and the second image generation unit 8b based on the acquired viewpoint information and the selection instruction information. At this time, the viewpoint information output unit 63 outputs a plurality of viewpoint information to the first image generation unit 8a, and outputs one or more viewpoint information selected from the plurality of viewpoint information based on the selection instruction information to the second image generation unit 8b. In this way, the viewpoint information output unit 63 outputs viewpoint information to the first image generation unit 8a and the second image generation unit 8b, and controls so that the virtual viewpoint images generated by each processing unit are displayed on the multiple image display unit 9a and the output image display unit 9b.

[0036] Next, the process performed by the information processing system 100 in this embodiment will be described with reference to FIG. 2. FIG. 2(a) is a flowchart of the process performed by the information processing device 6. The process of FIG. 2(a) is performed by the CPU 501 of the information processing device 6 reading and executing a program stored in the RAM 502 and the ROM 503. In the following description, the process steps are simply referred to as S. The process starts when viewpoint information is input from the viewpoint generating unit 5. Furthermore, the processes of S201 to S205 are repeatedly executed for each video frame of the virtual viewpoint image to be generated. That is, when the video to be generated is 60 FPS (frames per second), the processes of S201 to S205 are repeated every 1 / 60th of a second.

[0037] In S201, the viewpoint information acquisition unit 61 acquires a plurality of pieces of viewpoint information generated by operating the operation units of the plurality of viewpoint generation units 5. In S202, the viewpoint information output unit 63 outputs the acquired plurality of pieces of viewpoint information to the first image generation unit 8a.

[0038] In S203, the selection receiving unit 62 judges whether a new selection instruction for viewpoint information has been acquired from the selection unit 7. For example, when a selection instruction is received for the first time, or when a selection instruction for selecting viewpoint information different from the viewpoint information selected in S204 in the previous iteration of S201 to S205 has been received, the process proceeds to S204. On the other hand, when a new selection instruction is not acquired, or when there is no change from the viewpoint information selected in S204 in the previous iteration of S201 to S205, the process proceeds to S205.

[0039] In S204, the viewpoint information output unit 63 outputs to the second image generating unit 8b one or more pieces of viewpoint information selected by the selection instruction acquired by the selection instruction receiving unit 62 from among the multiple pieces of viewpoint information acquired by the viewpoint information acquisition unit 61. In S205, the viewpoint information acquisition unit 61 determines whether acquisition of the viewpoint information has been completed based on, for example, an end instruction from the viewpoint generating unit 5. If it is determined that acquisition has been completed, the process is terminated, and if not, the processes of S201 to S205 are repeated.

[0040] Next, the processing performed by the first image generating unit 8a and the second image generating unit 8b will be described with reference to FIG. 2(b). The processing in FIG. 2(b) is performed by the CPUs of the first image generating unit 8a and the second image generating unit 8b reading and executing programs stored in the RAM and the ROM. It is assumed that the first image generating unit 8a and the second image generating unit 8b each independently perform the processing shown in FIG. 2(b). In the following description, the first image generating unit 8a and the second image generating unit 8b will be simply referred to as the image generating unit 8 unless otherwise specified. When the image generating unit 8 acquires viewpoint information, processing begins.

[0041] In S211, the image generation unit 8 acquires viewpoint information from the information processing device 6. At this time, the first image generation unit 8a acquires the multiple pieces of viewpoint information output in S202. In addition, the second image generation unit 8b acquires the selected viewpoint information output in S204.

[0042] In S212, the image generating unit 8 acquires control information related to time synchronization from the time control unit 10. An example of control by the time control unit 10 will be described. The multiple viewpoint generating units 5 are operated independently. For this reason, the timing at which the viewpoint information acquiring unit 61 acquires multiple viewpoint information does not necessarily match, and the time information included in the multiple viewpoint information may be out of sync. Therefore, the time control unit 10 outputs a synchronization signal for synchronizing the time represented by the time information included in the multiple viewpoint information as control information to the image generating unit 8. The image generating unit 8 adjusts the time of the multiple viewpoint information based on the synchronization signal. This enables the image generating unit 8 to acquire multiple viewpoint information whose times are synchronized. Note that the method of controlling the time performed by the time control unit 10 is not limited to this, and for example, a signal specifying time information for acquiring data may be output to the image generating unit 8. The image generating unit 8 is able to acquire data corresponding to the time specified by the signal acquired from the time control unit 10 from the storage unit 4.

[0043] In S213, the image generating unit 8 acquires data used to generate a virtual viewpoint image from the storage unit 4 based on the time information included in the viewpoint information. The acquired data is, for example, a three-dimensional model of an object, a foreground image, a background model, a background texture, and the like. Here, since the time information included in the multiple viewpoint information is synchronized in S212, even when multiple virtual viewpoint images corresponding to the multiple viewpoint information are generated, it is sufficient to acquire data corresponding to one piece of time information. This makes it possible to reduce the processing load and data transmission amount related to data access when the image generating unit 8 reads data from the storage unit 4.

[0044] In S214, the image generating unit 8 generates a virtual viewpoint image based on the viewpoint information and the data acquired from the storage unit 4. An example of the generation of a virtual viewpoint image will be described. It is assumed that the image generating unit 8 acquires a three-dimensional model corresponding to certain time information and a foreground image for coloring the three-dimensional model from the storage unit 4. When there is a plurality of viewpoint information for which a virtual viewpoint image is to be generated, a plurality of virtual viewpoint images are generated by coloring the three-dimensional model based on the position of the virtual viewpoint and the line of sight from the virtual viewpoint represented by each viewpoint information.

[0045] In S214, the first image generating unit 8a generates a virtual viewpoint image of lower quality than the virtual viewpoint image generated by the second image generating unit 8b. As an example, when the number of acquired viewpoint information is four, the first image generating unit 8a generates four virtual viewpoint images of Half HD (960×540 pixels). When one of the four viewpoint information is selected, the second image generating unit 8b generates a virtual viewpoint image of Full HD (1920×1080 pixels).

[0046] As another example, when four pieces of viewpoint information are acquired, the first image generating unit 8a generates four virtual viewpoint images with a video frame rate of 30 FPS. When one piece of viewpoint information is selected from the four pieces of viewpoint information, the second image generating unit 8b generates one virtual viewpoint image with a video frame rate of 60 FPS.

[0047] In S215, the first image generating unit 8a outputs the generated virtual viewpoint image to the multiple image display unit 9a. The second image generating unit 8b outputs the generated virtual viewpoint image to the output image display unit 9b and the video distribution device 11. FIG. 3 shows an example of the display on the multiple image display unit 9a and the output image display unit 9b when there are four pieces of viewpoint information. As shown in FIG. 3, the multiple image display unit 9a simultaneously displays a plurality of virtual viewpoint images A to D corresponding to the plurality of pieces of viewpoint information. The output image display unit 9b displays a virtual viewpoint image A corresponding to the viewpoint information selected from the four pieces of viewpoint information. At this time, the virtual viewpoint image A displayed on the output image display unit 9b has a higher quality than the plurality of virtual viewpoint images A to D displayed on the multiple image display unit 9a.

[0048] A user operating the selection unit 7 operates a plurality of switches (switchers) as shown in Fig. 3 while viewing a plurality of virtual viewpoint images displayed on the multiple image display unit 9a to select a virtual viewpoint image to be output to the video distribution device 11. In addition, the user can check the virtual viewpoint image to be distributed in a high quality state by displaying the selected virtual viewpoint image on the output image display unit 9b.

[0049] Although the number of pieces of viewpoint information is four in the above-mentioned example, the number of pieces of viewpoint information is not limited to this. For example, in the case of 16 pieces of viewpoint information, the first image generating unit 8a generates 16 virtual viewpoint images with a resolution of 480×270 pixels, which is a lower resolution. This can reduce the processing load related to the rendering of the first image generating unit 8a and the processing load related to the display of the multiple image display unit 9a. In addition, although the number of pieces of viewpoint information selected by the selection unit 7 is one in the above-mentioned example, two or more pieces of viewpoint information may be selected.

[0050] In S216, the image generating unit 8 judges whether acquisition of the viewpoint information has been completed. If it is judged that acquisition has been completed, the process is terminated, and if not, the process of S211 to S216 is repeated.

[0051] According to the configuration described above, the information processing device 6 performs display control so that the virtual viewpoint image is displayed on the multiple image display unit 9a and the output image display unit 9b by outputting viewpoint information to the image generation unit 8. The user can easily confirm viewpoint information to be used for distribution from among multiple pieces of viewpoint information obtained by multiple operations while suppressing the processing load related to the display of the virtual viewpoint image.

[0052] (Modification of the first embodiment) In the first embodiment, an example in which a plurality of virtual viewpoint images are arranged in a tiled manner and displayed on the multiple image display unit 9a, which is one display device, has been described, but the present invention is not limited to this. For example, a configuration in which a plurality of virtual viewpoint images are displayed on different display devices may be used.

[0053] In the first embodiment, the viewpoint generating unit 5 is configured with an operation unit and a display unit and is operated by a user, but is not necessarily limited to this. The viewpoint generating unit 5 may be, for example, a device that automatically generates the position of a virtual viewpoint corresponding to each time and the line of sight direction from the virtual viewpoint. With this configuration, an operation unit where a user performs an input operation and a display unit that displays a virtual viewpoint image are not necessary.

[0054] In the first embodiment, the first image generating unit 8a and the second image generating unit 8b are different devices, but the present invention is not limited to this. For example, the first image generating unit 8a and the second image generating unit 8b may be the same device. In this case, the device is capable of switching the quality of the virtual viewpoint image to be generated depending on the timing of generating the virtual viewpoint image, such as the timing when viewpoint information is selected by the selection unit 7. In this configuration, while the device is functioning as the second image generating unit 8b, it is not possible to generate multiple virtual viewpoint images corresponding to multiple viewpoint information based on multiple operations. Instead, the multiple image display unit 9a may be configured to display the generated virtual viewpoint image.

[0055] In addition, in this embodiment, the viewpoint selection unit 6, the first image generation unit 8a, and the second image generation unit 8b are separate components, but this is not necessarily limited to this.

[0056] In the first embodiment, the time control unit 10 outputs control information to each of the first image generating unit 8a and the second image generating unit 8b, but the present invention is not limited to this. For example, as shown in FIG. 4, the time control unit 10 may output control information to the information processing device 6. In this case, the information processing device 6 outputs time information to the image generating unit 8 together with viewpoint information. With this configuration, when the configuration of the image generating unit 8, such as the number of image generating units 8, is changed, it is not necessary to change the connection destination of the time control unit 10. In addition, in this configuration, since the time information and viewpoint information are all collected once in the information processing device 6, this information may be recorded in a recording means inside or outside the information processing device 6. In this way, viewpoint information that was not selected during image distribution can also be recorded. In addition, a virtual viewpoint image can be generated based on this recorded information, and edited and used as a virtual viewpoint image of a different viewpoint. In addition, the time control unit 10 may be included inside the information processing device 6.

[0057] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions. [Explanation of symbols]

[0058] 6. Information processing equipment 61 Viewpoint information acquisition unit 63 Viewpoint information output unit

Claims

1. an acquisition means for acquiring a plurality of pieces of viewpoint information representing a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint, the position being used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images by a plurality of imaging devices; a first output means for outputting the plurality of pieces of viewpoint information to a first generation means for generating a plurality of virtual viewpoint images; a second output means for outputting viewpoint information selected from the plurality of pieces of viewpoint information to a second generation means for generating a virtual viewpoint image having an image size larger than each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to a virtual viewpoint image, The acquiring means acquires a plurality of viewpoint information including synchronized time information.

23. An information processing apparatus comprising:

2. 2. The information processing apparatus according to claim 1, wherein the first generating means and the second generating means are different devices.

3. 2. The information processing apparatus according to claim 1, wherein the first generating means and the second generating means are a single device.

4. An information processing device as described in any one of claims 1 to 3, characterized in that the selected viewpoint information is selected from the multiple viewpoint information based on a user's operation.

5. 5. The information processing apparatus according to claim 1, further comprising a control means for controlling so as to synchronize times represented by time information included in the plurality of pieces of viewpoint information acquired by the acquisition means.

6. 6. The information processing apparatus according to claim 1, wherein the plurality of virtual viewpoint images generated by the first generating means are simultaneously displayed on a display means.

7. 7. The information processing apparatus according to claim 1, wherein the virtual viewpoint image generated by the second generating means is a virtual viewpoint image for distribution.

8. An information processing device as described in any one of claims 1 to 7, characterized in that the multiple virtual viewpoint images generated by the first generation means and the virtual viewpoint image generated by the second generation means correspond to the same time.

9. An information processing system for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images using a plurality of imaging devices, a first display control means for controlling display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints having synchronized time information; an acquisition means for acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoints after displaying the plurality of virtual viewpoint images; a second display control means for controlling display of a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having an image size larger than each of the plurality of virtual viewpoint images; An information processing system comprising:

10. An acquisition means for acquiring a plurality of pieces of viewpoint information representing a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint, the viewpoint information being used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images by a plurality of imaging devices; a first output means for outputting the plurality of pieces of viewpoint information to a first generation means for generating a plurality of virtual viewpoint images; a second output means for outputting viewpoint information selected from the plurality of pieces of viewpoint information to a second generation means for generating a virtual viewpoint image having a frame rate higher than that of each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to a virtual viewpoint image, The acquiring means acquires a plurality of viewpoint information including synchronized time information.

23. An information processing apparatus comprising:

11. An information processing system for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images by a plurality of imaging devices, a first display control means for controlling display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints having synchronized time information; an acquisition means for acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoints after displaying the plurality of virtual viewpoint images; a second display control means for controlling display of a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having a frame rate higher than that of each of the plurality of virtual viewpoint images; An information processing system comprising:

12. An acquisition means for acquiring a plurality of pieces of viewpoint information representing a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint, the viewpoint information being used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images by a plurality of imaging devices; a first output means for outputting the plurality of pieces of viewpoint information to a first generation means for generating a plurality of virtual viewpoint images; a second output means for outputting viewpoint information selected from the plurality of pieces of viewpoint information to a second generation means for generating a virtual viewpoint image having a higher resolution than each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to a virtual viewpoint image, The acquiring means acquires a plurality of viewpoint information including synchronized time information.

23. An information processing apparatus comprising:

13. An information processing system for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images by a plurality of imaging devices, a first display control means for controlling display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints having synchronized time information; an acquisition means for acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoints after displaying the plurality of virtual viewpoint images; a second display control means for controlling display of a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having a higher resolution than each of the plurality of virtual viewpoint images; An information processing system comprising:

14. An acquisition step of acquiring a plurality of viewpoint information representing a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint, the viewpoint information being used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images using a plurality of imaging methods; a first output step of outputting the plurality of pieces of viewpoint information to a first generating means that generates a plurality of virtual viewpoint images; a second output step of outputting viewpoint information selected from the plurality of pieces of viewpoint information to a second generation means that generates a virtual viewpoint image having an image size larger than each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to a virtual viewpoint image, The acquiring step acquires a plurality of viewpoint information including synchronized time information.

23. An information processing method comprising:

15. An information processing method for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images using a plurality of imaging methods, comprising: a first display control step of controlling display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints having synchronized time information; an acquisition step of acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoints after displaying the plurality of virtual viewpoint images; a second display control step of controlling display of a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having an image size larger than each of the plurality of virtual viewpoint images; 13. An information processing method comprising:

16. An acquisition step of acquiring a plurality of pieces of viewpoint information representing a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint, the viewpoint information being used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images using a plurality of imaging methods; a first output step of outputting the plurality of pieces of viewpoint information to a first generating means that generates a plurality of virtual viewpoint images; a second output step of outputting viewpoint information selected from the plurality of pieces of viewpoint information to a second generation means that generates a virtual viewpoint image having a frame rate higher than that of each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to a virtual viewpoint image, The acquiring step acquires a plurality of viewpoint information including synchronized time information.

23. An information processing method comprising:

17. An information processing method for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by a plurality of imaging methods, comprising: a first display control step of controlling display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints having synchronized time information; an acquisition step of acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoints after displaying the plurality of virtual viewpoint images; a second display control step of controlling display of a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having a frame rate higher than that of each of the plurality of virtual viewpoint images; 13. An information processing method comprising:

18. An acquisition step of acquiring a plurality of viewpoint information representing a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint, the viewpoint information being used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images using a plurality of imaging methods; a first output step of outputting the plurality of pieces of viewpoint information to a first generating means that generates a plurality of virtual viewpoint images; a second output step of outputting viewpoint information selected from the plurality of pieces of viewpoint information to a second generation means that generates a virtual viewpoint image having a higher resolution than each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to a virtual viewpoint image, The acquiring step acquires a plurality of viewpoint information including synchronized time information.

23. An information processing method comprising:

19. An information processing method for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by a plurality of imaging methods, comprising: a first display control step of controlling display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints having synchronized time information; an acquisition step of acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoints after displaying the plurality of virtual viewpoint images; a second display control step of controlling display of a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having a higher resolution than each of the plurality of virtual viewpoint images; 13. An information processing method comprising:

20. A program for causing a computer to carry out the information processing method according to any one of claims 14 to 19.