Information processing device, information processing system, and program

Abstract

To reduce a processing load when selecting a specific virtual viewpoint from a plurality of virtual viewpoints, and appropriately create a virtual viewpoint image corresponding to the selected virtual viewpoint.SOLUTION: An information processing apparatus 6 has: a viewpoint information acquisition unit 61 that acquires viewpoint information representing the position of a virtual viewpoint used for creation of a virtual viewpoint image created on the basis of a plurality of picked-up images obtained by a plurality of imaging apparatuses picking up images, and a visual line direction from the virtual viewpoint; and a viewpoint information output unit 63 that outputs a plurality of pieces of viewpoint information acquired on the basis of a plurality of pieces of operation to first creation means that creates a plurality of virtual viewpoint images based on a condition related to first quality on the basis of the viewpoint information, and outputs one or more pieces of viewpoint information selected from the plurality of pieces of viewpoint information to second creation means that creates a virtual viewpoint image based on a condition related to second quality higher than the first quality on the basis of the viewpoint information.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a technology for generating a virtual viewpoint image. [Background technology]

[0002] In recent years, a technology has been attracting attention in which a plurality of imaging devices (cameras) are installed at different positions to capture images from different directions, and a virtual viewpoint image is generated using the captured images. The technology for generating a virtual viewpoint image allows a user to view an image as seen from a specified viewpoint (virtual viewpoint). The technology for generating a virtual viewpoint image in this manner allows a user to view, for example, highlight scenes of a soccer or basketball game from various angles, thereby providing a user with a higher sense of realism than with ordinary images. Patent Document 1 describes a technology for generating a virtual viewpoint image corresponding to a virtual viewpoint specified based on a user's operation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-050593 Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 does not take into consideration the case where multiple virtual viewpoints are specified by multiple operations. Furthermore, when multiple virtual viewpoints are specified and a specific virtual viewpoint is selected from among them, a process is required to generate multiple virtual viewpoint images and present them to the user, which increases the processing load associated with generating the virtual viewpoint images. Furthermore, there is a demand for the selected virtual viewpoint image to be generated with appropriate quality.

[0005] The present disclosure aims to reduce the processing load when selecting a specific virtual viewpoint from among multiple virtual viewpoints, while appropriately generating a virtual viewpoint image corresponding to the selected virtual viewpoint. [Means for solving the problem]

[0006] The information processing device according to the present disclosure is configured to generate a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices, and to generate viewpoint information representing a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint. multiple Acquisition method and multiple The aforementioned a first output means for outputting viewpoint information to a first generating means for generating a plurality of virtual viewpoint images; The aforementioned viewpoint information selected from the viewpoint information The image size is larger than each of the multiple virtual viewpoint images. a second output means for outputting the virtual viewpoint image to a second generation means for generating the virtual viewpoint image; 、 With The viewpoint information includes time information indicating a time related to the virtual viewpoint image, and the acquisition means acquires a plurality of viewpoint information including synchronized time information. [Effects of the Invention]

[0007] According to the present disclosure, a virtual viewpoint image corresponding to the selected virtual viewpoint is appropriately generated while suppressing the processing load when selecting a specific virtual viewpoint from among a plurality of virtual viewpoints. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system. [Figure 2] 10 is a flowchart illustrating a process related to display of a virtual viewpoint image. [Figure 3] 10A and 10B are diagrams showing examples of displays on a multiple image display unit and an output image display unit. [Figure 4] FIG. 10 is a diagram illustrating a configuration of an information processing system according to a modified example. [Figure 5] FIG. 2 is a diagram illustrating a hardware configuration of an information processing device. [Figure 6] FIG. 2 is a diagram illustrating a functional configuration of the information processing device. [Figure 7] FIG. 2 is a diagram illustrating an example of an arrangement of a plurality of imaging units. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are examples of embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure.

[0010] (First embodiment) The hardware configuration of the information processing device in this embodiment will be described below. Fig. 5 is a block diagram showing the hardware configuration of the information processing device 6. The information processing device 6 has a CPU 501, a RAM 502, a ROM 503, an operation unit 504, an output unit 505, an auxiliary storage device 506, an I / F 507, and a bus 508.

[0011] The CPU 501 controls the entire computer using computer programs and data stored in the RAM 502 and the ROM 503, and also executes each process performed by the information processing device 6 according to this embodiment. That is, the CPU 501 functions as each processing unit of the information processing device 6. Each processing unit will be described later.

[0012] The RAM 502 has an area for temporarily storing computer programs and data loaded from the auxiliary storage device 506, data acquired from the outside via an I / F (interface) 507, etc. The RAM 502 also has a work area used when the CPU 501 executes various processes. That is, the RAM 502 can be allocated as a frame memory, for example, or can provide various other areas as needed.

[0013] The ROM 503 stores setting data for the computer, a boot program, etc. The operation unit 504 is composed of a keyboard, a mouse, a joystick, a touch panel, etc., and can be operated by a user of the computer to input various instructions to the CPU 501. The output unit 505 displays the results of processing by the CPU 501. The output unit 505 is composed of, for example, a liquid crystal display, a touch panel, etc.

[0014] The auxiliary storage device 506 is a large-capacity information storage device, typified by a hard disk drive. The auxiliary storage device 506 stores an operating system (OS) and computer programs for causing the CPU 501 to implement the functions of each unit shown in FIG. 1. The auxiliary storage device 506 may also store image data to be processed. The computer programs and data stored in the auxiliary storage device 506 are loaded into the RAM 502 as appropriate under the control of the CPU 501, and become the objects of processing by the CPU 501.

[0015] The I / F 507 can connect to a network such as a LAN or the Internet, or to other devices such as a projector or display device, and the computer can acquire and send various information via this I / F 507. For example, if the information processing device 6 is connected to an external device via a wired connection, a communication cable is connected to the I / F 507. If the information processing device 6 has a function for wireless communication with an external device, the I / F 507 is equipped with an antenna. In this embodiment, multiple image capture devices are connected to the I / F 507, which acquires captured images and controls each image capture device. The bus 508 connects the above-mentioned components.

[0016] In this embodiment, the operation unit 504, the output unit 505, and the auxiliary storage device 506 are assumed to exist inside the information processing device 200, but this is not limiting. The information processing device 6 may be configured such that at least one of the operation unit 504, the output unit 505, and the auxiliary storage device 506 is connected to an external device of the information processing device 6. Note that although the above is a description of the hardware configuration of the information processing device 6, the hardware configuration shown in Fig. 5 can also be applied to other devices included in the information processing system 100, which will be described later.

[0017] Next, an example of the configuration of an image processing system for generating a virtual viewpoint image according to this embodiment is shown in FIG. 1. The information processing system 100 according to this embodiment includes a plurality of imaging units 1, a synchronization unit 2, a three-dimensional shape estimation unit 3, a storage unit 4, a viewpoint generation unit 5, an information processing device 6, a selection unit 7, an image generation unit 8, a display unit 9, a time control unit 10, and a video distribution device 11. In this embodiment, the generated virtual viewpoint image is a moving image, but is not limited to this. This embodiment can also be applied to a virtual viewpoint image of a still image. Each processing unit will be described below.

[0018] The imaging unit 1 is an imaging device such as a camera. The synchronization unit 2 is a device such as a time server, and outputs a synchronization signal to the multiple imaging units 1. The multiple imaging units 1 capture images of the imaging area in high-precision synchronization with each other based on the synchronization signal transmitted from the synchronization unit 2. The multiple imaging units 1 can be installed to surround the imaging area, for example, as shown in FIG. 7. Note that the arrangement and number of imaging units 1 are not limited to the example of FIG. 7. The imaging area can be outdoors, such as a sports stadium, field, or park, or indoors, such as a gymnasium, concert hall, stage, or studio. Each imaging unit 1 outputs the captured image to the three-dimensional shape estimation unit 3.

[0019] The three-dimensional shape estimation unit 3 generates a three-dimensional model representing the three-dimensional shape of the subject using multiple captured images acquired from multiple imaging units 1. An example of the process of generating a three-dimensional model by the three-dimensional shape estimation unit 3 is described below. The three-dimensional shape estimation unit 3 generates a foreground image and a background image using the multiple captured images acquired.

[0020] Here, a foreground image is an image in which an object region (foreground region) is extracted from an image captured by an imaging device. An object extracted as a foreground region is a dynamic object (moving body) that moves (its absolute position and shape can change) when images are captured from the same direction in chronological order. Examples of objects include people such as players and referees on the field where a sport is being played, a ball in a ball game, or singers, musicians, performers, and presenters in concerts and entertainment events. Furthermore, objects extracted as a foreground region can be, for example, items used by people.

[0021] Furthermore, a background image is an image of at least a region (background region) different from the foreground object. Specifically, a background image is an image in which the foreground object has been removed from the captured image. Furthermore, the background refers to an imaged object that remains stationary or nearly stationary when images are captured from the same direction in chronological order. Examples of such imaged objects include a stage for a concert, a stadium where an event such as a competition is held, a structure such as a goal used in a ball game, and a field. However, the background is at least a region different from the foreground object, and the imaged object may include other objects in addition to the object and background. Furthermore, three-dimensional model data is data that represents the above-mentioned object in a three-dimensional shape.

[0022] An example of a generation method by which the three-dimensional shape estimation unit 3 generates a foreground image and a background image will be described. The three-dimensional shape estimation unit 3 detects areas where pixel values do not change by comparing multiple captured images obtained by multiple imaging units 1 capturing images at consecutive times. The three-dimensional shape estimation unit 3 determines that the detected area is a background area, and generates a background image based on the detected area. The three-dimensional shape estimation unit 3 also compares the generated background image with the captured image, determines that an area where the difference in image values is equal to or greater than a predetermined threshold is a foreground area, and generates a foreground image based on that area.

[0023] Next, an example of a generation method in which the three-dimensional shape estimation unit 3 generates a three-dimensional model of an object corresponding to a foreground region will be described. Based on the foreground region and background region determined in the above-mentioned process, the three-dimensional shape estimation unit 3 generates a silhouette image of the object, expressing each region in binary. At this time, the three-dimensional shape estimation unit 3 generates silhouette images of the object viewed from multiple directions by using captured images captured from multiple directions. The three-dimensional shape estimation unit 3 generates a three-dimensional model using the multiple silhouette images, for example, by using a well-known technique called shape-from-silhouette.

[0024] In the above example, the three-dimensional shape estimation unit 3 generates a foreground image, a background image, and a silhouette image. However, the present invention is not limited to this. For example, the imaging unit 1 or another device connected to the imaging unit 1 may have a function to generate at least one of a foreground image, a background image, and a silhouette image. In this case, the imaging unit 1 or another device outputs the generated image to the three-dimensional shape estimation unit 3, and the three-dimensional shape estimation unit 3 generates a three-dimensional model using the acquired image. The three-dimensional shape estimation unit 3 outputs the generated three-dimensional model to the storage unit 4.

[0025] The storage unit 4 is, for example, a device such as a database server, and saves and stores the following data groups as data for generating virtual viewpoint images. Specifically, the three-dimensional model input from the three-dimensional shape estimation unit 3. Camera parameters representing the position, orientation, and optical characteristics of each imaging unit 1. The storage unit 4 may also record a background model and background texture image in advance in order to render the background of the virtual viewpoint image. The storage unit 4 may also acquire and store captured images from the imaging unit 1.

[0026] The viewpoint generators 5a to 5d each include an operation unit, which is an input device such as a joystick, and a display unit for displaying a virtual viewpoint image corresponding to the viewpoint being operated. Based on input from the operation unit, the viewpoint generators 5a to 5d generate viewpoint information representing the position of the virtual viewpoint and the line of sight from the virtual viewpoint, and input the generated viewpoint information to the information processing device 6. Note that the viewpoint information may include information corresponding to internal parameters of the camera, such as focal length and angle of view, in addition to the information representing the position of the virtual viewpoint and the line of sight from the virtual viewpoint. In the example of FIG. 4, the viewpoint generators 5a to 5d are four in number, but the number may be three or less or five or more. In the following description, unless there is a particular need to distinguish between them, they will be referred to as the viewpoint generation unit 5.

[0027] The information processing device 6 acquires viewpoint information from the viewpoint generation unit 5 and outputs it to a first image generation unit 8a and a second image generation unit 8b, which will be described later. At this time, the information processing device 6 also receives an instruction to select viewpoint information from a selection unit 7, which will be described later. The configuration of the information processing device 6 will be described later.

[0028] The selection unit 7 is an input device such as a mouse, a keyboard, or a switch, and transmits to the information processing device 6 an instruction to select one or more pieces of viewpoint information from the viewpoint information output by the viewpoint generation unit 5.

[0029] The first image generation unit 8a and the second image generation unit 8b are devices such as rendering servers, and generate virtual viewpoint images based on viewpoint information acquired from the information processing device 6. When generating the virtual viewpoint images, the first image generation unit 8a and the second image generation unit 8b acquire data necessary for generating the virtual viewpoint images from the storage unit 4 based on time information included in the viewpoint information. Here, the time information is information related to the time associated with the virtual viewpoint image corresponding to the viewpoint information, and specifically, may be a time code indicating the time at which the image capture unit 1 captured the image. In this embodiment, similar time information is also assigned to data such as the three-dimensional model and foreground image stored in the storage unit 4, and by specifying the time information, necessary data can be acquired from the storage unit 4. Note that the time information is not limited to the image capture time of the image capture unit 1, and may be expressed, for example, as a relative time from a predetermined time or a frame number of a video.

[0030] The first image generation unit 8a and the second image generation unit 8b perform rendering processing using a three-dimensional model and a foreground image to generate a virtual viewpoint image of the object that represents how it appears from the virtual viewpoint. The first image generation unit 8a and the second image generation unit 8b also generate a background image using a background model and a background texture, and synthesize the background image with the virtual viewpoint image of the object to generate a virtual viewpoint image with a background.

[0031] The first image generation unit 8a generates a plurality of virtual viewpoint images based on predetermined quality conditions, based on a plurality of viewpoint information generated by operating the plurality of viewpoint generation unit 5. Here, the conditions related to the quality of the virtual viewpoint images in this embodiment are conditions related to the resolution of the generated images and the frame rate of the video. The higher the resolution and the higher the frame rate, the higher the quality of the virtual viewpoint images. The first image generation unit 8a outputs the generated virtual viewpoint images to the plurality of image display unit 9a. Note that the first image generation unit 8a and the second image generation unit 8b in this embodiment are different devices.

[0032] The second image generation unit 8b acquires one or more pieces of viewpoint information selected by the selection unit 7 from among the plurality of pieces of viewpoint information, and generates a virtual viewpoint image. At this time, the generated virtual viewpoint image has higher quality than the virtual viewpoint image generated by the first image generation unit 8a. The second image generation unit 8b outputs the generated virtual viewpoint images to the output image display unit 9b and the video distribution device 11. That is, the first image generation unit 8a generates a plurality of virtual viewpoint images so as to satisfy the conditions for generating a virtual viewpoint image having at least one of a predetermined resolution and a predetermined frame rate. Moreover, the second image generation unit 8b generates a virtual viewpoint image so as to satisfy the conditions for generating a virtual viewpoint image having at least one of a resolution higher than the predetermined resolution and a frame rate higher than the predetermined frame rate.

[0033] The multiple image display unit 9a and the output image display unit 9b display the virtual viewpoint images acquired from the first image generation unit 8a and the second image generation unit 8b, respectively. The time control unit 10 controls the first image generation unit 8a and the second image generation unit 8b to synchronize the time information used when generating the virtual viewpoint images. The video distribution device 11 distributes the virtual viewpoint images transmitted from the second image generation unit 8b, for example, via television broadcasting or the Internet.

[0034] Next, the functional configuration of the information processing device 6 will be described with reference to Fig. 6. The information processing device 6 has a viewpoint information acquisition unit 61, a reservation acceptance unit 62, and a viewpoint information output unit 63. The viewpoint information acquisition unit 61 acquires viewpoint information transmitted from the viewpoint generation unit 5 and outputs it to the viewpoint information output unit 63. If there are multiple viewpoint generation units 5, the viewpoint information acquisition unit 61 acquires multiple pieces of viewpoint information and outputs it to the viewpoint information output unit 63. The selection acceptance unit 62 accepts a selection instruction for viewpoint information from the selection unit 7, and outputs information representing the accepted instruction to the viewpoint information output unit 63.

[0035] The viewpoint information output unit 63 outputs viewpoint information to the first image generation unit 8a and the second image generation unit 8b based on the acquired viewpoint information and selection instruction information. At this time, the viewpoint information output unit 63 outputs a plurality of viewpoint information to the first image generation unit 8a, and outputs one or more viewpoint information selected from the plurality of viewpoint information based on the selection instruction information to the second image generation unit 8b. In this way, the viewpoint information output unit 63 outputs viewpoint information to the first image generation unit 8a and the second image generation unit 8b, and controls so that the virtual viewpoint images generated by each processing unit are displayed on the multiple image display unit 9a and the output image display unit 9b.

[0036] Next, the processing performed by the information processing system 100 in this embodiment will be described with reference to FIG. 2. FIG. 2(a) is a flowchart of the processing performed by the information processing device 6. The processing of FIG. 2(a) is performed by the CPU 501 of the information processing device 6 by reading and executing a program stored in the RAM 502 and ROM 503. In the following description, processing steps will be simply referred to as S. The processing starts when viewpoint information is input from the viewpoint generation unit 5. Furthermore, the processing of S201 to S205 is repeatedly executed for each video frame of the virtual viewpoint image to be generated. That is, if the video to be generated is 60 FPS (frames per second), the processing of S201 to S205 is repeated every 1 / 60 seconds.

[0037] In S201, the viewpoint information acquisition unit 61 acquires a plurality of pieces of viewpoint information generated by operating the operation units of the plurality of viewpoint generation units 5. In S202, the viewpoint information output unit 63 outputs the acquired plurality of pieces of viewpoint information to the first image generation unit 8a.

[0038] In S203, the selection receiving unit 62 determines whether a new instruction to select viewpoint information has been received from the selection unit 7. For example, if a selection instruction has been received for the first time, or if a selection instruction to select viewpoint information different from the viewpoint information selected in S204 the previous time in the repeated processing of S201 to S205 has been received, the processing proceeds to S204. On the other hand, if a new selection instruction has not been received, or if there is no change from the viewpoint information selected in S204 the previous time in the repeated processing of S201 to S205, the processing proceeds to S205.

[0039] In S204, the viewpoint information output unit 63 outputs to the second image generation unit 8b one or more pieces of viewpoint information selected by the selection instruction received by the selection instruction receiving unit 62 from the plurality of pieces of viewpoint information received by the viewpoint information receiving unit 61. In S205, the viewpoint information receiving unit 61 determines whether the acquisition of viewpoint information has ended, for example, based on an end instruction from the viewpoint generation unit 5. If it is determined that the acquisition has ended, the processing ends, and if not, the processing of S201 to S205 is repeated.

[0040] Next, the processing performed by the first image generation unit 8a and the second image generation unit 8b will be explained using FIG. 2(b). The processing of FIG. 2(b) is performed by the CPUs of the first image generation unit 8a and the second image generation unit 8b reading and executing programs stored in the RAM and ROM. Note that the first image generation unit 8a and the second image generation unit 8b each independently perform the processing shown in FIG. 2(b). In the following explanation, unless otherwise specified, the first image generation unit 8a and the second image generation unit 8b will be simply referred to as the image generation unit 8. When the image generation unit 8 acquires viewpoint information, processing begins.

[0041] In S211, the image generation unit 8 acquires viewpoint information from the information processing device 6. At this time, the first image generation unit 8a acquires the plurality of viewpoint information output in S202. In addition, the second image generation unit 8b acquires the selected viewpoint information output in S204.

[0042] In S212, the image generation unit 8 acquires control information related to time synchronization from the time control unit 10. An example of control by the time control unit 10 will be described. The multiple viewpoint generation units 5 are each operated independently. Therefore, the viewpoint information acquisition unit 61 does not necessarily acquire multiple viewpoint information at the same time, and the time information included in the multiple viewpoint information may be out of sync. Therefore, the time control unit 10 outputs a synchronization signal to the image generation unit 8 as control information for synchronizing the times represented by the time information included in the multiple viewpoint information. The image generation unit 8 adjusts the time of the multiple viewpoint information based on the synchronization signal. This allows the image generation unit 8 to acquire multiple viewpoint information whose times are synchronized. Note that the time control method performed by the time control unit 10 is not limited to this. For example, the time control unit 10 may output a signal specifying time information for acquiring data to the image generation unit 8. The image generation unit 8 can acquire data corresponding to the time specified by the signal acquired from the time control unit 10 from the storage unit 4.

[0043] In S213, the image generation unit 8 acquires data to be used for generating a virtual viewpoint image from the storage unit 4 based on the time information included in the viewpoint information. The acquired data includes, for example, a three-dimensional model of an object, a foreground image, a background model, and a background texture. Here, since the time information included in the plurality of viewpoint information is synchronized in S212, even when generating a plurality of virtual viewpoint images corresponding to a plurality of viewpoint information, it is only necessary to acquire data corresponding to one piece of time information. This makes it possible to reduce the processing load and data transmission volume associated with data access when the image generation unit 8 reads data from the storage unit 4.

[0044] In S214, the image generation unit 8 generates a virtual viewpoint image based on the viewpoint information and the data acquired from the storage unit 4. An example of the generation of a virtual viewpoint image will be described. It is assumed that the image generation unit 8 acquires a three-dimensional model corresponding to certain time information and a foreground image for coloring the three-dimensional model from the storage unit 4. When there is a plurality of viewpoint information for which a virtual viewpoint image is to be generated, a plurality of virtual viewpoint images are generated by coloring the three-dimensional model based on the position of the virtual viewpoint and the line of sight from the virtual viewpoint represented by each viewpoint information.

[0045] In S214, the first image generation unit 8a generates a virtual viewpoint image of lower quality than the virtual viewpoint image generated by the second image generation unit 8b. As an example, when four pieces of viewpoint information are acquired, the first image generation unit 8a generates four virtual viewpoint images of Half HD (960 x 540 pixels). When one piece of viewpoint information is selected from the four pieces of viewpoint information, the second image generation unit 8b generates a virtual viewpoint image of Full HD (1920 x 1080 pixels).

[0046] As another example, when four pieces of viewpoint information are acquired, the first image generation unit 8a generates four virtual viewpoint images with a video frame rate of 30 FPS. When one piece of viewpoint information is selected from the four pieces of viewpoint information, the second image generation unit 8b generates one virtual viewpoint image with a video frame rate of 60 FPS.

[0047] In S215, the first image generation unit 8a outputs the generated virtual viewpoint image to the multiple image display unit 9a. Furthermore, the second image generation unit 8b outputs the generated virtual viewpoint image to the output image display unit 9b and the video distribution device 11. FIG. 3 shows an example of the display on the multiple image display unit 9a and the output image display unit 9b when there are four pieces of viewpoint information. As shown in FIG. 3, the multiple image display unit 9a simultaneously displays multiple virtual viewpoint images A to D corresponding to the multiple pieces of viewpoint information. Furthermore, the output image display unit 9b displays virtual viewpoint image A corresponding to selected one of the four pieces of viewpoint information. At this time, the virtual viewpoint image A displayed on the output image display unit 9b has higher quality than the multiple virtual viewpoint images A to D displayed on the multiple image display unit 9a.

[0048] A user operating the selection unit 7 operates a plurality of switches (switchers) as shown in Fig. 3 while viewing a plurality of virtual viewpoint images displayed on the multiple image display unit 9a to select a virtual viewpoint image to be output to the video distribution device 11. In addition, the selected virtual viewpoint image is displayed on the output image display unit 9b, allowing the user to check the distributed virtual viewpoint image in a high-quality state.

[0049] Although the number of pieces of viewpoint information is four in the above-described example, the number of pieces of viewpoint information is not limited to this. For example, if there are 16 pieces of viewpoint information, the first image generation unit 8a generates 16 virtual viewpoint images with a lower resolution of 480 × 270 pixels. This reduces the processing load related to rendering on the first image generation unit 8a and the processing load related to display on the multiple image display unit 9a. Furthermore, although one piece of viewpoint information is selected by the selection unit 7 in the above-described example, two or more pieces of viewpoint information may be selected.

[0050] In S216, the image generating unit 8 determines whether acquisition of viewpoint information has been completed. If it is determined that acquisition has been completed, the process ends, and if not, the processes of S211 to S216 are repeated.

[0051] According to the configuration described above, the information processing device 6 outputs viewpoint information to the image generation unit 8, thereby performing display control to display virtual viewpoint images on the multiple image display unit 9a and the output image display unit 9b. The user can easily confirm viewpoint information to be used for distribution from among multiple pieces of viewpoint information obtained by multiple operations, while suppressing the processing load related to the display of virtual viewpoint images.

[0052] (Modification of the first embodiment) In the first embodiment, an example has been described in which a plurality of virtual viewpoint images are arranged in a tiled pattern and displayed on the multiple image display unit 9a, which is a single display device, but the present invention is not limited to this. For example, a configuration in which a plurality of virtual viewpoint images are displayed on different display devices may also be used.

[0053] In the first embodiment, the viewpoint generation unit 5 is configured with an operation unit and a display unit and is operated by a user, but this is not necessarily limited to this. The viewpoint generation unit 5 may be, for example, a device that automatically generates the position of a virtual viewpoint corresponding to each time and the line of sight direction from the virtual viewpoint. With this configuration, an operation unit where a user performs input operations and a display unit that displays a virtual viewpoint image are not required.

[0054] In the first embodiment, the first image generation unit 8a and the second image generation unit 8b are different devices, but this is not limiting. For example, the first image generation unit 8a and the second image generation unit 8b may be the same device. In this case, this device is capable of switching the quality of the virtual viewpoint image to be generated depending on the timing of generating the virtual viewpoint image, such as the timing when viewpoint information is selected by the selection unit 7. Note that in this configuration, while the device is functioning as the second image generation unit 8b, it is not possible to generate multiple virtual viewpoint images corresponding to multiple viewpoint information based on multiple operations. Instead, the generated virtual viewpoint image may be displayed on the multiple image display unit 9a.

[0055] In addition, in this embodiment, the viewpoint selection unit 6, the first image generation unit 8a, and the second image generation unit 8b are separate components, but this is not necessarily limited to this.

[0056] In the first embodiment, the time control unit 10 outputs control information to each of the first image generation unit 8a and the second image generation unit 8b. However, this is not limiting. For example, as shown in FIG. 4, the time control unit 10 may output control information to the information processing device 6. In this case, the information processing device 6 outputs time information to the image generation unit 8 together with viewpoint information. This configuration eliminates the need to change the connection destination of the time control unit 10 when the configuration of the image generation unit 8, such as the number of image generation units 8, changes. In this configuration, since the time information and viewpoint information are all collected in the information processing device 6, this information may be recorded in a recording device inside or outside the information processing device 6. This allows viewpoint information that was not selected during image distribution to be recorded. Furthermore, a virtual viewpoint image can be generated based on this recorded information and edited and used as a virtual viewpoint image of a different viewpoint. The time control unit 10 may also be included within the information processing device 6.

[0057] (Other embodiments) The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0058] 6. Information processing equipment 61 Viewpoint information acquisition unit 63 Viewpoint information output unit

Claims

1. an acquisition means for acquiring a plurality of pieces of viewpoint information representing a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint, which are used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images by a plurality of imaging devices; a first output means for outputting the plurality of pieces of viewpoint information to a first generation means for generating a plurality of virtual viewpoint images; a second output means for outputting viewpoint information selected from the plurality of pieces of viewpoint information to a second generation means for generating a virtual viewpoint image having an image size larger than each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to the virtual viewpoint image, The acquisition means acquires a plurality of viewpoint information pieces including synchronized time information pieces.

1. An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the first generating means and the second generating means are different devices.

3. 2. The information processing apparatus according to claim 1, wherein the first generating means and the second generating means are the same device.

4. An information processing device described in any one of claims 1 to 3, characterized in that the selected viewpoint information is selected from the plurality of viewpoint information based on an operation by a user.

5. 5. The information processing apparatus according to claim 1, further comprising a control unit that controls the synchronization of times represented by time information included in the plurality of viewpoint information acquired by the acquisition unit.

6. 6. The information processing apparatus according to claim 1, wherein the plurality of virtual viewpoint images generated by the first generating means are simultaneously displayed on a display means.

7. 7. The information processing apparatus according to claim 1, wherein the virtual viewpoint image generated by the second generating means is a virtual viewpoint image for distribution.

8. An information processing device described in any one of claims 1 to 7, characterized in that the multiple virtual viewpoint images generated by the first generation means and the virtual viewpoint images generated by the second generation means correspond to the same time.

9. An information processing system for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images using a plurality of imaging devices, a first display control means for controlling the display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints whose time information is synchronized; an acquisition means for acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoint images after displaying the plurality of virtual viewpoint images; a second display control means for controlling the display of a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having an image size larger than that of each of the plurality of virtual viewpoint images; An information processing system comprising:

10. An acquisition means for acquiring a plurality of pieces of viewpoint information representing the position of a virtual viewpoint and the line of sight direction from the virtual viewpoint, which are used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images from a plurality of imaging devices; a first output means for outputting the plurality of pieces of viewpoint information to a first generation means for generating a plurality of virtual viewpoint images; a second output means for outputting viewpoint information selected from the plurality of pieces of viewpoint information to a second generation means for generating a virtual viewpoint image having a frame rate higher than that of each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to the virtual viewpoint image, The acquisition means acquires a plurality of viewpoint information including synchronized time information.

1. An information processing device comprising:

11. An information processing system for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images using a plurality of imaging devices, a first display control means for controlling the display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints whose time information is synchronized; an acquisition means for acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoint images after displaying the plurality of virtual viewpoint images; a second display control means for controlling display of a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having a frame rate higher than that of each of the plurality of virtual viewpoint images; An information processing system comprising:

12. An acquisition means for acquiring a plurality of pieces of viewpoint information representing the position of a virtual viewpoint and the line of sight direction from the virtual viewpoint, which are used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images from a plurality of imaging devices; a first output means for outputting the plurality of pieces of viewpoint information to a first generation means for generating a plurality of virtual viewpoint images; a second output means for outputting viewpoint information selected from the plurality of pieces of viewpoint information to a second generation means for generating a virtual viewpoint image having a higher resolution than each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to the virtual viewpoint image, The acquisition means acquires a plurality of viewpoint information pieces including synchronized time information pieces.

1. An information processing device comprising:

13. An information processing system for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images using a plurality of imaging devices, a first display control means for controlling the display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints whose time information is synchronized; an acquisition means for acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoint images after displaying the plurality of virtual viewpoint images; a second display control means for controlling the display of a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having a higher resolution than each of the plurality of virtual viewpoint images; An information processing system comprising:

14. An acquisition step of acquiring a plurality of pieces of viewpoint information representing the position of a virtual viewpoint and the line of sight direction from the virtual viewpoint, which are used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images using a plurality of imaging methods; a first output step of outputting the plurality of pieces of viewpoint information to a first generation means that generates a plurality of virtual viewpoint images; a second output step of outputting viewpoint information selected from the plurality of pieces of viewpoint information to second generation means that generates a virtual viewpoint image having an image size larger than each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to the virtual viewpoint image, The acquiring step acquires a plurality of viewpoint information including synchronized time information.

1. An information processing method comprising:

15. An information processing method for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by a plurality of imaging methods, comprising: a first display control step of controlling the display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints whose time information is synchronized; an acquisition step of acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoints after displaying the plurality of virtual viewpoint images; a second display control step of controlling the display of a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having an image size larger than each of the plurality of virtual viewpoint images; An information processing method comprising:

16. An acquisition step of acquiring a plurality of pieces of viewpoint information representing the position of a virtual viewpoint and the line of sight direction from the virtual viewpoint, which are used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images using a plurality of imaging methods; a first output step of outputting the plurality of pieces of viewpoint information to a first generation means that generates a plurality of virtual viewpoint images; a second output step of outputting viewpoint information selected from the plurality of pieces of viewpoint information to a second generation means that generates a virtual viewpoint image having a frame rate higher than that of each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to the virtual viewpoint image, The acquiring step acquires a plurality of viewpoint information including synchronized time information.

1. An information processing method comprising:

17. An information processing method for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by a plurality of imaging methods, comprising: a first display control step of controlling the display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints whose time information is synchronized; an acquisition step of acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoints after displaying the plurality of virtual viewpoint images; a second display control step of controlling the display of a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having a frame rate higher than that of each of the plurality of virtual viewpoint images; An information processing method comprising:

18. An acquisition process for acquiring a plurality of pieces of viewpoint information representing the position of a virtual viewpoint and the line of sight direction from the virtual viewpoint, which are used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing images using a plurality of imaging methods; a first output step of outputting the plurality of pieces of viewpoint information to a first generation means that generates a plurality of virtual viewpoint images; a second output step of outputting viewpoint information selected from the plurality of pieces of viewpoint information to a second generation means that generates a virtual viewpoint image having a higher resolution than each of the plurality of virtual viewpoint images, the viewpoint information includes time information indicating a time related to the virtual viewpoint image, The acquiring step acquires a plurality of viewpoint information including synchronized time information.

1. An information processing method comprising:

19. An information processing method for displaying a virtual viewpoint image generated based on a plurality of captured images obtained by a plurality of imaging methods, comprising: a first display control step of controlling the display of a plurality of virtual viewpoint images generated based on a plurality of virtual viewpoints whose time information is synchronized; an acquisition step of acquiring an input corresponding to a user operation of selecting a virtual viewpoint from the plurality of virtual viewpoints after displaying the plurality of virtual viewpoint images; a second display control step of performing control to display a virtual viewpoint image based on the selected virtual viewpoint, the virtual viewpoint image having a higher resolution than each of the plurality of virtual viewpoint images; An information processing method comprising:

20. A program for causing a computer to carry out the information processing method according to any one of claims 14 to 19.

Images