Information processing device, control method for information processing device, and program

Abstract

To provide a technique which enables rendering where camera switching is inconspicuous for referring to a region where the camera is not familiar with without switching the camera rendering method.SOLUTION: An information processing device edits a database containing arrangement information of a plurality of photographing devices for imaging a subject, and photographed images, and for generating a virtual viewpoint image. The information processing device includes: determination means for determining the arrangement of virtual photographing devices to be arranged in a virtual space, on the basis of arrangement information of the plurality of photographing devices contained in the database; generation means for generating virtual photographed images of the virtual photographing devices in the arrangement; and editing means for editing the database by registering the arrangement information of the virtual photographing devices, and the virtual photographed images in the database.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] A technology that uses multiple cameras installed at different positions to synchronously shoot multiple viewpoints and generate a virtual viewpoint image using the multiple viewpoint images obtained by the shooting has been attracting attention. The technology that generates a virtual viewpoint image from multiple viewpoint images allows video creators to include viewpoints that were not previously included by cameras in the video, and therefore allows them to create content with powerful viewpoints.

[0003] Such a virtual viewpoint image can be generated by reconstructing an imaging space based on a group of captured images and capturing (i.e., rendering) this imaging space with a virtual camera. In Non-Patent Document 1, viewpoint-dependent rendering is performed to generate a high-definition virtual viewpoint image by projecting an image captured by a capture device close to the virtual viewpoint onto the virtual viewpoint image in order to color the virtual viewpoint image.

[0004] Viewpoint-dependent rendering techniques generate virtual viewpoint images by projecting images from a camera close to the virtual viewpoint, and while this technique can generate high-definition images, when the number of cameras used to capture the images is sparse, the switching between cameras responsible for coloring can be noticeable.

[0005] In response to this, Patent Document 1 discloses a technique for performing rendering by switching the rendering method when the cameras are sparsely arranged. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2018-92580 A [Non-patent literature]

[0007] [Non-Patent Document 1] W. Matusik et al. "Polyhedral Visual Hulls for Real-Time Rendering", Proceedings of Eurographics Workshop on Rendering 2001, pp.115-125, 2001. Summary of the Invention [Problem to be solved by the invention]

[0008] However, the technique described in Patent Document 1 has a problem in that it is necessary to store a plurality of rendering methods and select one at the time of rendering.

[0009] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that realizes rendering in which the switching of reference cameras is not noticeable even in areas where cameras are sparse, without switching rendering methods. [Means for solving the problem]

[0010] To achieve the above object, an information processing device according to one aspect of the present invention comprises: An information processing device that edits a database for generating a virtual viewpoint image, the database including arrangement information and photographed images of a plurality of photographing devices that photograph a subject, a determination means for determining an arrangement of virtual camera devices to be placed in a virtual space based on the arrangement information of the plurality of camera devices contained in the database; a generating means for generating a virtual photographed image of the virtual photographing device in the arrangement; an editing means for registering the arrangement information of the virtual photography device and the virtual photographed image in the database and editing the database; The present invention is characterized by comprising: Effect of the Invention

[0011] According to the present invention, it is possible to realize viewpoint-dependent rendering in which switching of the reference camera is not noticeable, without switching the rendering method. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an example of a schematic configuration of a virtual visual point image generating system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the hardware configuration of the information processing device according to the first embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of the functional arrangement of the information processing device according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of a flowchart illustrating a procedure of processing executed by the information processing device according to the first embodiment. [Diagram 5] FIG. 2A is a flowchart showing a procedure for processing for determining the placement of a virtual camera according to the first embodiment, and FIGS. 2B and 2C are explanatory views of the processing for determining the placement of a virtual camera according to the first embodiment. [Figure 6] FIG. 4 is a diagram for explaining the effect of the first embodiment. [Figure 7] 10 is a flowchart for explaining a flow of determining the angle of view of the virtual camera according to the second embodiment. [Figure 8] FIG. 11 is a diagram showing an example of the functional configuration of an information processing device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] (Embodiment 1) In this embodiment, in a database for generating a virtual viewpoint image acquired using a virtual viewpoint image generation system, a virtual camera is added to a location where a camera is insufficient, and its arrangement information and data of the virtual photographed image are added to the database. Specifically, a virtual camera for texture attachment (coloring) is added to a location where cameras are sparse, and the virtual photographed image is generated by model-based rendering.

[0015] In this embodiment, the virtual viewpoint image is an image at an arbitrary virtual viewpoint, and the virtual photographed image is an image photographed by a virtual photographing device for texture (coloring) to be registered in the database. The virtual viewpoint image is generated based on photographed images of a plurality of photographing devices and / or virtual photographed images of one or more virtual photographing devices added (to the virtual space).

[0016] <System configuration> 1 shows an example of the configuration of a virtual viewpoint image generation system 1 for generating a virtual viewpoint image according to this embodiment. In the virtual viewpoint image generation system 1, a subject 3 in a target area 2 is photographed by a plurality of image capturing devices 10. A virtual viewpoint image is generated based on a plurality of captured images obtained by the plurality of image capturing devices 10. The plurality of image capturing devices 10 are arranged to surround the subject 3, for example, as shown in FIG. 1, and each captures the target area 2 from a different photographing position.

[0017] The modeling device 20 generates three-dimensional shape information indicating the three-dimensional shape of the subject 3 using a plurality of captured images obtained by the photographing device 10, and stores the information in the database 30. In addition to the three-dimensional shape information generated by the modeling device 20, the database 30 also stores arrangement information (position information, angle of view information, etc.) of the photographing device 10 and information required for rendering the photographed images, etc.

[0018] The rendering device 40 uses data stored in the database 30 to render a virtual viewpoint image based on virtual viewpoint information input from the input device 50. The rendering device 40 performs rendering using a viewpoint-dependent rendering method that mainly uses an image captured by the closest camera based on the input virtual viewpoint position. The input device 50 is a device that inputs the position and orientation of the virtual viewpoint, and supplies information on the virtual viewpoint to be rendered to the rendering device 40. The display device 60 displays the virtual viewpoint image generated by the rendering device 40.

[0019] Viewpoint-dependent rendering is a method of generating color information of a subject model using a virtual viewpoint for rendering. For example, color information is determined based on an image captured by a camera with a line of sight close to that of the virtual viewpoint. For areas that are invisible from a camera with a line of sight closest to the virtual viewpoint, color information is determined based on an image captured by a camera with the next closest line of sight. In this case, color information may be determined from one camera, or information from one camera may be combined using some weighting. The camera used to determine color information is selected depending on the position and orientation of the virtual viewpoint, so the color information of the subject model changes when the virtual viewpoint moves. For this reason, this method is called viewpoint-dependent rendering.

[0020] The information processing device 100 according to the present embodiment edits a database for generating a virtual viewpoint image, including the arrangement information of a plurality of image capturing devices that capture an object and the captured images. The information processing device 100 acquires data stored in the database 30, edits the data by adding a virtual image capturing device to a portion (area) where the camera switching is noticeable during rendering, and writes the data back to the database 30.

[0021] The multiple image capturing devices 10 can continuously capture images in a synchronized manner. In this case, the virtual viewpoint image generating system 1 can generate three-dimensional shape information of a subject in time series, and can further generate virtual viewpoint images in time series, i.e., virtual viewpoint videos. In this configuration, the input device 50 can specify a virtual viewpoint whose position changes in time series. The virtual viewpoint that moves in this way is also called a virtual camera path or camera work.

[0022] <Hardware configuration> 2 is a block diagram showing an example of the hardware configuration of a computer applicable to the information processing device 100 according to this embodiment. The information processing device 100 has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a communication I / F 205, and a bus 206. The modeling device 20 and the rendering device 40 can also be realized using similar hardware. The rendering device 40 may be composed of a plurality of information processing devices connected via a network, for example.

[0023] The CPU 201 realizes the functions of each processing unit of the information processing device 100 by controlling the entire information processing device 100 using computer programs or data stored in the ROM 202 or the RAM 203. The information processing device 100 may have one or more dedicated hardware pieces different from the CPU 201, in which case the dedicated hardware pieces can execute at least a part of the processing by the CPU 201. Examples of the dedicated hardware pieces include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0024] The ROM 202 is a memory that stores programs that do not require modification. The RAM 203 is a memory that temporarily stores programs or data supplied from the auxiliary storage device 204, and data supplied from the outside via the communication I / F 205. The auxiliary storage device 204 is composed of a storage such as a hard disk drive, and stores various data such as image data or audio data. The communication I / F 205 is used for communication with an external device of the information processing device 100. For example, when the information processing device 100 is connected to an external device by wire, a communication cable is connected to the communication I / F 205. When the information processing device 100 communicates wirelessly with an external device, the communication I / F 205 includes an antenna. The bus 206 connects each part of the information processing device 100 to transmit information.

[0025] <Functional configuration> 3 shows an example of the configuration of the information processing device 100 according to this embodiment. The information processing device 100 includes an acquisition unit 310, a layout determination unit 320, a virtual captured image generation unit 330, and an editing unit 340.

[0026] The acquisition unit 310 acquires arrangement information (position information and angle of view information) of the multiple image capture devices 10, captured images, and three-dimensional shape information of the subject 3 from the database 30 of the virtual viewpoint image generation system 1. Here, the angle of view information includes internal parameters including the focal length of the image capture device and external parameters indicating the position and orientation, and can indicate the angle of view or the capture range of the image capture device. The captured image may be an image of the capture target area 2 captured by the image capture device 10, or may be an image of only the subject 3 extracted.

[0027] The placement determination unit 320 determines the placement of a virtual camera that captures the subject 3 based on the placement information of the multiple camera devices acquired by the acquisition unit 310. The virtual camera is a virtual camera that behaves as if it virtually captured the subject 3, unlike the camera device 10 that actually captured the subject 3. In this embodiment, the placement of one or more virtual camera devices is determined so that a virtual camera device is added to an area where the camera devices 10 are sparse. An area where the camera devices 10 are sparse is a location (area) where switching of the reference camera is noticeable in viewpoint-dependent rendering because the appearances of the camera devices are different. The placement determination unit 320 supplies the determined placement information to the virtual captured image generation unit 330.

[0028] The virtual captured image generation unit 330 receives the placement information of the virtual camera from the placement determination unit 320, and generates a virtual captured image captured in that placement. Here, by generating the virtual captured image by a method different from the viewpoint-dependent rendering of the rendering unit 40, an image is generated that is not affected by switching of the reference camera. The virtual captured image generation unit 330 supplies the generated virtual captured image and the placement information of the virtual camera received from the placement determination unit 320 to the editing unit 340.

[0029] The editing unit 340 receives the virtual photographing device arrangement information and the virtual photographed image supplied from the virtual photographed image generating unit 330 , and registers them in the database 30 , thereby editing the database 30 .

[0030] <Processing> Fig. 4 is a flowchart showing the procedure of processing performed by the information processing device 100 according to this embodiment. The database 30 is edited by the processing according to Fig. 4. The processing shown in Fig. 4 can be realized by the CPU 201 executing a control program stored in the ROM 202 and read into the RAM 203.

[0031] In S401, the acquisition unit 310 acquires, from the database 30, the arrangement information (position information, angle of view information) of the image capturing device 10 and the captured image.

[0032] In S402, the placement determination unit 320 determines whether or not a virtual camera needs to be added based on the placement information of the camera 10 acquired in S401, and if so, determines the position to add the virtual camera and the placement of the virtual camera. A specific flow of determining the placement of the virtual camera will be described later with reference to Figs. 5(a) to 5(c).

[0033] In S403, the virtual captured image generating unit 330 virtually generates an image captured by the virtual camera based on the arrangement of the virtual camera determined in S402. That is, a virtual captured image of the virtual camera (virtual captured image of the virtual camera) is generated in the determined arrangement (position and angle of view). As described above, by generating an image in a manner different from that of the rendering device 40, the switching of the reference camera in viewpoint-dependent rendering can be made less noticeable. In this embodiment, the virtual captured image is generated by model-based rendering.

[0034] In model-based rendering, the three-dimensional shape information of the subject 3 acquired in S401 is colored using the photographed image to generate a colored model of the subject. The method of coloring using the photographed image may involve acquiring pixels corresponding to points of interest from the photographed image and averaging the pixels, or any method that allows coloring may be used. The generated colored model is projected onto a virtual photography device to generate a virtual photographed image.

[0035] In S404, the editing unit 340 registers the virtual photographed image generated in S403 together with the arrangement information of the virtual photographing device determined in S402 in the database 30. If the database 30 holds information related to the photographing device, such as the visibility of the photographing device with respect to the three-dimensional shape information, in addition to the arrangement information of the photographing device and the photographed image, the information may be edited together.

[0036] By using the database 30 edited by this process, the virtual camera device is used in the same way as an actual camera device in rendering in the rendering device 40. As a result, in a portion where the camera devices are far apart and the change in the referenced images is noticeable, the virtual camera image of the virtual camera device is used, so that the influence of the change in the images can be reduced.

[0037] <Details of the process for determining the placement (position, angle of view) of the virtual imaging device> Next, the details of the arrangement determination process of S402 in Fig. 4 will be described with reference to Fig. 5(a) to Fig. 5(c). Fig. 5(a) is a flow chart showing the procedure of the arrangement determination process of the virtual photography device. Fig. 5(b) and Fig. 5(c) are diagrams for explaining the flow, and show a bird's-eye view of the photography target area from directly above for the purpose of explanation.

[0038] In S501, the placement determination unit 320 sets one or more evaluation points 51 in the photographing target area 2. For example, as shown in FIG. 5(b), the evaluation points 51 are placed in a grid pattern at regular intervals. The evaluation points 51 may be placed in a three-dimensional grid pattern including height, or a single reference height may be determined and the evaluation points 51 may be placed in a planar pattern only at that height. In this embodiment, as shown in FIG. 5(b), the evaluation points 51 are placed in a grid pattern of six points in the X direction and six points in the Y direction. The same number of evaluation points 51 are also placed in the Z direction, but this is not shown.

[0039] In S502, the placement determination unit 320 selects an evaluation point for which calculation has not been completed from the evaluation points 51 set in S501 as an evaluation point 52 to be calculated. In S503, the placement determination unit 320 extracts, from among the image capture devices 10, an image capture device 53 (image capture device indicated by diagonal lines in FIG. 5(b)) that captures the evaluation point 52 to be calculated selected in S502 within its angle of view. Here, if a virtual image capture device has already been added, the virtual image capture device is also extracted.

[0040] In S504, the placement determination unit 320 calculates the direction from which the camera 53 from which the evaluation point 52 to be calculated was extracted is photographing, as shown in Figure 5 (c), and further calculates the angle 54 between the photographing directions of adjacent camera devices 53.

[0041] In S505, the placement determination unit 320 judges whether the calculated angle between the camera devices is equal to or greater than a preset threshold. If the angle is equal to or greater than the threshold, it is determined that a virtual camera device needs to be added, and the process proceeds to S506. On the other hand, if the angle is less than the threshold, it is determined that a virtual camera device does not need to be added, and the process proceeds to S508.

[0042] In S506, the placement determination unit 320 determines a position to add the virtual camera 55. Specifically, the midpoint between two camera devices (e.g., 56 and 57 in FIG. 5) that form an angle equal to or larger than a threshold is determined as the position of the virtual camera device 55.

[0043] At this time, if the angle between the imaging devices is equal to or greater than the threshold even when the virtual imaging device is placed at the midpoint, the location is determined to be not the midpoint but a third point, a fourth point, or the like, so that the angle between the imaging devices is equal to or less than the threshold. For example, first, the location of the virtual imaging devices is determined so that two virtual imaging devices are placed at the third points of the two imaging devices. If the angle between the imaging devices is equal to or greater than the threshold even when the virtual imaging device is placed at the third point, the location of the virtual imaging devices is determined so that three virtual imaging devices are placed at the fourth points of the two imaging devices. Thereafter, the location may be increased in the same manner to five points, sextant points, and so on.

[0044] In S507, the placement determination unit 320 determines the angle of view of the virtual camera 55. Specifically, the orientation is set so as to face the evaluation point 52, and the longer focal length of the two camera devices (for example, 56 and 57 in FIG. 5) is used as the focal length. The method of determining the angle of view of the virtual camera device is not limited to this, and any method may be used as long as the evaluation point 52 falls within the angle of view. On the other hand, by setting the virtual camera device 55 to a wide angle, the number of virtual camera devices can be reduced.

[0045] In S508, the placement determination unit 320 judges whether all evaluation points have been calculated. If there are evaluation points that have not been calculated, the process returns to S502. If the calculation of all evaluation points has been completed, the process of determining the placement of the virtual camera is terminated.

[0046] <Effects> The effect of this embodiment will be described with reference to FIG. 6. The arrangement of the photographing devices is the same as in FIG. 5(b) and FIG. 5(c), and the same devices are assigned the same numbers. The rendering device 40 will be described as selecting and rendering an image of the photographing device with the closest light vector. The virtual camera 600 rendered by the rendering device 40 moves from the vicinity of the photographing device 56 to the photographing device 57. If the database 30 has not been edited and there is no virtual photographing device 55, the image of the photographing device 56 is selected in the areas 601 and 602, and the image of the photographing device 57 is selected in the areas 603 and 604. Therefore, the image of the photographing device 56, which looks very different between the areas 602 and 603, is switched to the image of the photographing device 57, and the switch is noticeable. On the other hand, if the database 30 is edited by the information processing device 100 and the virtual photographing device 55 is added, the image of the photographing device 56 is selected in the area 601. Then, in areas 602 and 603, the image of the virtual camera 55 is selected, and in area 604, the image of the camera 57 is selected. The number of switchings increases, or the switching becomes less noticeable because the appearances of the cameras do not differ greatly.

[0047] In this way, by editing the database 30 in the information processing device 100, it is possible to make the switching between cameras less noticeable in the rendering device 40 that employs the viewpoint-dependent rendering method.

[0048] [Variations] In the above embodiment, the arrangement determination unit 320 sets unique grid-shaped evaluation points in the photographing target area 2, but the method of setting the evaluation points is not limited to this example. The evaluation points may be set randomly, or may be dynamically determined in accordance with the subject 3. When matching with the subject 3, the three-dimensional shape of the subject is also acquired in S401. Then, by arranging the evaluation points set in S501 on the three-dimensional shape of the subject, it is possible to search for a shortage area of ​​the photographing device optimized for the subject.

[0049] In the above embodiment, the virtual photographed image generating unit 330 generates the virtual photographed image by model-based rendering, but this is not limiting. As long as the virtual photographed image can be generated in the arrangement of the virtual photographing device determined by the arrangement determining unit 320, any known method may be used. For example, the virtual photographed image may be generated by morphing a plurality of photographed images of a plurality of photographing devices arranged nearby. For example, the virtual photographed image may be generated by morphing an image of the photographing device 56 and an image of the photographing device 57. Alternatively, the virtual photographed image of the virtual photographing device may be generated by machine learning. For example, a technology for generating a new viewpoint by machine learning such as NeRF (Neural Radiance Fields) may be used.

[0050] In addition, in the rendering method of the rendering device 40, if the method of selecting the image capture device includes weighting other than the line of sight direction and the position of the image capture device, the priority of the virtual image capture device may be set to be lower than that of the actual image capture device. For example, if weighting is performed based on the resolution, the priority of the virtual image capture device can be lowered by setting the focal length of the virtual image capture device to be short and wide-angle to lower the resolution. This narrows the areas 602 and 603, and the range in which the actual image capture image, which is more realistic than the virtual image capture image, is used can be expanded.

[0051] (Embodiment 2) In the first embodiment, an example was described in which the database is edited so that virtual camera devices are placed in areas where there is a shortage of camera devices, thereby reducing the noticeable switching of cameras to be referenced during viewpoint-dependent rendering. In contrast, in the present embodiment, an example is described in which camera devices in the database are rearranged regardless of the presence or absence of camera devices. For example, by rearranging them evenly, the final image quality can be made uniform.

[0052] The configuration of the information processing device 100 according to this embodiment is the same as that of embodiment 1. The editing unit 340 in embodiment 1 only adds data to the database 30, but in this embodiment, it also deletes original data.

[0053] <Processing> The process flow is the same as that in Fig. 4 described in the first embodiment, but differs in the process of determining the arrangement of the virtual camera in S402. Also, as described above, data deletion is added when registering data in the database in S404.

[0054] The process of determining the placement of the virtual camera in S402 in this embodiment will be described with reference to Fig. 7(a) and Fig. 7(b). Fig. 7(a) is a flowchart showing the procedure of the process of determining the placement of the virtual camera according to this embodiment. Fig. 7(b) is a diagram for explaining the flow of Fig. 7(a), and shows a bird's-eye view of the area to be photographed from directly above for the purpose of explanation.

[0055] In S701, the placement determination unit 320 determines the angle between the virtual camera devices. That is, the angle between the shooting directions of the adjacent virtual camera devices is determined to be a predetermined angle. This angle may be determined in advance, or the user may be asked to input the angle.

[0056] In S702, the placement determination unit 320 generates a plurality of virtual camera devices 70 with the inter-camera angle determined in S701. At this time, by increasing the distance between the virtual camera devices 70 and the subject area 2 and increasing the focal length of the virtual camera devices, the camera devices can be placed so that the angle between them does not change even at the corners of the subject area. This focal length may also be determined in advance, or the user may be asked to input it.

[0057] In S703, the placement determination unit 320 determines the positions of the multiple virtual camera devices 70 by rotating them based on the center of the shooting target area 2. The closer the angle of view of the virtual camera device 70 is to the camera device 10, the higher the resolution of the virtual captured image generated for the virtual camera device 70. Therefore, the rotation angle 71 is optimized so that the camera device 10 and the virtual camera device 70 are closest to each other. In other words, the placement of the multiple virtual camera devices is determined so that there are many virtual camera devices whose placement (position and angle of view) is close to that of the multiple camera devices.

[0058] 4, for the virtual camera device 70 determined in this manner, the virtual camera image generation unit 330 generates a virtual camera image in the same manner as in embodiment 1. Then, in S404, the editing unit 340 deletes the information on the camera device 10 registered in the database 30, and registers in the database 30 the information on the virtual camera device generated in S402 and S403.

[0059] As described above, in this embodiment, the arrangement of the virtual camera devices is determined so that the angle between the shooting directions of adjacent virtual camera devices is a predetermined angle and so that there are many virtual camera devices whose arrangement (position and angle of view) is close to that of the multiple camera devices. Then, the original information is deleted from the database, and information on the multiple virtual camera devices is registered in the database.

[0060] This makes it possible to create a database with an ideal, uniform layout that would not be possible due to physical constraints in the area being photographed, and reduces the impact of camera switching during rendering.

[0061] (Embodiment 3) In the third embodiment, an example will be described in which an information processing device includes a drawing unit that performs viewpoint-dependent rendering equivalent to that of the rendering device 40, and a virtual camera is added to a portion where differences become large when the viewpoint-dependent rendering is actually performed. This makes it possible to directly evaluate issues that arise when performing actual rendering.

[0062] <Functional configuration> 8 shows an example of the functional configuration of an information processing device 800 according to this embodiment. In addition to the configuration of the information processing device 100 of embodiment 1, a drawing unit 550 is provided. Also, instead of the placement determination unit 320, a placement determination unit 520 with a different function is provided. Processing units having the same functions as those of the information processing device 100 of embodiment 1 are given the same reference numerals, and description thereof will be omitted.

[0063] The drawing unit 550 draws and generates a virtual viewpoint image by viewpoint-dependent rendering equivalent to that of the rendering device 40, based on the information of the database 30 acquired by the acquisition unit 310. When a virtual viewpoint has already been determined, the virtual viewpoint to be generated may be the determined virtual viewpoint. When a virtual viewpoint has not been determined, a viewpoint that rotates while keeping the shooting target area 2 at the center is rendered so that the camera switching can be understood, which is a problem of virtual viewpoint-dependent rendering.

[0064] The placement determination unit 520 evaluates the virtual viewpoint image drawn by the drawing unit 550 and determines the placement of the virtual camera. For example, the placement of the virtual camera is determined so that the virtual camera is placed at a location where image switching is noticeable when the virtual viewpoint is moved. Specifically, an evaluation value indicating whether image switching is noticeable when the virtual viewpoint is moved is calculated, and a location where the evaluation value is equal to or greater than a threshold, that is, the virtual viewpoint where the evaluation value is equal to or greater than the threshold, is determined as the placement (position and angle of view) of the virtual camera. A specific evaluation method may be any method that can detect switching, such as an inter-frame difference.

[0065] The disclosure of this specification includes the following information processing device, a control method for an information processing device, and a program.

[0066] (Item 1) An information processing device that edits a database for generating a virtual viewpoint image, the database including arrangement information and photographed images of a plurality of photographing devices that photograph a subject, a determination means for determining an arrangement of virtual camera devices to be placed in a virtual space based on the arrangement information of the plurality of camera devices contained in the database; a generating means for generating a virtual photographed image of the virtual photographing device in the arrangement; an editing means for registering the arrangement information of the virtual photography device and the virtual photographed image in the database and editing the database; An information processing device comprising:

[0067] (Item 2) the database is a database for generating a viewpoint-dependent virtual viewpoint image, 2. The information processing device according to item 1, wherein the database includes three-dimensional shape information of the subject, the arrangement information of the multiple image capturing devices, and the captured images.

[0068] (Item 3) The determining means is determining whether an angle between the imaging directions of the two imaging devices is equal to or greater than a threshold value based on the arrangement information of the plurality of imaging devices; 3. The information processing device according to item 1 or 2, characterized in that, when the angle is equal to or greater than the threshold value, a position of the virtual camera is determined so that the virtual camera is positioned between the two camera devices.

[0069] (Item 4) 4. The information processing device according to item 3, wherein the determination means determines the placement of the virtual camera so that the virtual camera is placed at a midpoint between the two camera devices.

[0070] (Item 5) The information processing device described in item 3 or 4, characterized in that if the virtual camera device is placed at the midpoint of the two camera devices but the angle is equal to or greater than the threshold value, the determination means determines the placement of the virtual camera device so that two virtual camera devices are placed at a third point between the two camera devices.

[0071] (Item 6) The information processing device described in item 1 or 2, characterized in that the determination means determines the arrangement of the multiple virtual camera devices so that the angle between the shooting directions of adjacent virtual camera devices is a predetermined angle and so that there are many virtual camera devices that are close to the arrangement of the multiple camera devices.

[0072] (Item 7) The generating means generates a plurality of virtual photographed images of the plurality of virtual photographing devices in the arrangement, The information processing device described in item 6, characterized in that the editing means edits the database by registering the layout information of the plurality of virtual shooting devices and the plurality of virtual photographed images in the database, and deleting the layout information of the plurality of shooting devices and the photographed images.

[0073] (Item 8) The information processing device described in any one of items 1 to 7, characterized in that the generation means generates the virtual captured image by coloring a three-dimensional shape of the subject and projecting the colored model of the subject onto the virtual shooting device.

[0074] (Item 9) 8. The information processing device according to any one of items 1 to 7, wherein the generating means generates the virtual captured image by morphing a plurality of captured images from a plurality of image capturing devices disposed nearby.

[0075] (Item 10) 8. The information processing device according to any one of items 1 to 7, wherein the generating means generates the virtual captured image by machine learning.

[0076] (Item 11) A rendering unit that renders a virtual viewpoint image based on information in the database, The information processing device described in item 1 or 2, characterized in that the determination means calculates an evaluation value indicating whether image switching is noticeable when the virtual viewpoint is moved, and determines the placement of the virtual shooting device at a location where the evaluation value is equal to or greater than a threshold value.

[0077] (Item 12) 12. The information processing device according to any one of items 1 to 11, wherein the arrangement information of the plurality of image capturing devices includes information on positions and angles of view of the plurality of image capturing devices.

[0078] (Item 13) 13. The information processing device according to any one of items 1 to 12, wherein the arrangement information of the virtual camera includes information on a position and an angle of view of the virtual camera.

[0079] (Item 14) A method for controlling an information processing device that edits a database for generating a virtual viewpoint image, the database including arrangement information and photographed images of a plurality of photographing devices that photograph a subject, comprising: a determination step of determining an arrangement of virtual camera devices to be placed in a virtual space based on the arrangement information of the plurality of camera devices contained in the database; a generating step of generating a virtual captured image of the virtual capture device at the position; an editing step of registering the arrangement information of the virtual photography device and the virtual photographed image in a database and editing the database; 13. A method for controlling an information processing apparatus comprising the steps of:

[0080] (Item 15) Item 15. A program for causing a computer to execute the control method for an information processing device according to item 14.

[0081] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0082] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0083] 30: database, 100: information processing device, 310: acquisition unit, 320: arrangement determination unit, 330: virtual captured image generation unit, 340: editing unit

Claims

1. An information processing device that edits a database for generating a virtual viewpoint image, the database including arrangement information and photographed images of a plurality of photographing devices that photograph a subject, a determination means for determining an arrangement of virtual camera devices to be placed in a virtual space based on the arrangement information of the plurality of camera devices contained in the database; a generating means for generating a virtual photographed image of the virtual photographing device in the arrangement; an editing means for registering the arrangement information of the virtual photography device and the virtual photographed image in the database and editing the database; An information processing device comprising:

2. the database is a database for generating a viewpoint-dependent virtual viewpoint image, 2. The information processing apparatus according to claim 1, wherein the database includes three-dimensional shape information of the subject, the arrangement information of the plurality of image capturing devices, and the captured images.

3. The determining means is determining whether an angle between the imaging directions of the two imaging devices is equal to or greater than a threshold value based on the arrangement information of the plurality of imaging devices; The information processing apparatus according to claim 1 , wherein, when the angle is equal to or greater than the threshold value, a location of the virtual camera is determined such that the virtual camera is located between the two camera devices.

4. 4. The information processing apparatus according to claim 3, wherein the determining means determines the location of the virtual camera so that the virtual camera is located at a midpoint between the two camera devices.

5. The information processing device according to claim 3, characterized in that, when the virtual camera is positioned at the midpoint of the two camera devices but the angle is equal to or greater than the threshold value, the determination means determines the position of the virtual camera so that two virtual camera devices are positioned at a third point between the two camera devices.

6. The information processing device according to claim 1, characterized in that the determination means determines the arrangement of the multiple virtual camera devices so that the angle between the shooting directions of adjacent virtual camera devices is a predetermined angle and so that there are more virtual camera devices that are close to the arrangement of the multiple camera devices.

7. The generating means generates a plurality of virtual photographed images of the plurality of virtual photographing devices in the arrangement, 7. The information processing apparatus according to claim 6, wherein the editing means edits the database by registering the arrangement information of the plurality of virtual imaging devices and the plurality of virtual photographed images in the database, and deleting the arrangement information of the plurality of imaging devices and the photographed images.

8. 2 . The information processing apparatus according to claim 1 , wherein the generating means generates the virtual photographed image by coloring a three-dimensional shape of the subject and projecting the colored model of the subject onto the virtual photographing device.

9. The information processing apparatus according to claim 1 , wherein the generating means generates the virtual captured image by morphing a plurality of captured images taken by a plurality of image taking devices disposed nearby.

10. The information processing apparatus according to claim 1 , wherein the generating means generates the virtual captured image by machine learning.

11. A rendering unit that renders a virtual viewpoint image based on information in the database, The information processing device according to claim 1, characterized in that the determination means calculates an evaluation value indicating whether image switching is noticeable when the virtual viewpoint is moved, and determines the placement of the virtual shooting device at a location where the evaluation value is equal to or greater than a threshold value.

12. The information processing apparatus according to claim 1 , wherein the arrangement information of the plurality of image capturing devices includes information on positions and angles of view of the plurality of image capturing devices.

13. The information processing apparatus according to claim 1 , wherein the arrangement information of the virtual camera includes information on a position and an angle of view of the virtual camera.

14. A method for controlling an information processing device that edits a database for generating a virtual viewpoint image, the database including arrangement information and photographed images of a plurality of photographing devices that photograph a subject, comprising: a determination step of determining an arrangement of virtual camera devices to be placed in a virtual space based on the arrangement information of the plurality of camera devices contained in the database; a generating step of generating a virtual captured image of the virtual capture device at the position; an editing step of registering the arrangement information of the virtual photography device and the virtual photographed image in a database and editing the database; 13. A method for controlling an information processing apparatus comprising the steps of:

15. A program for causing a computer to execute the method for controlling an information processing device according to claim 14.

Images