Generation device, generation method, and program

Abstract

To provide a generation technique for a virtual viewpoint image capable of representing reflection of a subject.SOLUTION: A generation device 10 generates, based upon a first virtual viewpoint image generated based upon a first virtual viewpoint positioned in a three-dimensional space including an imaging space picked up by a plurality of imaging devices and a plurality of picked-up images, and a second virtual viewpoint image generated based upon a second virtual viewpoint arranged so as to sandwich a predetermined plane in the three-dimensional space with the first virtual viewpoint, and the plurality of picked-up images, a third virtual viewpoint image corresponding to the first virtual viewpoint and including an image representing a state in which an object positioned in the imaging space reflects on the predetermined plane.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating a virtual viewpoint image. [Background technology]

[0002] In recent years, a technology has been attracting attention in which multiple imaging devices are arranged around an imaging area to capture images, and the multiple captured images obtained from each imaging device are used to generate an image (virtual viewpoint image) viewed from a specified viewpoint (virtual viewpoint).

[0003] Patent Document 1 describes a method of generating a virtual viewpoint image by synthesizing an image of a subject, which is generated using three-dimensional shape data of the subject, with an image of the background. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-211828 A Summary of the Invention [Problem to be solved by the invention]

[0005] Depending on the background depicted in the virtual viewpoint image, it is required to give a sense of reality by expressing the subject being reflected on the floor or wall of the background in the virtual viewpoint image. However, such a thing is not taken into consideration in Patent Document 1.

[0006] The present disclosure has been made in view of the above-mentioned problems. It is an object of the present disclosure to provide a technology for generating a virtual viewpoint image capable of expressing the reflection of a subject. [Means for solving the problem]

[0007] The generating means according to the present disclosure is a generating unit for generating a plurality of captured images acquired by imaging performed by a plurality of imaging devices. andA first virtual viewpoint image generated based on a first virtual viewpoint and information indicating surface irregularities on a predetermined surface included in a three-dimensional space. a first acquisition means for acquiring the plurality of captured images and the first virtual viewpoint; Regarding the above a second acquisition means for acquiring a second virtual viewpoint image generated based on a second virtual viewpoint disposed on either side of a predetermined plane; The above A first virtual viewpoint image; The blurring process is performed based on the information. Based on the second virtual viewpoint image, the predetermined surface corresponds to the first virtual viewpoint. Images are captured by multiple imaging devices and generating means for generating a third virtual viewpoint image including an image showing a state in which an object located in the imaging space is reflected. Effect of the Invention

[0008] According to the present disclosure, it is possible to generate a virtual viewpoint image capable of expressing the reflection of a subject. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating a hardware configuration of a generating device. [Diagram 2] FIG. 2 is a diagram for explaining a configuration of a generation system according to the first embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of a virtual viewpoint image generated by the generating device. [Figure 4] 1A to 1C are diagrams illustrating an example of a virtual viewpoint, a reflected viewpoint, and a reflected viewpoint image corresponding to the reflected viewpoint. [Diagram 5] 13 is a flowchart illustrating a process performed by a generating device. [Figure 6] FIG. 11 is a diagram for explaining a configuration of a generation system according to a second embodiment. [Figure 7] 13 is a flowchart for explaining a process performed by a generation system in a second embodiment. [Figure 8] FIG. 13 is a diagram showing an example of a reflection viewpoint. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are examples of the embodiments of the present disclosure, and the present disclosure is not limited to them.

[0011] (First embodiment) In this embodiment, a generating device that generates a virtual viewpoint image that imitates a subject being reflected on a background will be described. The virtual viewpoint image in this embodiment is also called a free viewpoint video, but is not limited to an image corresponding to a viewpoint (virtual viewpoint) freely (arbitrarily) specified by a user, and includes, for example, an image corresponding to a viewpoint selected by a user from a plurality of candidates. In addition, in this embodiment, the case where the virtual viewpoint is specified by a user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis, etc. In addition, in this embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

[0012] 1 is a diagram illustrating a hardware configuration of a generating device 10 according to this embodiment. The generating device 10 includes a CPU 501, a RAM 502, a ROM 503, an operation unit 504, an output unit 505, an auxiliary storage device 506, an I / F 507, and a bus 508.

[0013] The CPU 501 controls the entire generating device 10 using computer programs and data stored in the RAM 502 and the ROM 503. That is, the CPU 501 functions as each processing unit in the generating device 10.

[0014] The RAM 502 temporarily stores computer programs and data read from the auxiliary storage device 506, and data acquired from the outside via an I / F (interface) 507. The RAM 502 also has a work area used when the CPU 501 executes various processes. That is, the RAM 502 can be allocated as a frame memory, for example, and can provide various other areas as appropriate.

[0015] The ROM 503 stores data related to the settings of the generating device 10, a boot program, etc. The operation unit 504 is composed of a keyboard, a mouse, a joystick, etc. A user of the generating device 10 can input various instructions to the CPU 501 by operating the operation unit 504. The output unit 505 is also composed of, for example, a liquid crystal display, and displays the processing results by the CPU 501, etc.

[0016] The auxiliary storage device 506 is an information storage device, typified by a hard disk drive device. The auxiliary storage device 506 stores an OS (operating system), computer programs for causing the CPU 501 to realize the functions of each processing unit of the generating device 10, and the like. The computer programs and data stored in the auxiliary storage device 506 are loaded into the RAM 502 as appropriate under the control of the CPU 501, and become targets for processing by the CPU 501. The auxiliary storage device 506 also stores image data and the like to be processed by the generating device 10.

[0017] The I / F 507 is connected to a network such as a LAN or the Internet, and to other devices such as a projector or display device, and transmits and receives information. For example, when the generating device 10 is connected to an external device by wire, a communication cable is connected to the I / F 507. When the generating device 10 has a function of wirelessly communicating with an external device, the I / F 507 is equipped with an antenna. The bus 508 connects the above-mentioned components to transmit information.

[0018] 1, the operation unit 504, the output unit 505, and the auxiliary storage device 506 are included inside the refinement device 10, but are not limited thereto. For example, at least one of the operation unit 504, the output unit 505, and the auxiliary storage device 506 may be connected to the outside of the generation device 10 as another device.

[0019] FIG. 2 is a diagram for explaining the configuration of a generation system that generates a virtual viewpoint image in this embodiment. The generation system 20 in FIG. 1 has a plurality of imaging units 1a, 1z, a synchronization unit 2, a three-dimensional shape estimation unit 3, a storage unit 4, and a generation device 10. In addition, with respect to the reference symbols in FIG. 2, symbols that differ only in the alphabet attached after the number indicate different instances of devices having the same function. For example, the imaging unit 1a and the imaging unit 1z indicate different instances having the same function. In addition, having the same function refers to having at least a specific function (such as an imaging function), and for example, the imaging unit 1a and the imaging unit 1z may have different functions and performances. In addition, in the following description, the imaging unit 1 is a collective term for the imaging units 1a to 1z. The number of imaging units 1 may be changed arbitrarily.

[0020] The multiple imaging units 1 are imaging devices such as cameras. The multiple imaging units 1 are installed to surround a subject and capture images of the subject from multiple directions. In this embodiment, the subject captured by the multiple imaging units 1 is an object for which three-dimensional shape data is generated by a three-dimensional shape estimation unit 3 described later. The subject may be, for example, a person and a tool used by the person. The multiple imaging units 1 output captured images obtained by imaging to the three-dimensional shape estimation unit 3. The synchronization unit 2 generates a synchronization signal and transmits it to the multiple imaging units 1. The multiple imaging units 1 capture images in synchronization with each other based on the synchronization signal received from the synchronization unit 2.

[0021] The three-dimensional shape estimation unit 3 generates three-dimensional shape data representing the three-dimensional shape of the subject using the input multiple captured images. The three-dimensional shape estimation unit 3 in this embodiment generates a silhouette image obtained by extracting the area of ​​the subject in the captured image. The three-dimensional shape estimation unit 3 also generates three-dimensional shape data of the subject using the multiple silhouette images by a method based on the shape-from-silhouette method, which is a known technology. Note that the method of estimating the three-dimensional shape is not limited to this, and other methods may be used. The generated three-dimensional shape data of the subject includes information representing the position of the subject in the imaging space and information representing the three-dimensional shape of the subject. The silhouette image may be generated by a device different from the three-dimensional shape estimation unit 3. The three-dimensional shape estimation unit 3 outputs the generated three-dimensional shape data of the subject and the multiple captured images acquired from the multiple imaging units 1 to the storage unit 4.

[0022] The storage unit 4 stores a plurality of captured images and three-dimensional shape data output from the three-dimensional shape estimation unit 3. The storage unit 4 also stores three-dimensional shape data of an object (hereinafter referred to as background) different from the subject represented by the three-dimensional shape data. The background here may be, for example, a wall or a floor. The three-dimensional shape data of the background in this embodiment is represented by a polygon mesh and has coordinate information in three-dimensional space. The three-dimensional shape data of the background is generated by the following method. For example, like the three-dimensional shape data of the subject, the three-dimensional shape data of the background is generated based on a captured image obtained by the imaging unit 1 performing imaging. At this time, the captured image used to generate the three-dimensional shape data of the background may be a captured image based on imaging performed at a time or place different from that of the subject. Also, for example, by expressing a virtual object that does not exist in the imaging space in which the subject exists using computer graphics (CG), three-dimensional shape data of the background that imitates a virtual space may be generated. Also, the three-dimensional shape data of the background includes material information of each mesh, such as the reflectance and surface shape of the mesh. The material information of the mesh is not limited to the above, and may include information on at least one of the reflectance and the surface shape, or may further include information other than the reflectance and the surface shape. The information on the reflectance and the surface shape may be information representing the reflectance and the surface shape of an object corresponding to the background in the real space, or may be information preset by the user. For example, when the background is a virtual object generated using CG, the reflectance and the surface shape are set by the user.

[0023] In this embodiment, a virtual viewpoint image is generated that mimics the subject being reflected in the background. At this time, it is necessary to identify the surface on which the subject is reflected (hereinafter, referred to as the reflective surface). The reflective surface in this embodiment is an area on which the image of the reflective subject is superimposed, and does not have to be an actual reflective surface. That is, by using the technique described in this embodiment, it is possible to express the subject being reflected even on a non-reflective floor surface that is actually captured, or a virtual floor surface generated using CG. For example, the reflective surface is a glossy surface such as a metal surface or glass. Alternatively, the reflective surface may be a virtual surface that is virtually expressed in the background as a texture having such a glossy surface. Typically, the reflective surface is a surface on which specular reflection occurs, or a virtual surface expressed with a texture that causes specular reflection.

[0024] The three-dimensional shape data of the background in this embodiment includes the coordinates of the reflecting surface in the three-dimensional space set in advance by the user as the three-dimensional coordinate information. Therefore, the reflecting surface can be specified by referring to the three-dimensional shape data of the background. Also, for example, the area where the reflection occurs may be specified as the reflecting surface based on the reflectance information. In the following description, the three-dimensional coordinate information for specifying the reflecting surface and the information such as the reflectance of the reflecting surface are referred to as the reflecting surface information. Note that the form of the three-dimensional shape data of the background is not limited to the above. Also, the reflectance of the reflecting surface may not be uniform, and may have different reflectances depending on the position on the reflecting surface.

[0025] The storage unit 4 also stores background texture data for coloring the background three-dimensional shape data. Specifically, the background texture data is an image to be pasted onto each polygon mesh. The storage unit 4 also stores camera parameters such as the installation positions, attitudes, and optical characteristics of the multiple imaging units 1.

[0026] It is not necessary that the entire plurality of captured images are stored in the storage unit 4. For example, it is also possible that an image in which an area required for coloring three-dimensional shape data is extracted from the captured images is stored in the storage unit 4. The storage unit 4 transmits the stored data to the generating device 10 in response to an instruction from the generating device 10.

[0027] Next, the functional configuration of the generation device 10 will be described with reference to FIGS. 2 to 4. The generation device 10 has a virtual viewpoint input unit 5, a reflected viewpoint generation unit 6, a first image generation unit 7a, a second image generation unit 7b, a superimposition unit 8, and a display unit 9.

[0028] The virtual viewpoint input unit 5 accepts an input for determining a virtual viewpoint. The virtual viewpoint input unit 5 accepts input information obtained by, for example, a user operating the operation unit 504, and inputs virtual viewpoint information determined based on the accepted input information to the first image generating unit 7a and the reflected viewpoint generating unit 6. Here, the virtual viewpoint information is information including parameters indicating the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint. The virtual viewpoint information also includes parameters for identifying the time in the generated virtual viewpoint image. Furthermore, the virtual viewpoint information may also include parameters indicating the angle of view (width of the field of view) of the virtual viewpoint. The virtual viewpoint information is information including at least parameters indicating the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint. The virtual viewpoint input unit 5 may be configured to acquire virtual viewpoint information stored in, for example, the auxiliary storage device 506 and an externally connected storage device.

[0029] The first image generating unit 7a generates a virtual viewpoint image based on the virtual viewpoint information acquired from the virtual viewpoint input unit 5. The first image generating unit 7a acquires data required for generating the virtual viewpoint image from the storage unit 4 based on the virtual viewpoint information. The first image generating unit 7a uses the three-dimensional shape data of the background and the background texture data from the acquired data to draw the background in the virtual viewpoint image. The first image generating unit 7a also uses the three-dimensional shape data of the subject and the captured image from the acquired data to draw the subject in the virtual viewpoint image. At this time, the subject viewed from the virtual viewpoint is drawn by coloring the three-dimensional shape data of the subject based on the captured image. As a result, the first image generating unit 7a generates and outputs a virtual viewpoint video as shown in FIG. 3(a).

[0030] The reflection viewpoint generating unit 6 acquires three-dimensional shape data of the background from the storage unit 4, and identifies a reflection surface based on information included in the acquired three-dimensional shape data. The reflection surface is, for example, the reflection surface 200 in FIG. 3(a). In this embodiment, the reflection surface is identified based on information indicating the three-dimensional coordinates of the reflection surface included in the three-dimensional shape data. The reflection viewpoint generating unit 6 generates a reflection viewpoint based on the virtual viewpoint information acquired from the virtual viewpoint input unit 5 and the identified reflection surface. The reflection viewpoint is a virtual viewpoint having a predetermined positional relationship with the reflection surface and the designated virtual viewpoint, and is acquired by converting the position of the designated virtual viewpoint based on the position of the reflection surface. The reflection viewpoint generating unit 6 inputs virtual viewpoint information (hereinafter referred to as reflection viewpoint information) representing the reflection viewpoint to the second image generating unit 7b.

[0031] The second image generating unit 7b generates a virtual viewpoint image (hereinafter, referred to as a reflected viewpoint image) corresponding to the reflected viewpoint based on the reflected viewpoint information and the data acquired from the storage unit 4. For example, the reflected viewpoint image corresponding to the virtual viewpoint image shown in Fig. 3(a) is an image as shown in Fig. 3(b). A detailed method of generating the reflected viewpoint and the reflected viewpoint image will be described later.

[0032] The superimposing unit 8 superimposes the virtual viewpoint image generated by the first image generating unit 7a, the reflected viewpoint image generated by the second image generating unit 7b, and the reflection surface information, on the virtual viewpoint image and the reflected viewpoint information. As a result, a virtual viewpoint image that imitates the reflection of the subject is generated, as shown in Fig. 3(d). The display unit 9 displays the virtual viewpoint image generated by the superimposing unit 8.

[0033] Next, the processing performed by the generating device 10 will be described with reference to Figs. 3 to 5. Fig. 5 is a flowchart for explaining the processing performed by the generating device 10. The processing shown in Fig. 5 is executed by the CPU 501 of the generating device 10 reading and executing a program stored in the ROM 503 or the auxiliary storage device 506. The processing shown in Fig. 5 is started when the user performs an operation to specify a virtual viewpoint.

[0034] In S101, the virtual viewpoint input unit 5 accepts an input operation for specifying the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint, which is performed on the operation unit 504. In addition, the virtual viewpoint input unit 5 transmits virtual viewpoint information determined based on the accepted input operation to the first image generation unit 7a and the reflected viewpoint generation unit 6. Here, it is assumed that the virtual viewpoint 300 in FIG. 4(a) is determined as the virtual viewpoint.

[0035] In S102, the first image generating unit 7a obtains the captured image, the three-dimensional shape data of the subject, and the three-dimensional shape data of the background as data required for generating a virtual viewpoint image based on the obtained virtual viewpoint information from the storage unit 4. In S103, the first image generating unit 7a and the reflection viewpoint generating unit 6 obtain the three-dimensional shape data of the background from the storage unit 4, and identify a reflection surface based on information included in the obtained three-dimensional shape data. Here, the reflection surface 200 in FIG. 3(a) is identified.

[0036] Next, a description will be given of the processes S104 to S106 performed by the first image generation unit 7a and the processes S107 to S109 performed by the reflection viewpoint generation unit 6 and the second image generation unit 7b. The processes S104 to S106 and the processes S107 to S109 can be performed in parallel.

[0037] In S104, the first image generating unit 7a performs coloring (rendering) of the three-dimensional shape data of the subject and the three-dimensional shape data of the background based on the captured image and the virtual viewpoint information. In S105, the first image generating unit 7a generates a virtual viewpoint image in which the subject and background are drawn by rendering. Here, FIG. 3(a) is generated as a virtual viewpoint image corresponding to the virtual viewpoint 300 in FIG. 4(a). The first image generating unit 7a transmits the generated virtual viewpoint image to the superimposing unit 8.

[0038] In S106, the first image generating unit 7a generates an image (hereinafter, referred to as a reflecting surface image) by extracting an area corresponding to the reflecting surface from the generated virtual viewpoint image. The reflecting surface image is, for example, an image obtained by excluding the area of ​​the subject from the area of ​​the reflecting surface, as shown in FIG. 3(c). The reflecting surface image is generated, for example, by the following method. The first image generating unit 7a uses the reflecting surface information acquired in S103 to perform rendering so that the inside of the reflecting surface has a predetermined pixel value (for example, 1). The first image generating unit 7a also performs rendering so that the subject and the background other than the reflecting surface have a pixel value (for example, 0) different from that of the reflecting surface. As a result, a reflecting surface image is generated in which the area of ​​the reflecting surface excluding the subject is rendered in a predetermined color, as shown in FIG. 3(c). The first image generating unit 7a transmits the generated reflecting surface image to the superimposing unit 8.

[0039] In S107, the reflected viewpoint generating unit 6 generates reflected viewpoint information based on the virtual viewpoint information acquired from the virtual viewpoint input unit 5. Here, a reflected viewpoint 301 is generated by inverting the virtual viewpoint 300 in Fig. 4(a) with respect to the reflecting surface 200. In other words, the reflected viewpoint 301 is a virtual viewpoint obtained by inverting both the position of the virtual viewpoint 300 and the line of sight from the virtual viewpoint 300 with respect to the reflecting surface.

[0040] The process of generating a reflected viewpoint by the reflected viewpoint generating unit 6 will be described with reference to Fig. 8. Fig. 8(a) is a diagram for explaining the attitude of a virtual camera. For convenience, the virtual viewpoint will be replaced with a virtual camera (hereinafter, referred to as a virtual camera) in the description. The position and line of sight of the virtual viewpoint correspond to the position and attitude of the virtual camera, respectively. The attitude of the virtual camera is represented by the rotation angle of the virtual camera in the horizontal direction (pan), the rotation angle of the virtual camera in the vertical direction (tilt), and the rotation angle of the virtual camera in the optical axis direction (roll).

[0041] FIG. 8(b) is a diagram showing an example of a virtual viewpoint and a reflection viewpoint. In the example of FIG. 8(b), (x, y, z, p, t, 0°) are set as parameters representing the position of the virtual viewpoint 300 and the line of sight direction from the virtual viewpoint. Here, x, y, and z are the z-coordinate, y-coordinate, and z-coordinate values ​​of the virtual viewpoint 300 in a three-dimensional space represented by the xyz coordinate system, respectively. Also, p, t, and 0° are the values ​​of the pan, tilt, and roll angles of the virtual viewpoint 300 described in FIG. 8(a), respectively. The pan, tilt, and roll angles are represented as rotation angles from the respective reference directions (e.g., the x-axis, the y-axis, and the z-axis). Here, it is assumed that the coordinate representing the reflection surface is z=0. The reflection viewpoint generation unit 6 determines the position of the reflection viewpoint 301 so that it is a position that is plane-symmetrical to the virtual viewpoint 300 with respect to the reflection surface (z=0). As a result, the values ​​of the x-coordinate, the y-coordinate, and the z-coordinate of the reflection viewpoint 301 are x, y, and -z, respectively. Furthermore, the reflection viewpoint generating unit 6 determines the line of sight from the reflection viewpoint 301 based on the reflection surface (z=0). At this time, the reflection viewpoint generating unit 6 changes the line of sight in the order of pan, tilt, and roll. The pan value of the reflection viewpoint 301 is the same as the pan value of the virtual viewpoint 300. Furthermore, the tilt value of the reflection viewpoint 301 is a value obtained by inverting the tilt value of the virtual viewpoint 300. Furthermore, the roll value of the reflection viewpoint 301 is converted from 0° to 180°. As a result, the parameters representing the reflection viewpoint 301 are (x, y, -z, p, -t, 180°). In this way, the position of the reflection viewpoint 301 is in a plane-symmetrical position with respect to the reflection surface of the position of the virtual viewpoint 300. Furthermore, the line of sight from the reflection viewpoint 301 is a line of sight that intersects with the reflection surface at the intersection of the reflection surface and the line of sight from the virtual viewpoint 300. That is, the line of sight from the reflecting viewpoint 301 and the line of sight from the virtual viewpoint 300 intersect at the same position on the reflecting surface. The position of the reflecting viewpoint 301 may be set based on a plane-symmetrical position of the position of the virtual viewpoint 300 with respect to the reflecting surface, or may be shifted from the plane-symmetrical position of the position of the virtual viewpoint 300 with respect to the reflecting surface. Also, the intersection of the line of sight from the reflecting viewpoint 301 and the reflecting surface may be shifted from the intersection of the reflecting surface and the line of sight from the virtual viewpoint 300. That is, the parameters (x, y, -z, p, -t, 180°) representing the reflecting viewpoint 301 may be shifted to some extent.

[0042] The method described above generates parameters (reflection viewpoint information) representing the reflection viewpoint. Although the example in FIG. 8 shows a case where the reflection surface is at z=0, each parameter can be converted based on the three-dimensional position of the reflection surface for reflection surfaces other than z=0. In the above-mentioned method, when the roll value is not 0, the calculation of the conversion can become complicated. Therefore, when performing rotation in the roll direction, the superimposition unit 8 may be controlled so that the virtual viewpoint image generated in the subsequent stage is rotated. The reflection viewpoint generation unit 6 transmits the generated reflection viewpoint information to the second image generation unit 7b.

[0043] In S108, the second image generating unit 7b generates a reflected viewpoint image based on the reflection surface information and the reflected viewpoint information. Here, a method of generating a reflected viewpoint image shown in FIG. 3(b) will be described. The second image generating unit 7b renders the color of the object seen from the reflected viewpoint based on the shape data of the object and the reflected viewpoint information. At this time, the second image generating unit 7b uses the reflection surface information to perform rendering only within the area corresponding to the reflection surface. Since only the object reflected on the reflection surface needs to be drawn as the reflected viewpoint image, unnecessary rendering processing can be reduced by performing rendering only within the reflection surface. At this time, the second image generating unit 7b does not perform rendering of the reflection surface. Note that the second image generating unit 7b in this embodiment is configured to render only the object reflected on the reflection surface, but may also be configured to further render the background reflected on the reflection surface.

[0044] Here, the positional relationship of the subject when viewed from the reflection viewpoint 301 is inverted left and right as shown in FIG. 4(c). Therefore, in S109, the second image generation unit 7b inverts left and right the image obtained by rendering within the reflection surface in S108. As a result, the second image generation unit 7b generates the reflected viewpoint image shown in FIG. 3(b). The second image generation unit 7b transmits the generated reflected viewpoint image to the superimposition unit 8.

[0045] In S110, the superimposing unit 8 generates a virtual viewpoint image simulating the reflection of the subject as shown in FIG. 3(d) based on the virtual viewpoint image, the reflecting surface image, and the reflected viewpoint image. The virtual viewpoint image generated by the superimposing unit 8 is a virtual viewpoint image corresponding to the virtual viewpoint represented by the virtual viewpoint information acquired by the virtual viewpoint input unit 5. The virtual viewpoint image shown in FIG. 3(d) is generated as follows. The superimposing unit 8 extracts an area corresponding to the reflecting surface 202 represented by the reflecting surface image (FIG. 3(c)) from the reflected viewpoint image (FIG. 3(b)) acquired from the second image generating unit 7b. The superimposing unit 8 also generates the virtual viewpoint image shown in FIG. 3(d) by superimposing the extracted area on the virtual viewpoint image (FIG. 3(a)). This virtual viewpoint image includes an image showing a state in which an object located in the imaging space is reflected by the reflecting surface. Also, even if an object located in the imaging space is not included in the virtual viewpoint image generated by the first image generating unit 7a, it may be included in the virtual viewpoint image generated by the second image generating unit 7b. In this case, the virtual viewpoint image generated by the superimposing unit 8 can include an image showing the state in which the object is reflected on the reflecting surface, making it possible to obtain a more realistic image.

[0046] In addition, when the superimposing unit 8 superimposes the reflected viewpoint image on the virtual viewpoint image, the superimposing unit 8 can reproduce a more realistic reflection by performing a predetermined image processing on the reflected viewpoint image. For example, the superimposing unit 8 performs processing to change at least one of the transmittance or blending ratio of the reflected viewpoint image based on the reflectance included in the reflection surface information, and then superimposes it on the virtual viewpoint image. The transmittance of the reflected viewpoint image is a parameter that determines the ratio of pixel values ​​of the reflected viewpoint image when the reflected viewpoint image is superimposed on the virtual viewpoint image. The higher the transmittance, the smaller the ratio of pixel values. In addition, the blending ratio is a parameter that determines the ratio of pixel values ​​of each image when the virtual viewpoint image and the reflected viewpoint image are synthesized. The higher the blending ratio of the reflected viewpoint image, the larger the ratio of pixel values ​​of the reflected viewpoint image and the smaller the ratio of pixel values ​​of the virtual viewpoint image.

[0047] For example, the superimposition unit 8 uses the reflected viewpoint image before image processing as a reference and performs at least one of a process of lowering the transmittance of the reflected viewpoint image and a process of increasing the blending ratio as the reflectance included in the reflective surface information becomes higher.

[0048] Also, for example, the superimposing unit 8 performs processing to blur or distort the reflected viewpoint image based on information that represents the surface shape of the background, such as bumps and recesses, contained in the three-dimensional shape data of the background, and then superimposes it on the virtual viewpoint image.

[0049] Also, for example, in the reflection surface 200 drawn in the virtual viewpoint image, the overlapping ratio of the reflected viewpoint image is made low in the part with high brightness, and the overlapping ratio of the reflected viewpoint image is made high in the part with low brightness. This process can simulate the state in which the subject is relatively strongly reflected on the reflection surface of a dark color, and relatively weakly reflected on the reflection surface of a light color, or is not reflected at all. In this process, for example, the overlapping unit 8 refers to a predetermined threshold value related to brightness, and changes the pixel value of the reflected viewpoint image depending on whether the brightness of the reflection surface in the virtual viewpoint image is higher or lower than the predetermined threshold. Also, for example, the overlapping unit 8 may refer to a table that associates the brightness of the reflection surface with the ratio of the pixel value of the reflection viewpoint image, and change the pixel value of the reflected viewpoint image depending on the brightness of the reflection surface.

[0050] The various image processes for the reflected viewpoint image described above may be performed by the second image generating unit 7b.

[0051] In S111, the superimposing unit 8 causes the generated virtual viewpoint image to be displayed on the display unit 9. A virtual viewpoint image that mimics the reflection of a subject is generated by the above-mentioned processes in S101 to S111. If the virtual viewpoint image is a video, the processes in S101 to S110 are repeated for each video frame. For example, if the frame rate of the video is 60 fps (frames per second), the processes in S101 to S111 are repeated every 1 / 60 seconds.

[0052] In addition, if a reflected viewpoint image is generated when the virtual viewpoint represented by the virtual viewpoint information acquired from the virtual viewpoint input unit 5 is located on the back side of the reflecting surface (for example, below the reflecting floor surface), the following problem occurs. That is, the virtual viewpoint image generated in the superimposing unit 8 becomes an unnatural virtual viewpoint image in which the subject is reflected on the back side of the floor surface that is not imaged by the imaging unit 1. Therefore, when the virtual viewpoint is located in a specific space, the reflected viewpoint generating unit 6 does not output the reflected viewpoint information. The specific space is, for example, a space that is in contact with the back side of the reflecting surface among the three-dimensional spaces depicted in the virtual viewpoint image. The front and back of the reflecting surface are specified based on normal information of the polygon mesh included in the three-dimensional shape data of the background. That is, the opposite direction to the normal direction is the back side of the mesh. In the generation of the virtual viewpoint image in this embodiment, the three-dimensional shape data of the background is generated so that the front side of the reflecting surface is in contact with the space in which the subject exists. Therefore, it can be said that the specific space is a space in which no subject exists with respect to the reflecting surface.

[0053] When the virtual viewpoint is located in a specific space (a space where the subject does not exist relative to the reflecting surface), the reflected viewpoint generating unit 6 does not output reflected viewpoint information. Also, the first image generating unit 7a renders the subject and background excluding the reflecting surface. This prevents the second image generating unit 7b from generating a reflected image, and prevents the superimposing unit 8 from superimposing an unnatural reflected image.

[0054] The above-described method makes it possible to express the reflection of a subject in a virtual viewpoint image. In addition, for example, a ray tracing method can be used to generate an image that mimics the reflection of a subject, but this method requires calculating the path of a light ray for each pixel, which imposes a large processing load and makes high-speed processing difficult. In contrast, the above-described method uses a simple method of synthesizing two virtual viewpoint images, which reduces the processing load on the hardware and enables high-speed processing.

[0055] (Modification of the first embodiment) The second image generating unit 7b of the first embodiment described above renders only the inside of the reflection surface in S108 of FIG. 5. However, the present invention is not necessarily limited to this method. For example, the second image generating unit 7b renders the entire subject as seen from the reflection viewpoint as shown in FIG. 4(b) as a reflection viewpoint image. In addition, the superimposing unit 8 may be configured to extract an area corresponding to the reflection surface 200 from the reflection viewpoint image based on the reflection surface information 202 and superimpose it on the virtual viewpoint image. In this case, a virtual viewpoint image similar to that shown in the first embodiment can be obtained. However, the second image generating unit 7b can reduce the amount of processing by rendering only the inside of the reflection surface when generating the reflection viewpoint image.

[0056] In the above-mentioned first embodiment, the 3D shape data of the background includes one reflective surface 200, but there may be a plurality of reflective surfaces. For example, when there are two reflective surfaces, a floor surface and a wall surface, in the 3D shape data of the background, the reflective viewpoint generating unit 6 sequentially generates a reflective viewpoint corresponding to the floor surface and a reflective viewpoint corresponding to the wall surface. The second image generating unit 7b sequentially generates reflective viewpoint images corresponding to the two reflective viewpoints generated in the reflective viewpoint generating unit 6. The superimposing unit 8 superimposes the two generated reflective viewpoint images on the virtual viewpoint image. As a result, a virtual viewpoint image that imitates the reflection of the subject when there are a plurality of reflective surfaces is generated. When there are three or more reflective surfaces, the generation can be performed in the same manner. In this case, the configuration may be configured to include a plurality of second image generating units 7b, and each of the plurality of second image generating units 7b may generate a reflective viewpoint image in parallel.

[0057] In the first embodiment described above, the reflection surface is a flat surface, but the reflection surface may be a curved surface. When the reflection surface is a curved surface, the reflection viewpoint generating unit 6 and the second image generating unit 7b can generate the reflection viewpoint and the reflection viewpoint image in the same manner as the above-mentioned method after approximating the curved surface to a plurality of flat surfaces.

[0058] In the above-described first embodiment, the superimposing unit 8 is configured to output the generated virtual viewpoint image to the display unit 9 for display, but the present invention is not necessarily limited to this. For example, the superimposing unit 8 may be configured to output the generated virtual viewpoint image as an image data file.

[0059] Second embodiment The configuration of a generating system 21 for generating a virtual viewpoint image in this embodiment will be described with reference to FIG. 6. This system has a first generating device 601 and a second generating device 602. The first generating device 601 has a storage unit 5, a virtual viewpoint input unit 5, a reflected viewpoint generating unit 6, a first image generating unit 7a, and a second image generating unit 7b. The storage unit 4 may be a device connected to the outside of the first generating device 601. The second generating device 602 has a third image generating unit 7c and a superimposing unit 8. In this embodiment, processing units with the same names and symbols as those in the first embodiment have the same functions as those in the first embodiment.

[0060] In this embodiment, a first generating device 601 generates a virtual viewpoint image of only a subject, and a second generating device 602 generates a virtual viewpoint image by further rendering the background of the virtual viewpoint image generated by the first generating device 601. The process performed by the generating system 21 will be described with reference to FIG. 7.

[0061] The processes of S701 to S708 are processes performed by the first generating device 601. In S701, the virtual viewpoint input unit 5 receives an input operation for specifying the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint, which is performed on the operation unit 504. In addition, the virtual viewpoint input unit 5 transmits virtual viewpoint information determined based on the received input operation to the reflected viewpoint generating unit 6, the first image generating unit 7a, and the third image generating unit 7c.

[0062] In S702, the first image generation unit 7a acquires the captured image and three-dimensional shape data of the subject from the storage unit 4 as data necessary for generating a virtual viewpoint image based on the acquired virtual viewpoint information. In S703, the first image generation unit 7a acquires reflective surface information from the storage unit 4. In this embodiment, since the background is mainly rendered by the second generation device 602, it is sufficient that only information capable of identifying the position of the reflective surface is acquired here. In addition, the storage unit 4 may not be included in the first generation device 601, but may be included in a different device.

[0063] In S704, the first image generation unit 7a performs rendering of the subject based on the virtual viewpoint information and the data acquired from the storage unit 4. As a result, in S705, a virtual viewpoint image of only the subject is generated. The first image generation unit 7a transmits the generated virtual viewpoint image of only the subject to the superimposition unit 8.

[0064] In S706, the reflection viewpoint generating unit 6 generates reflection viewpoint information based on the reflection surface information and the virtual viewpoint information. The method of generating the reflection viewpoint information is the same as that of the first embodiment. In S707, the second image generating unit 7b renders the subject in the reflection surface based on the reflection surface information and the reflection viewpoint information. As a result, in S708, a reflection viewpoint image is generated. The method of generating the reflection viewpoint image is also the same as that of the first embodiment. The second image generating unit 7b transmits the generated reflection viewpoint image to the superimposing unit 8. The above is the process performed in the first generating device 601.

[0065] The processes of S709 to S711 are processes performed by the second generating device 602. In S709, the third image generating unit 7c obtains three-dimensional shape data of the background from the storage unit 4 based on the virtual viewpoint information, and generates a virtual viewpoint image of only the background obtained by rendering the background. The superimposing unit 8 superimposes the generated virtual viewpoint image of only the background and the reflected viewpoint image obtained from the second image generating unit 7b. When superimposing the reflected viewpoint image, the superimposing unit 8 performs image processing such as blurring and deformation of the reflected viewpoint image and changing the ratio of pixel values ​​at the time of superimposition, as in the first embodiment, before superimposing.

[0066] In S709, the superimposing unit 8 superimposes the virtual viewpoint image of only the subject generated in S705 on the virtual viewpoint image generated in S709. At this time, the superimposing unit 8 compares the distance information of the subject and the distance information of the background for each pixel, and draws the color of the one closer to the position of the virtual viewpoint. That is, when the distance between the virtual viewpoint and the subject is closer than the distance between the virtual viewpoint and the background, the superimposing unit 8 draws the pixel value of the virtual viewpoint image of only the subject. Also, when the distance between the virtual viewpoint and the subject is farther than the distance between the virtual viewpoint and the background, the superimposing unit 8 draws the pixel value of the virtual viewpoint image of only the background. As a result, the superimposing unit 8 generates a virtual viewpoint image that imitates the reflection of the subject as shown in FIG. 3(d). In S711, the superimposing unit 8 causes the display unit 9 to display the generated virtual viewpoint image.

[0067] With the configuration shown in this embodiment, it is possible to share the generation of the virtual viewpoint image and the reflected viewpoint image of the subject, and the virtual viewpoint image of the background and other image editing processes among different devices. This configuration has the advantage that, in video production, after capturing an image of the subject and generating it, it is possible to take time to create the background and edit the video.

[0068] In this embodiment, the second generating device 602 acquires the three-dimensional shape data of the background from the storage unit 4, but this is not limiting. The second generating device 602 can be configured to acquire the three-dimensional shape data from a storage device that is connected inside or outside the device and is different from the storage unit 4. In this configuration, the storage unit 4 does not need to store all of the three-dimensional shape data of the background, and only needs to store the three-dimensional shape data of the subject and the reflective surface information.

[0069] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions. [Explanation of symbols]

[0070] 10 Generator 7a First image generating unit 7b Second image generating unit 8. Overlapping section

Claims

1. a first acquisition means for acquiring a plurality of captured images acquired by imaging performed by a plurality of imaging devices, a first virtual viewpoint image generated based on a first virtual viewpoint, and information indicating surface irregularities on a predetermined surface included in a three-dimensional space; a second acquisition means for acquiring a second virtual viewpoint image generated based on the plurality of captured images and a second virtual viewpoint disposed on either side of the predetermined plane with respect to the first virtual viewpoint; a generating means for generating a third virtual viewpoint image, which corresponds to the first virtual viewpoint and includes an image showing a state in which an object located in an imaging space imaged by the plurality of imaging devices is reflected on the predetermined surface, based on the first virtual viewpoint image and the second virtual viewpoint image that has been subjected to a blurring process based on the information; A generating device comprising:

2. The generating device according to claim 1 , wherein the position of the second virtual viewpoint is a position that is symmetrical with respect to the predetermined plane to the position of the first virtual viewpoint.

3. The generating device according to claim 1 or 2, characterized in that the line of sight direction from the second virtual viewpoint is a line of sight direction that intersects with the specified surface at an intersection point between the specified surface and the line of sight direction from the first virtual viewpoint.

4. The generating device according to claim 1 , wherein the third virtual viewpoint image is generated by combining the first virtual viewpoint image and the second virtual viewpoint image.

5. The generating device according to claim 4 , wherein the first virtual viewpoint image and the second virtual viewpoint image are synthesized in a region of the third virtual viewpoint image corresponding to the predetermined surface.

6. The generating device described in claim 4 or 5, characterized in that the first virtual viewpoint image and the second virtual viewpoint image are synthesized based on a synthesis ratio determined based on at least one of the reflectance of the specified surface and a luminance specified by pixel values ​​of an area corresponding to the specified surface in the first virtual viewpoint image.

7. The generating device according to any one of claims 1 to 6, characterized in that the second acquisition means does not generate a second virtual viewpoint image when the first virtual viewpoint is located in a specific space, and generates a second virtual viewpoint image when the first virtual viewpoint is not located in the specific space.

8. The generating device according to claim 7 , wherein the specific space is a space different from the imaging space captured by the imaging device.

9. The generating device according to any one of claims 1 to 8, characterized in that the first virtual viewpoint image is generated based on three-dimensional shape data representing the three-dimensional shape of a first object existing in the imaging space and three-dimensional shape data representing the three-dimensional shape of a second object having the specified surface.

10. The generating device according to claim 9, characterized in that the second virtual viewpoint image is generated based on three-dimensional shape data representing the three-dimensional shape of the first object, and not based on three-dimensional shape data representing the three-dimensional shape of the second object.

11. The generating device according to any one of claims 1 to 10, characterized in that the third virtual viewpoint image is generated based on the first virtual viewpoint image and the second virtual viewpoint image that has been subjected to a specified processing based on surface information regarding the specified surface.

12. the surface information includes information representing a reflectance of the predetermined surface, The generating device according to claim 11 , wherein the predetermined processing is processing for changing pixel values ​​of the second virtual viewpoint image based on a reflectance included in the surface information.

13. The generating device according to claim 11 or 12, characterized in that the specified processing is a processing of changing pixel values ​​of the second virtual viewpoint image based on a luminance specified by pixel values ​​of an area corresponding to the specified surface in the first virtual viewpoint image.

14. A generating device according to any one of claims 1 to 13, characterized in that pixel values ​​of an area corresponding to the specified surface of the third virtual viewpoint image are determined based on pixel values ​​of an area corresponding to the specified surface of the second virtual viewpoint image and surface information regarding the specified surface.

15. The generating device described in claim 14, characterized in that the third virtual viewpoint image is generated by replacing pixel values ​​of an area corresponding to the specified surface of the first virtual viewpoint image with pixel values ​​of an area corresponding to the specified surface of the determined third virtual viewpoint image.

16. the second acquisition means, when there are a plurality of the predetermined surfaces, acquires a plurality of second virtual viewpoint images corresponding to the plurality of predetermined surfaces, respectively; The generating means generates a third virtual viewpoint image corresponding to the first virtual viewpoint based on the first virtual viewpoint image and the plurality of second virtual viewpoint images.

16. A generating device according to any one of claims 1 to 15.

17. a first acquisition step of acquiring a plurality of captured images acquired by imaging performed by a plurality of imaging devices, a first virtual viewpoint image generated based on a first virtual viewpoint, and information indicating surface irregularities on a predetermined surface included in a three-dimensional space; a second acquisition step of acquiring a second virtual viewpoint image generated based on the plurality of captured images and a second virtual viewpoint disposed on either side of the predetermined plane with respect to the first virtual viewpoint; a generating step of generating a third virtual viewpoint image based on the first virtual viewpoint image and the second virtual viewpoint image subjected to blurring processing based on the information, the third virtual viewpoint image including an image showing a state in which an object located in an imaging space imaged by the plurality of imaging devices is reflected on the predetermined surface, the third virtual viewpoint image corresponding to the first virtual viewpoint; A method for producing the same, comprising:

18. A program for causing a computer to function as the generating device according to any one of claims 1 to 16.

Images