Information processing device, information processing method, and program
The information processing device integrates three-dimensional shape information with image quality indicators to facilitate easy evaluation and visualization of virtual viewpoint images, addressing the challenge of quality assessment and device dependency in existing technologies.
Patent Information
- Application Number
- JP2022182798
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-11-15
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-11-15
AI Technical Summary
Existing technologies face challenges in evaluating the image quality of virtual viewpoint images, requiring repetitive generation and adjustment processes, and often necessitate dedicated devices, without a straightforward method to ensure high-quality output.
An information processing device that associates three-dimensional shape information with image quality indicators, allowing easy evaluation and visualization of virtual viewpoint image quality using general-purpose computer graphics tools, without the need for dedicated devices.
Enables easy evaluation of virtual viewpoint image quality, facilitating efficient generation of high-quality images by allowing users to specify optimal viewpoints based on image quality assessment, reducing processing time and device dependency.
Smart Images

Figure 0007595626000001 
Figure 0007595626000002 
Figure 0007595626000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and in particular to a technology for generating a virtual viewpoint image. [Background technology]
[0002] A technology that generates a virtual viewpoint image using images from multiple viewpoints obtained by synchronously shooting from multiple imaging devices installed in different positions has been attracting attention. This technology makes it possible to create powerful content that includes images from viewpoints that imaging devices could not reach in the past.
[0003] Such a virtual viewpoint image can be generated by generating a three-dimensional model of a subject based on a group of captured images, and photographing (i.e., rendering) this three-dimensional model with a virtual camera. Patent Document 1 proposes that, when generating a virtual viewpoint image, an image captured by an imaging device close to the virtual viewpoint is selected, and the color of the target object is determined depending on the viewpoint. [Prior art documents] [Non-patent literature]
[0004] [Patent Document 1] JP 2001-291116 A Summary of the Invention [Problem to be solved by the invention]
[0005] When generating a virtual viewpoint image, the position of the viewpoint is determined so that a desirable virtual viewpoint image can be obtained. At this time, it is also required to determine the position of the viewpoint so that a virtual viewpoint image with high image quality can be obtained. However, it was not easy to evaluate the image quality of the virtual viewpoint image at each viewpoint. For example, according to a method of determining the color of a target object depending on the viewpoint, it is possible to generate a high-definition virtual viewpoint image, but it was necessary to generate a virtual viewpoint image in order to judge the final image quality. Therefore, in order to generate a virtual viewpoint image with higher quality, it was necessary to repeat the generation of the virtual viewpoint image and the adjustment of the virtual viewpoint. Moreover, there was a problem that the generation of the virtual viewpoint image requires processing time, and in addition, a dedicated device is generally used to generate the virtual viewpoint image.
[0006] An object of the present disclosure is to provide a technology for generating a virtual viewpoint image that makes it easy to evaluate the quality of the resulting virtual viewpoint image when setting the virtual viewpoint. [Means for solving the problem]
[0007] An information processing device according to an embodiment includes the following configuration: generated by estimating a three-dimensional shape of the object based on a captured image of the object; An acquisition means for acquiring three-dimensional shape information indicating a three-dimensional shape of the subject, which is used for generating a virtual viewpoint image of the subject; In an image of the subject captured by an imaging device, Each of the multiple locations A determination means for determining image quality; The subject Each of the multiple locations Image quality information of , three-dimensional shape information of the object Corresponding to multiple positions of the three-dimensional shape of the object indicated by An output means for outputting the signal; The present invention is characterized by comprising: Effect of the Invention
[0008] In a technology for generating a virtual viewpoint image, when a virtual viewpoint is set, it is possible to easily evaluate the quality of the resulting virtual viewpoint image. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram showing an example of a schematic configuration of an image generating system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the hardware configuration of an information processing apparatus according to an embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of the functional configuration of an information processing device according to an embodiment. [Figure 4] FIG. 1 is a diagram showing an example of a flowchart of an information processing method according to an embodiment. [Diagram 5] FIG. 1 is a diagram showing an example of a flowchart of an image quality evaluation method according to an embodiment. [Figure 6] 11A and 11B are diagrams showing examples of textures applied to three-dimensional shape information; [Figure 7] 11A and 11B are diagrams showing examples of textures applied to three-dimensional shape information; [Figure 8] FIG. 4 is a diagram for explaining an area in which image quality is guaranteed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although the embodiments describe a plurality of features, not all of these features are necessarily essential, and the plurality of features may be combined in any manner. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.
[0011] An information processing device according to an embodiment associates information indicating the image quality of a subject in a captured image of the subject with three-dimensional shape information indicating the three-dimensional shape of the subject. When a virtual viewpoint image is generated based on captured images obtained by a plurality of imaging devices, the subject on the virtual viewpoint image is generated based on the captured image of the subject. Therefore, the information indicating the image quality in the captured image of the subject can be referred to as an index of the image quality of the generated virtual viewpoint image. In an embodiment, it is possible to roughly visualize the final image quality of a virtual viewpoint image on a general-purpose three-dimensional computer graphics (CG) tool without using a dedicated image generating device for generating a virtual viewpoint image.
[0012] Information indicating the image quality of a subject may take various forms. In the following, a case where texture is used as information indicating the image quality of a subject will be described first. In the following example, textures are added to multiple positions on the three-dimensional shape of the subject according to the image quality of each position in the captured image of the subject.
[0013] FIG. 1 shows an example of an image generating system 1 for generating a virtual viewpoint image. In the image generating system 1, a subject 3 in a shooting area 2 is captured by a plurality of imaging devices 10. A virtual viewpoint image is generated based on the captured images obtained by the plurality of imaging devices 10. The plurality of imaging devices 10 are arranged to surround the subject 3, for example, as shown in FIG. 1, and each captures the shooting area 2 from a different imaging position. The modeling device 20 generates three-dimensional shape information indicating the three-dimensional shape of the subject 3 using the captured images obtained by the imaging devices 10, and stores the information in the database 30. The rendering device 40 renders a virtual viewpoint image from a virtual viewpoint specified by the input device 50 using the three-dimensional shape information of the subject stored in the database 30. The display device 60 displays the virtual viewpoint image generated by the rendering device 40.
[0014] The information processing device 100 evaluates the image quality using the data stored in the database 30 as described below. Then, the information processing device 100 outputs information indicating the evaluated image quality to the database 30. Here, the information processing device 100 can output CG data conforming to a general-purpose CG format including the information indicating the evaluated image quality to the database 30. At this time, the user can specify a virtual viewpoint via the input device 50 while referring to the information indicating the image quality output by the information processing device 100. A general-purpose three-dimensional computer graphics (CG) tool may be operating in the input device 50. In this case, the input device 50 can generate a virtual viewpoint image from a specific viewpoint according to the CG data output by the information processing device 100 and display it to the user. The user can specify a virtual viewpoint of the virtual viewpoint image to be generated by the rendering device 40 while viewing the virtual viewpoint image displayed by the input device 50.
[0015] The multiple imaging devices 10 can continuously capture images in a synchronized manner. In this case, the image generating system 1 can generate three-dimensional shape information of a subject in time series, and can also generate virtual viewpoint images in time series, i.e., virtual viewpoint videos. In this configuration, the input device 50 can specify a virtual viewpoint whose position changes in time series. The virtual viewpoint that moves in this way is also called a virtual camera path or camera work.
[0016] 2 is a block diagram showing an example of the hardware configuration of a computer that can be used as an information processing device 100 according to an embodiment. The information processing device 100 has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a communication I / F 205, and a bus 206. The modeling device 20 and the input device 50 can also be realized using similar hardware. On the other hand, the information processing device 100 may be configured by a plurality of information processing devices connected via a network. Furthermore, one or more of the modeling device 20, the rendering device 40, and the input device 50 may be configured by a plurality of information processing devices connected via a network.
[0017] The ROM 202 is a memory that stores programs that do not require modification. The RAM 203 is a memory that temporarily stores programs or data supplied from the auxiliary storage device 204, and data supplied from the outside via the communication I / F 205. The auxiliary storage device 204 stores various data such as image data or audio data. The auxiliary storage device 204 may be a storage such as a hard disk drive. The communication I / F 205 is used for communication with an external device of the information processing device 100. For example, when the information processing device 100 is connected to an external device by wire, a communication cable is connected to the communication I / F 205. Also, when the information processing device 100 communicates wirelessly with an external device, the communication I / F 205 includes an antenna. The bus 206 connects each unit of the information processing device 100 and transmits information.
[0018] In this way, a processor such as the CPU 201 controls the entire information processing device 100 using a computer program or data stored in a memory such as the ROM 202 or the RAM 203. In this way, the functions of each processing unit of the information processing device 100 shown in FIG. 3 are realized. The information processing device 100 may have one or more dedicated hardware different from the CPU 201. In this case, at least a part of the processing performed by the CPU 201 can be performed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor). In addition, the information processing device 100 according to one embodiment may be realized by an information processing system configured by a plurality of information processing devices connected via a network, for example.
[0019] An example of the configuration of the information processing device 100 according to this embodiment is shown in Fig. 3. The information processing device 100 has a shape acquisition section 310, a view angle acquisition section 320, an evaluation section 330, a generation section 340, and an output section 350.
[0020] The shape acquisition unit 310 acquires three-dimensional shape information indicating the three-dimensional shape of the subject. As described above, this three-dimensional shape information is used to generate a virtual viewpoint image of the subject. The three-dimensional shape information acquired by the shape acquisition unit 310 may be three-dimensional shape information generated in the image generation system 1, or may be three-dimensional shape information prepared separately. The shape acquisition unit 310 may also acquire virtual three-dimensional shape information. When using three-dimensional shape information prepared separately, it is possible to perform conversion of the coordinate system, for example, scaling or adjustment of the position and orientation, in accordance with the image generation system 1. In the following, a case where the three-dimensional shape information is a mesh model will be described. However, the three-dimensional shape information is not limited to a mesh model, and may be other information indicating a three-dimensional shape, such as point cloud data. The three-dimensional shape information may also be volume data expressed by a collection of voxels.
[0021] The angle of view acquisition unit 320 acquires information indicating parameters of each of the multiple imaging devices 10 in the image generation system 1. The parameters may include external camera parameters indicating the position and orientation of the imaging device. The parameters may also include internal parameters indicating the focal length of the imaging device. Hereinafter, the parameters of the imaging device 10 acquired by the angle of view acquisition unit 320 are referred to as angle of view information. This angle of view information may indicate the angle of view or imaging range of the imaging device.
[0022] The evaluation unit 330 specifies the image quality of the subject in the image captured by the imaging device 10. The evaluation unit 330 can evaluate the image quality of each of the multiple positions of the subject in the image captured by the imaging device 10. Here, the evaluation unit 330 can determine the resolution of the subject in the image captured by the imaging device 10. The evaluation unit 330 can determine the resolution according to the angle of view information acquired by the angle of view acquisition unit 320. The evaluation unit 330 can perform such image quality evaluation for the captured images captured by each of the multiple imaging devices 10. In this way, the evaluation unit 330 can determine the degree of resolution at which a specific position of the subject is captured by each imaging device. Then, the evaluation unit 330 can evaluate the image quality of the subject based on the determined resolution. The evaluation unit 330 can perform such evaluation for each position of the subject. In the embodiment described below, the evaluation unit 330 performs such evaluation for each surface of a mesh model representing the three-dimensional shape of the subject. The image quality evaluated in this manner is considered to be close to the image quality of a specific position of the subject in the virtual viewpoint image generated by the rendering device 40.
[0023] The generating unit 340 generates information indicating the image quality of the subject according to the evaluation result of the image quality by the evaluating unit 330. The generating unit 340 can generate information indicating the image quality of the subject for each of a plurality of positions of the subject. Here, the generating unit 340 can generate information indicating the image quality of the subject based on the image quality of the subject in the captured images captured by each of the plurality of imaging devices 10. In addition, the generating unit 340 can generate visualized information indicating the image quality of the subject. The visualized information may be, for example, image information or shape information. For example, the information indicating the image quality of the subject generated by the generating unit 340 may be texture information of a three-dimensional model or shape information of a three-dimensional model.
[0024] In this embodiment, the generating unit 340 generates texture information indicating the texture of the three-dimensional model as information indicating the image quality of the object. The generating unit 340 can generate a texture according to the image quality of the object. In one embodiment, the texture includes repetition of elements. The elements can be any image. In this way, a repeating pattern can be used as the texture. The size of the elements is selected according to the image quality of the object. In the following embodiment, the scale of the repeating pattern is changed based on the evaluation value of the image quality.
[0025] The output unit 350 outputs information indicating the image quality of the subject and three-dimensional shape information of the subject. The output unit 350 may output the information indicating the image quality of the subject in association with the three-dimensional shape information. The output unit 350 can output information indicating the image quality of each of a plurality of positions of the subject in association with the three-dimensional shape information. The output unit 350 can also output visualized information indicating the image quality of the subject. The output unit 350 can output image information such as a texture generated by the generation unit 340 in association with the three-dimensional shape information acquired by the shape acquisition unit 310. Here, the output unit 350 may output information representing a three-dimensional model based on the information indicating the image quality of the subject. For example, the data output by the output unit 350 may be data conforming to a general-purpose format. The data conforming to the general-purpose format is, for example, a combination of OBJ data and MTL data, or data having both three-dimensional shape information and texture data, such as PLY data. The output unit 350 may be a display unit that displays information indicating the image quality of the subject in association with the three-dimensional shape information. Moreover, the output unit 350 may be a functional unit that outputs information to an external device.
[0026] Fig. 4 is a flowchart of an information processing method performed by the information processing device 100 according to an embodiment. Image quality is evaluated by the process according to Fig. 4. The process shown in Fig. 4 can be realized by the CPU 201 executing a control program stored in the ROM 202 and read out to the RAM 203.
[0027] In S401, the shape acquisition unit 310 acquires three-dimensional shape information. The shape acquisition unit 310 may read a file of three-dimensional shape information stored in the database 30. The shape acquisition unit 310 may also receive the three-dimensional shape information via a network. On the other hand, the shape acquisition unit 310 may generate the three-dimensional shape information by estimating the three-dimensional shape of the subject based on the captured images of the subject by the imaging device 10. In this case, the shape acquisition unit 310 can generate the three-dimensional shape of the subject using the captured images of the subject by the multiple imaging devices 10 based on the volume intersection method or the like.
[0028] In S402, the angle-of-view acquisition unit 320 acquires the angle-of-view information of the multiple imaging devices 10. The angle-of-view acquisition unit 320 can acquire the angle-of-view information of all of the imaging devices 10 in the image generation system 1.
[0029] In S403, the evaluation unit 330 calculates an evaluation value of the image quality for each of the multiple positions of the subject based on the three-dimensional shape information acquired in S401 and the angle of view information acquired in S402. The evaluation unit 330 can calculate an evaluation value indicating the resolution on the captured image for a specific position of the subject. Figure 5 shows an example of a processing flow for such image quality evaluation for each of the multiple surfaces representing the three-dimensional shape.
[0030] In S501, the evaluation unit 330 selects a face for which an evaluation value has not been calculated from among the faces of a mesh model that is three-dimensional shape information. For the target face selected in S501, an evaluation value is calculated by the following process. By repeating the processes of S501 to S505, evaluation values are calculated for all faces of the mesh model of the subject. Note that the evaluation unit 330 may subdivide one face before starting the process according to FIG. 5. With this configuration, it is possible to evaluate the image quality more precisely when it is expected that the image quality will vary within a single face due to the large face.
[0031] In S502, the evaluation unit 330 selects, from among the multiple image capture devices 10, an image capture device that has not been subject to evaluation of image quality related to the target surface.
[0032] In S503, the evaluation unit 330 determines whether the target surface selected in S501 is visible from the imaging device selected in S502. If the target surface is not visible, for example, if the target surface is hidden by another surface of the mesh model, the process proceeds to S505. If the target surface is visible, the process proceeds to S504. To determine whether the target surface is visible, a method using distance information from the target surface to the imaging device can be used. For example, the following method may be used to determine whether the target surface is visible. First, the distance between the three-dimensional position of the target surface and the three-dimensional position of the imaging device is calculated. Meanwhile, the pixel value of the pixel corresponding to the target surface in the depth map indicating the distance from the imaging device to the nearest subject is acquired. Then, the difference between the calculated distance between the three-dimensional position of the target surface and the three-dimensional position of the imaging device and the distance indicated by the pixel value of the pixel corresponding to the target surface in the depth map is calculated. If the calculated difference is smaller than a threshold value, it is determined to be visible. Also, if the calculated difference is larger than a threshold value, it is determined to be not visible (invisible).
[0033] In S504, the evaluation unit 330 calculates the resolution of a specific position of the subject corresponding to the target plane in the captured image captured by the imaging device selected in S502. The resolution can be calculated based on the angle of view information of the imaging device and the position of the target plane. The resolution can also be expressed by the physical size of the specific position of the subject per pixel of the captured image. For example, if the position of the target plane is Xt, the position of the target imaging device is Xc, and the focal length is f(pix), the resolution r (mm / pix) can be calculated according to the following formula. In the following formula, √(Xt-Xc) 2 represents the distance between Xt and Xc. r[mm / pix]=√(Xt-Xc) 2 / f
[0034] In S505, if there is an image pickup device that has not been evaluated for image quality on the target surface, the process returns to S502. If the image quality of all image pickup devices on the target surface has been evaluated, the process proceeds to S506. In this manner, the processes of S502 to S505 are repeated for each image pickup device.
[0035] In S506, the evaluation unit 330 determines an evaluation value of the target surface based on the resolution calculated for each of the multiple imaging devices 10 in S504. As an example, the evaluation unit 330 can use the highest resolution among the resolutions calculated for each of the multiple imaging devices 10 as the evaluation value of the target surface. In this way, the evaluation unit 330 can determine an evaluation value indicating the image quality (resolution in this example) of a specific position of the subject in the image of the subject captured by the imaging device 10. In S503, an imaging device in which the specific position of the subject is visible is selected from the multiple imaging devices 10. Therefore, in S506, the image quality of the specific position of the subject is evaluated based on the image of the subject captured by the selected imaging device.
[0036] In a viewpoint-dependent rendering method that determines the color of an object depending on the viewpoint, coloring of a specific position of a three-dimensional shape is performed by projecting an image captured by an imaging device where the specific position is visible. Therefore, the image quality of the image captured by the visible imaging device is related to the image quality of the virtual viewpoint image that is finally obtained. Furthermore, the best image quality for a specific position of the virtual viewpoint image is determined by the image quality of the image captured with the highest image quality among the images captured by the multiple images. For this reason, the evaluation value determined as described above can be used to evaluate the image quality of a specific position (i.e., the target surface) of the virtual viewpoint image.
[0037] On the other hand, the method of determining the evaluation value is not limited to the above method. Any index that indicates the image quality when the virtual viewpoint image is rendered can be used. For example, the evaluation value may be a statistical value such as an average value or a median value of the resolution calculated for each of the multiple image capture devices 10. Furthermore, when calculating the statistical value, the weighting of the resolution may be performed based on the angle between the normal of a specific position (e.g., the target surface) and the optical axis of the image capture device corresponding to the resolution. For example, the smaller the angle between the normal and the optical axis, the larger the weighting can be.
[0038] In S507, if there is a surface for which an evaluation value has not been calculated among the surfaces of the mesh model that is the three-dimensional shape information, the process returns to S501. If there is no surface for which an evaluation value has not been calculated, the calculation of the evaluation value according to FIG. 5 ends. In this manner, the processes of S501 to S507 are repeated for each surface. Note that, when the three-dimensional shape information is a point cloud or the like, an evaluation value may be calculated for each point, or an evaluation value may be calculated for each region consisting of several points among the multiple points that make up the point cloud. Similarly, when the three-dimensional shape information is volume data consisting of voxels, an evaluation value may be calculated for each voxel, or an evaluation value may be calculated for several voxel groups.
[0039] In S404, the generation unit 340 generates textures according to the image quality of each of the multiple positions of the subject based on the evaluation values calculated by the evaluation unit 330. The generated textures are used as indexes of image quality. Fig. 6(A) shows an example of elements included in the texture, and Fig. 6(B) shows an example of a mesh model to which the texture has been applied. For ease of explanation, a cubic mesh model is shown in Fig. 6(B), but the three-dimensional shape is not limited to a cube.
[0040] The texture includes a pattern that is a repetition of elements as shown in FIG. 6(A). The element shown in FIG. 6(A) is an image with "NG" written on it. In this embodiment, the generation unit 340 generates a texture that includes elements whose size depends on the evaluation value. Here, the higher the evaluation value, the smaller the elements the texture includes, and the lower the evaluation value, the larger the elements the texture includes. For example, the generation unit 340 can generate such a texture by scaling the texture coordinates by a magnification factor according to the following formula. An appropriate value can be set as the weight. Multiplier = weight / evaluation value
[0041] The generating unit 340 can generate information indicating the image quality of each position of the three-dimensional shape of the subject indicated by the three-dimensional shape information. For example, the generating unit 340 can generate a texture for each face of the mesh model according to the evaluation value.
[0042] In S405, the output unit 350 outputs information indicating the image quality of the subject and the three-dimensional shape information. The output unit 350 can output the information indicating the image quality of the subject in association with the three-dimensional shape information. Furthermore, the output unit 350 can associate each position of the three-dimensional shape of the subject indicated by the three-dimensional shape information with information indicating the image quality of the position generated by the generation unit 340.
[0043] For example, the output unit 350 can output three-dimensional shape information indicating the three-dimensional shape of the subject and the textures attached to each of a plurality of positions of the three-dimensional shape of the subject. In this embodiment, the output unit 350 attaches the texture generated by the generation unit 340 in S404 to the three-dimensional shape information acquired by the shape acquisition unit 310 in S401. Specifically, the output unit 350 can attach textures corresponding to the image quality of each position in the captured image of the subject by the imaging device to a plurality of positions of the three-dimensional shape of the subject indicated by the three-dimensional shape information. For example, the output unit 350 can attach textures corresponding to the evaluation values generated by the generation unit 340 to each surface of the mesh model. As described above, the output unit 350 can output information indicating the image quality of the subject and the three-dimensional shape information as CG data conforming to a general-purpose format.
[0044] The CG data thus output by output unit 350 is displayed by a general-purpose CG tool as shown in Fig. 6(B). Surface 601 is a surface with a low evaluation value, and a texture including large-sized elements is applied to it. On the other hand, surface 602 is a surface with a high evaluation value, and a texture including small-sized elements is applied to it.
[0045] As described above, the user can specify the virtual viewpoint of the virtual viewpoint image using the input device 50 or the like. The information processing device 100, the input device 50, or other devices may have a display control unit (not shown) that causes a display device such as the display device 60 to display the three-dimensional shape of the subject according to the three-dimensional shape information and the information indicating the image quality of the subject output by the output unit 350. The information processing device 100, the input device 50, or other devices may also have a reception unit (not shown) that receives the specification of the virtual viewpoint corresponding to the virtual viewpoint image from the user. With this configuration, the user can specify the virtual viewpoint while checking the information indicating the image quality of the subject. Then, the rendering device 40 generates the virtual viewpoint image based on the captured image of the subject, the three-dimensional shape information, and the specified virtual viewpoint.
[0046] For example, a camerawork creator of a virtual viewpoint image can create camerawork while referring to the textured CG data output by the output unit 350. Therefore, it is possible to know at the time of creating the camerawork whether or not the final image quality of a virtual viewpoint image seen from a specific virtual viewpoint will be broken. Thus, according to this embodiment, when setting a virtual viewpoint, it is possible to easily evaluate the quality of the obtained virtual viewpoint image.
[0047] Specifically, when the three-dimensional model is viewed from a virtual viewpoint at a position according to the created camerawork, if the elements on the texture are visible, the final image quality is likely to be broken. On the other hand, if the elements on the texture are not visible, the image quality is likely to be guaranteed. For example, in FIG. 6(B), the character "NG" cannot be read from the texture of the surface 602, so the image quality of the surface 602 of the virtual viewpoint image is likely to be sufficient. However, the character "NG" can be read from the texture of the surface 601, so the image quality of the surface 601 of the virtual viewpoint image may be broken. In this case, the camerawork creator can change the camerawork so that the virtual viewpoint is moved away from the surface 601, the field of view is made wider by changing the focal length, or the virtual viewpoint is moved toward the surface 602 so that the image quality is guaranteed. In this way, by using the information indicating the image quality of the subject output by the information processing device 100, it becomes easy to set the virtual viewpoint so that the image quality of the virtual viewpoint image is not broken without rendering the virtual viewpoint image.
[0048] The texture generated by the generating unit 340 in S404 does not need to be an image as shown in FIG. 6. Any texture that helps the user to judge the image quality can be used. For example, a texture having a lattice pattern as an element can be used. FIG. 7(A) shows an example of a texture having a lattice pattern. FIG. 7(B) shows an example of a three-dimensional model to which a texture is added. In this case, when the evaluation value is high, the lattice pattern becomes fine, and when the evaluation value is low, the lattice pattern becomes coarse. Therefore, when the image quality can be guaranteed, the lattice pattern looks like a moire and is difficult to see. On the other hand, it is possible to know that the image quality is not guaranteed in the part where the lattice is clearly visible.
[0049] So far, the case where texture is used as information indicating image quality has been described. By adding texture to the three-dimensional shape information, the user can determine whether the final image quality of the virtual viewpoint image is acceptable or not based on the visibility of the texture. On the other hand, the information indicating the image quality of the subject may be area information indicating an area set around the three-dimensional shape of the subject. The size of such an area can be set according to the image quality of the subject. For example, information indicating an area within a predetermined distance from the subject can be added to the three-dimensional shape, and the predetermined distance can be made shorter as the image quality of the subject is higher. Such information indicating an area can be used to know the range in which the user can ensure image quality.
[0050] In the following example, a region is set around each of a plurality of positions on the three-dimensional shape of the subject. Here, the region set around a specific position on the three-dimensional shape of the subject is set according to information indicating the image quality of the specific position in an image of the subject captured by an imaging device. The virtual viewpoint being within this region indicates that there is a high possibility that the image quality of the virtual viewpoint image for the specific position will be broken, and the virtual viewpoint being outside this region indicates that there is a high possibility that the image quality of the virtual viewpoint image for the specific position will be guaranteed.
[0051] Furthermore, the area set around the three-dimensional shape of the subject may be an area including the areas set around each of the multiple positions of the three-dimensional shape of the subject as described above (for example, a union of the areas). Information indicating such an inclusion area, for example, three-dimensional model information, may be added to the three-dimensional shape information of the subject. According to such three-dimensional model information, it is possible to determine an area where the quality of the virtual viewpoint image is not guaranteed or an area where the quality is guaranteed. By using such information, it becomes possible to uniformly judge the image quality of the virtual viewpoint image.
[0052] In such a modified example, the evaluation unit 330 can further acquire information indicating the focal length of the virtual viewpoint. A focal length acquisition unit (not shown) may acquire information indicating the focal length of the virtual viewpoint (or the virtual camera) from a user. The focal length acquisition unit (not shown) may also acquire information indicating the focal length of the designated virtual viewpoint from the input device 50.
[0053] When the focal length of the virtual viewpoint is long, even if the distance from the virtual camera to the subject is long, the resolution of the subject in the captured image required to ensure the image quality of the virtual viewpoint image is high. On the other hand, when the focal length of the virtual viewpoint is short, a wide-angle image is obtained, so the resolution of the subject in the captured image required to ensure the image quality of the virtual viewpoint image is low. Therefore, in this modification, the size of the area set around the three-dimensional shape of the subject can be set further according to the focal length of the virtual viewpoint corresponding to the virtual viewpoint image. In this way, a three-dimensional model indicating an area where the image quality of the virtual viewpoint image is not guaranteed is generated according to the focal length of the virtual viewpoint. Since this area changes according to the focal length of the virtual viewpoint, in this modification, the evaluation unit 330 acquires an approximate focal length in advance. However, it is not essential for the evaluation unit 330 to acquire the focal length. For example, the evaluation unit 330 may generate information indicating the image quality of the subject based on a predetermined value of the focal length.
[0054] The generating unit 340 calculates a distance at which image quality is ensured for each position of the subject based on the evaluation value determined by the evaluating unit 330 and the focal length of the virtual viewpoint. The generating unit 340 also generates a three-dimensional model indicating an area where the quality of the virtual viewpoint image is not ensured based on the calculated distance. FIG. 8 is a schematic diagram showing an example of such a model. FIG. 8 shows an area 90 indicating the three-dimensional shape of the subject. The generating unit 340 calculates a distance 91 at which image quality is ensured for each position of the subject (for example, each target surface). For example, the generating unit 340 can calculate the distance 91 by the distance d=α·e·f. Here, the image quality evaluation value e (mm / pix) is the evaluation value of the target surface calculated in the same manner as in S506. The focal length f (pix) is the focal length of the virtual viewpoint. An appropriate value can be set as the weight α.
[0055] The generating unit 340 can thus determine an area within the distance 91 calculated for each position from each position of the subject (e.g., each target surface). The generating unit 340 can then determine an area 92 including the area determined for each position as an area where image quality is not guaranteed. The virtual viewpoint being inside the area 92 indicates that the image quality of the virtual viewpoint image may be compromised. The generating unit 340 can generate a three-dimensional model that indicates the area thus determined. This three-dimensional model may be, for example, a semi-transparent model.
[0056] The output unit 350 can output three-dimensional shape information indicating the three-dimensional shape of the subject and the three-dimensional shape indicating the area set around the three-dimensional shape of the subject. For example, the output unit 350 can output the three-dimensional model thus generated in association with the three-dimensional shape information of the subject. As an example, the output unit 350 can output CG data conforming to a general-purpose format including both the three-dimensional model of the subject and the semi-transparent model generated by the generation unit 340.
[0057] The CG data thus output by output unit 350 is displayed by a general-purpose CG tool as shown in Fig. 6(B). Surface 601 is a surface with a low evaluation value, and a texture including large-sized elements is applied to it. On the other hand, surface 602 is a surface with a high evaluation value, and a texture including small-sized elements is applied to it.
[0058] The user can specify a virtual viewpoint according to information indicating an area where the quality of the virtual viewpoint image is not guaranteed, such as the above-mentioned semi-transparent model. This makes it easy to specify a virtual viewpoint so that the image quality of the virtual viewpoint image is guaranteed. For example, a camerawork creator of a virtual viewpoint image can operate a virtual camera while viewing the CG data output by the output unit 350. At this time, when the virtual camera enters the area 92, the area 92 becomes invisible from the virtual camera. Also, when the virtual camera is outside the area 92, the area 92 is visible from the virtual camera. Therefore, by operating the virtual camera so that the area 92 is visible, it is possible to create camerawork so that the image quality of the virtual viewpoint image is guaranteed.
[0059] The information indicating the image quality of the subject is not limited to the above. For example, the information indicating the image quality of the subject does not need to be information visualized as a three-dimensional model (for example, a texture or a semi-transparent model). For example, the output unit 350 may output the information indicating the image quality of the subject as meta information in association with the three-dimensional shape information. Such information indicating the image quality of the subject may be, for example, the above-mentioned resolution r for each imaging device on each surface of the subject, the above-mentioned evaluation value for each surface of the subject, or the distance 91 at which the image quality is guaranteed for each surface of the subject.
[0060] In this case, a device for specifying a virtual viewpoint such as the input device 50 (for example, a camerawork production device) may have a function similar to that of the generation unit 340. For example, this device may display information indicating the image quality of the subject (for example, a texture or a semi-transparent model) to the user together with the three-dimensional model according to the meta information. On the other hand, this device may determine whether or not the image quality is guaranteed in a virtual viewpoint image from a specified virtual viewpoint according to the meta information, and notify the user of the determination result. For example, this device can determine whether or not the image quality of a virtual viewpoint image is guaranteed based on a distance 91 at which the image quality is guaranteed for each surface of the subject, which is indicated in the meta information, and the distance from the virtual viewpoint to each surface of the subject.
[0061] In addition, even if the shape acquisition unit 310 acquires three-dimensional shape information other than the mesh model, the output unit 350 can associate information indicating the image quality of the subject with the three-dimensional shape information. For example, when the shape acquisition unit 310 acquires point cloud data, the generation unit 340 can determine an area within a distance 91 from each point. In addition, the output unit 350 can associate information indicating an area including each area with the three-dimensional shape information. In addition, the output unit 350 can also impart the above-mentioned texture to the mesh model generated based on the point cloud data. In addition, the output unit 350 does not need to output the three-dimensional shape information of the subject and the information indicating the image quality of the subject in association with each other. For example, the output unit 350 may separately output information that associates the three-dimensional shape information of the subject with the information indicating the image quality of the subject. In addition, the output unit 350 may output the three-dimensional shape information of the subject and the information indicating the image quality of the subject at different timings. For example, the information that associates the three-dimensional shape information of the subject with the information indicating the image quality of the subject may be a time code.
[0062] So far, the information processing device 100 has been described as a part of the image generation system 1. However, the information processing device 100 may be a separate device separate from the image generation system 1. Also, a device for specifying a virtual viewpoint such as the input device 50 may have the functions of the information processing device 100. For example, the input device 50 may have each processing unit shown in FIG. 3. In this case, the input device 50 may display information (e.g., texture or semi-transparent model) indicating the image quality of the subject generated by the generation unit 340 to the user. On the other hand, the input device 50 may determine whether or not the image quality is guaranteed in the virtual viewpoint image from the specified virtual viewpoint according to the information indicating the image quality of the subject generated by the generation unit 340, and notify the user of the determination result.
[0063] (Other Examples) The present technology can also be realized by a process in which a program for realizing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for realizing one or more functions.
[0064] The disclosure of this specification includes the following imaging device, imaging system, method, and program.
[0065] (Item 1) An acquisition means for acquiring three-dimensional shape information indicating a three-dimensional shape of the subject, which is used for generating a virtual viewpoint image of the subject; a determination unit for determining an image quality of the subject in an image captured by an imaging device; an output means for outputting information indicating the image quality of the subject and three-dimensional shape information of the subject; An information processing device comprising:
[0066] (Item 2) 2. The information processing device according to item 1, wherein the output means outputs visualized information indicating an image quality of the subject.
[0067] (Item 3) 3. The information processing device according to item 1 or 2, wherein the output means outputs information representing a three-dimensional model based on information indicating image quality of the subject.
[0068] (Item 4) 4. The information processing device according to any one of items 1 to 3, wherein the information indicating the image quality of the subject is texture information indicating a texture of a three-dimensional model.
[0069] (Item 5) 5. The information processing device according to item 4, wherein the texture includes repetition of elements, and the size of the elements is selected according to the image quality of the subject.
[0070] (Item 6) the output means outputs three-dimensional shape information indicating a three-dimensional shape of the subject and textures applied to each of a plurality of positions on the three-dimensional shape of the subject; The information processing device according to any one of items 1 to 5, characterized in that the texture to be applied to a specific position of the three-dimensional shape of the subject is determined according to the image quality of the specific position in the image of the subject captured by the imaging device.
[0071] (Item 7) the output means outputs area information indicating an area set around a three-dimensional shape of the subject as information indicating an image quality of the subject; 4. The information processing device according to any one of items 1 to 3, wherein the size of the region is set according to the image quality of the subject.
[0072] (Item 8) 8. The information processing device according to item 7, wherein the size of the area is set further depending on a focal length of a virtual viewpoint corresponding to the virtual viewpoint image.
[0073] (Item 9) the region set around the three-dimensional shape of the subject includes regions set around each of a plurality of positions of the three-dimensional shape of the subject; Item 9. The information processing device according to item 7 or 8, characterized in that the area set around the specific position of the three-dimensional shape of the subject is set according to information indicating the image quality of the specific position in the image of the subject captured by the imaging device.
[0074] (Item 10) The output means outputs three-dimensional shape information indicating a three-dimensional shape of the subject and a three-dimensional shape indicating the area set around the three-dimensional shape of the subject. The information processing device described in any one of items 7 to 9.
[0075] (Item 11) 11. The information processing device according to any one of items 1 to 10, wherein the specifying means specifies the image quality of the subject based on angle-of-view information indicating an angle of view of the imaging device.
[0076] (Item 12) 12. The information processing device according to any one of items 1 to 11, wherein the determination means determines the image quality of the subject based on a resolution of the subject in an image of the subject captured by the imaging device.
[0077] (Item 13) The specifying means specifies image quality at each of a plurality of positions of the subject in an image captured by the imaging device of the subject, 13. The information processing device according to any one of items 1 to 12, wherein the output means outputs information indicating image quality at each of a plurality of positions of the subject in association with the three-dimensional shape information.
[0078] (Item 14) The information processing device according to any one of items 1 to 13, wherein the identification means identifies the image quality of the specific position of the subject based on an image of the subject captured by an imaging device selected from a plurality of imaging devices in which the specific position of the subject is visible.
[0079] (Item 15) 15. The information processing device according to any one of items 1 to 14, wherein the acquisition means generates the three-dimensional shape information by estimating a three-dimensional shape of the subject based on a captured image of the subject.
[0080] (Item 16) a display control means for causing a display device to display the three-dimensional shape of the subject according to the three-dimensional shape information and information indicating image quality of the subject; and a receiving unit for receiving, from a user, a designation of a virtual viewpoint corresponding to the virtual viewpoint image, 16. The information processing device according to any one of items 1 to 15, characterized in that the virtual viewpoint image is generated based on the captured image of the subject, the three-dimensional shape information, and the specified virtual viewpoint.
[0081] (Item 17) An acquisition means for acquiring three-dimensional shape information indicating a three-dimensional shape of a subject; an applying means for applying a texture to a plurality of positions of the three-dimensional shape of the subject indicated by the three-dimensional shape information according to image quality of each position in an image of the subject captured by an imaging device; An information processing device comprising:
[0082] (Item 18) An acquisition means for acquiring three-dimensional shape information indicating a three-dimensional shape of a subject; a generating means for generating area information indicating an area within a range corresponding to an image quality of the subject in an image captured by an imaging device of the subject from the three-dimensional shape of the subject indicated by the three-dimensional shape information; An information processing device comprising:
[0083] (Item 19) An information processing method performed by an information processing device, acquiring three-dimensional shape information indicating a three-dimensional shape of the subject, the three-dimensional shape information being used to generate a virtual viewpoint image of the subject; specifying an image quality of the subject in an image captured by an imaging device; outputting information indicating an image quality of the object and three-dimensional shape information of the object; An information processing method comprising:
[0084] (Item 20) 19. A program for causing a computer to function as the information processing device according to any one of items 1 to 18.
[0085] The present disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present application. [Explanation of symbols]
[0086] 100: information processing device, 310: shape acquisition unit, 320: angle of view acquisition unit, 330: evaluation unit, 340: generation unit, 350: output unit
Claims
1. An acquisition means for acquiring three-dimensional shape information indicating a three-dimensional shape of the subject, the three-dimensional shape information being used to generate a virtual viewpoint image of the subject, the three-dimensional shape information being generated by estimating a three-dimensional shape of the subject based on a captured image of the subject; a specifying unit for specifying image quality at each of a plurality of positions of the subject in an image captured by an imaging device of the subject; an output means for outputting information indicating image quality at each of a plurality of positions of the subject in association with a plurality of positions of the three-dimensional shape of the subject indicated by the three-dimensional shape information of the subject; An information processing device comprising:
2. The information processing apparatus according to claim 1 , wherein the output means outputs visualized information indicating the image quality of the subject.
3. The information processing apparatus according to claim 1 , wherein said output means outputs information representing a three-dimensional model based on information indicating an image quality of said subject.
4. The information processing apparatus according to claim 1 , wherein the information indicating the image quality of the subject is texture information indicating a texture of a three-dimensional model.
5. The information processing apparatus according to claim 4 , wherein the texture includes repetition of elements, and the size of the elements is selected according to the image quality of the subject.
6. the output means outputs three-dimensional shape information indicating a three-dimensional shape of the subject and textures applied to each of a plurality of positions on the three-dimensional shape of the subject; The information processing apparatus according to claim 4 , wherein the texture to be applied to the specific position of the three-dimensional shape of the subject is determined according to the image quality of the specific position in the image of the subject captured by the imaging device.
7. the output means outputs area information indicating an area set around a three-dimensional shape of the subject as information indicating an image quality of the subject; The information processing apparatus according to claim 1 , wherein the size of the region is set in accordance with the image quality of the subject.
8. The information processing apparatus according to claim 7 , wherein the size of the region is set further depending on a focal length of a virtual viewpoint corresponding to the virtual viewpoint image.
9. the region set around the three-dimensional shape of the subject includes regions set around each of a plurality of positions of the three-dimensional shape of the subject; The information processing device according to claim 7, characterized in that the area set around the specific position of the three-dimensional shape of the subject is set according to information indicating the image quality of the specific position in the image of the subject captured by the imaging device.
10. 8. The information processing apparatus according to claim 7, wherein the output means outputs three-dimensional shape information indicating a three-dimensional shape of the subject and a three-dimensional shape indicating the area set around the three-dimensional shape of the subject.
11. The information processing apparatus according to claim 1 , wherein the specifying unit specifies the image quality of the subject on the basis of information about an angle of view of the imaging device.
12. The information processing apparatus according to claim 1 , wherein the specifying unit specifies an image quality of the subject based on a resolution of the subject in an image of the subject captured by the imaging device.
13. The information processing device according to claim 1 , wherein the specifying means specifies the image quality of the specific position of the subject based on an image of the subject captured by an imaging device selected from a plurality of imaging devices in which the specific position of the subject is visible.
14. The information processing apparatus according to claim 1 , wherein the acquisition means generates the three-dimensional shape information by estimating a three-dimensional shape of the subject based on a captured image of the subject.
15. a display control means for causing a display device to display the three-dimensional shape of the subject according to the three-dimensional shape information and information indicating image quality of the subject; and a receiving unit for receiving, from a user, a designation of a virtual viewpoint corresponding to the virtual viewpoint image, The information processing apparatus according to claim 1 , wherein the virtual viewpoint image is generated based on the captured image of the subject, the three-dimensional shape information, and the designated virtual viewpoint.
16. An acquisition means for acquiring three-dimensional shape information indicating a three-dimensional shape of the subject, the three-dimensional shape information being used to generate a virtual viewpoint image of the subject, the three-dimensional shape information being generated by estimating a three-dimensional shape of the subject based on a captured image of the subject; an applying means for applying textures to a plurality of positions of the three-dimensional shape of the subject indicated by the three-dimensional shape information according to image quality of each of a plurality of positions in an image of the subject captured by an imaging device; An information processing device comprising:
17. An acquisition means for acquiring three-dimensional shape information indicating a three-dimensional shape of the subject, the three-dimensional shape information being generated by estimating the three-dimensional shape of the subject based on a captured image of the subject; a generating means for generating area information indicating an area within a range according to image quality of each of a plurality of positions of the three-dimensional shape of the subject indicated by the three-dimensional shape information, in an image of the subject captured by an imaging device; An information processing device comprising:
18. An information processing method performed by an information processing device, acquiring three-dimensional shape information indicating a three-dimensional shape of the subject, the three-dimensional shape information being used for generating a virtual viewpoint image of the subject, the three-dimensional shape information being generated by estimating a three-dimensional shape of the subject based on a captured image of the subject; specifying image quality at each of a plurality of positions of the subject in an image captured by an imaging device of the subject; outputting information indicating image quality at each of the plurality of positions of the subject in association with the plurality of positions of the three-dimensional shape of the subject indicated by the three-dimensional shape information of the subject; An information processing method comprising:
19. A program for causing a computer to function as the information processing device according to any one of claims 1 to 17.
Citation Information
Patent Citations
Device and method for generating three-dimensional image and program providing medium
JP2001291116A
Method for evaluating three-dimensional shape model and method and device for generating the model
JP2003115042A
Apparatus and method for image processing, and computer program
JP2003337953A
Image processing apparatus, image processing method, and program
JP2020166652A
Provision device, control method, and program
JP2021033525A
