Information processing device, video generation method and program

Abstract

A score map generation unit (83) (image quality prediction unit) provided to an information processing device (10a) predicts, for a 3D model (90M) (3D object), a relationship between the position of a virtual viewpoint (V) from which the 3D model (90M) is viewed and the image quality of a virtual image (J) acquired when the 3D model (90M) is viewed from the position of the virtual viewpoint (V).

Description

Technical Field

[0001] The present disclosure relates to an information processing device, a video generating method, and a program, and more particularly, to an information processing device, a video generating method, and a program capable of predicting the image quality of a virtual image obtained when a 3D object is captured from an arbitrary direction by a virtual camera. Background Art

[0002] In the related art, the following technology has been proposed: a 3D object is generated in a viewing space by using information obtained by sensing a real 3D space (for example, a multi-view video obtained by imaging an object from different viewpoints), and a video that looks as if the object exists in the viewing space is generated. In addition, a technology for synthesizing a virtual video obtained when an object is imaged from an arbitrary viewpoint position has been proposed (for example, Patent Document 1).

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: JP 4671873 B2 Summary of the invention

[0006] Technical issues

[0007] However, in Patent Document 1, the image quality of a virtual image captured from an arbitrary direction cannot be predicted.

[0008] The present disclosure proposes an information processing device, a video generating method, and a program capable of predicting the image quality of a virtual image obtained when a 3D object is imaged from a virtual viewpoint.

[0009] Solution to the problem

[0010] To solve the above problems, an information processing device according to an embodiment of the present disclosure includes: an image quality prediction unit, which predicts, for a 3D object, the relationship between the position of a virtual viewpoint from which the 3D object is viewed and the image quality of a virtual image obtained when the 3D object is viewed from the position of the virtual viewpoint. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram showing an example of a basic configuration of an information processing apparatus according to the first embodiment.

[0012] Figure 2 3D model generation apparatus generates a 3D model.

[0013] Figure 3 is a diagram illustrating a method in which a camera path prediction apparatus generates a virtual image obtained by imaging a single 3D model from a virtual viewpoint.

[0014] Figure 4 is a diagram showing an example in which a camera path prediction apparatus generates a virtual image obtained by imaging a single 3D model from a virtual viewpoint.

[0015] Figure 5 : is a flowchart showing an example of a process in which a camera path prediction device generates a virtual image.

[0016] Figure 6 is a hardware block diagram showing an example of the hardware configuration of the information processing apparatus according to the first embodiment.

[0017] Figure 7 is a functional block diagram showing an example of a functional configuration of the information processing apparatus according to the first embodiment.

[0018] Figure 8 This is a diagram for explaining the layout of a real imaging device for imaging a subject.

[0019] Fig. 9 is a diagram showing an example of calculating an image quality score.

[0020] Fig.10 is a diagram showing an example of the layout of a real imaging device.

[0021] Fig.11 is a diagram for explaining an example of generating a score map.

[0022] Fig.12 is a diagram showing a method of generating a camera path.

[0023] Fig.13 is a first diagram illustrating an example of a user interface for predicting a camera path.

[0024] Fig.14 is a second diagram illustrating an example of a user interface for predicting a camera path.

[0025] Fig.15 is a flowchart showing an example of the flow of processing performed by the camera path generation apparatus.

[0026] Fig.16 is a block diagram showing an example of a basic configuration of an information processing apparatus according to the second embodiment.

[0027] Fig.17 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to the second embodiment.

[0028] Fig.18 It is a diagram for explaining the operation of the mobile terminal according to the second embodiment.

[0029] Fig.19 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to the third embodiment.

[0030] Fig. 20 It is a diagram for explaining the operation of the mobile terminal according to the third embodiment.

[0031] Fig.21 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to the fourth embodiment.

[0032] Fig. 22 is a functional block diagram showing an example of a functional configuration of a 3D model generating device according to the fifth embodiment.

[0033] Fig.23 It is a diagram for explaining the operation of the 3D model generating device according to the fifth embodiment. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following embodiments, the same parts are denoted by the same reference numerals, and repeated descriptions are omitted.

[0035] Furthermore, the present disclosure will be described in the order of the following items.

[0036] 1. First Implementation

[0037] 1-1. Description of prerequisites - Generation of 3D models

[0038] 1-2. Description of Prerequisites - Generation of Virtual Viewpoint Video of a Single 3D Model

[0039] 1-3. Description of the hardware configuration of the information processing device

[0040] 1-4. Description of functional configuration of information processing device

[0041] 1-5. Description of the method for calculating image quality score

[0042] 1-6. Description of the method for generating a score map

[0043] 1-7. Description of the method for generating a camera path

[0044] 1-8. Description of the camera path generation auxiliary method

[0045] 1-9. Description of the Processing Flow Executed by the Camera Path Generation Device

[0046] 1-10. Effects of the First Embodiment

[0047] 2. Second Implementation

[0048] 3. Third Implementation

[0049] 4. Fourth Implementation

[0050] 5. Fifth Implementation

[0051] (1. First Embodiment)

[0052] The information processing device 10 a according to the first embodiment of the present disclosure is a device that generates an imaging device path that provides high image quality when 3D models of a plurality of subjects are imaged simultaneously. Figure 1 1 is a block diagram showing an example of a basic configuration of the information processing apparatus according to the first embodiment. The information processing apparatus 10 a includes a 3D model generation device 20 and a camera path generation device 30 .

[0053] The 3D model generating device 20 independently generates 3D models of a plurality of subjects.

[0054] When 3D models of multiple subjects are set in the same space and are imaged simultaneously, the camera path generation device 30 generates, for example, a camera path capable of maintaining an image quality of a predetermined value or more, ie, a movement trajectory of a virtual viewpoint.

[0055] Hereinafter, before describing the details of the information processing device 10 a , necessary prerequisite matters will be explained.

[0056] [1-1. Description of prerequisites - Generation of 3D models]

[0057] Figure 2 3D model generation apparatus generates a 3D model.

[0058] like Figure 2 As shown, the generation of the 3D model includes the process of imaging the subject 90 through a plurality of imaging devices 70 ( 70 a , 70 b , 70 c ) and generating a 3D model 90M having 3D information on the subject 90 through 3D modeling.

[0059] Specifically, Figure 2 As shown, a plurality of imaging devices 70 are disposed inwardly outside a subject 90 so as to surround the subject 90 existing in the real world. Figure 2 An example is shown in which the number of imaging devices is three and imaging devices 70a, 70b, and 70c are arranged around the subject 90. Note that Figure 2 In the example, a person who performs a predetermined operation is the subject 90. Furthermore, the number of imaging devices 70 is not limited to three, and a larger number of imaging devices may be provided.

[0060] 3D modeling is performed using a plurality of viewpoint images synchronously captured by volume imaging by the three imaging devices 70a, 70b, and 70c from different viewpoints, and a 3D model 90M of the subject 90 is generated in units of video frames of the three imaging devices 70a, 70b, and 70c.

[0061] The 3D model 90M is a model having 3D information about the subject 90. Note that the 3D model 90M is an example of a 3D object in the present disclosure. The 3D model 90M includes mesh data called a polygon mesh in which geometric information about the subject 90 is expressed by connections between vertices, and texture information and depth information (distance information) corresponding to each polygon mesh. Note that the information included in the 3D model 90M is not limited to this and may include other information. Note that the depth information about the subject 90 is calculated based on, for example, the parallax of the same area of ​​the subject 90 detected from an image captured by an adjacent imaging device. Note that the depth information can be obtained by installing a sensor including a distance measuring mechanism near the imaging device and measuring the distance to the subject 90 by the sensor.

[0062] The 3D model 90M is subjected to so-called texture mapping in which a texture indicating the color, pattern, or texture of a mesh is pasted according to the mesh position. In texture mapping, it is desirable to paste a texture that is view-dependent to the viewpoint position to improve the realism of the 3D model 90M. Therefore, when the 3D model 90M is imaged from an arbitrary virtual viewpoint, the texture changes according to the viewpoint position, so that a virtual image with higher image quality can be obtained (see Figure 4 ). However, since the amount of calculation increases, a view independent texture may be attached to the 3D model 90M.

[0063] The content data including the read 3D model 90M is sent to the device on the playback side and played back. The content data including the 3D model 90M is played back, and the 3D model 90M is presented, thereby displaying a 3D shape video on the viewing device of the viewer.

[0064] exist Figure 2 In the example of , a mobile terminal 80 such as a smartphone or a tablet terminal is used as a viewing device. That is, an image including a 3D model 90M is displayed on a display screen 84 of the mobile terminal 80 .

[0065] [1-2. Description of Prerequisites - Generation of Virtual Viewpoint Video of a Single 3D Model]

[0066] Figure 31 is a diagram showing a method in which a camera path generation apparatus generates a virtual image obtained by imaging a single 3D model from a virtual viewpoint. Figure 3 , the imaging device 70 (70a, 70b, 70c) is an imaging device for creating a 3D model 90M of a subject 90. Hereinafter, the imaging device 70 (70a, 70b, 70c) is referred to as a real camera 70. Note that the real camera 70 is an example of an imaging unit in the present disclosure. In various applications using the 3D model 90M, it is expected that the generated 3D model 90M can be imaged from various directions. Therefore, a technology for generating an image (virtual image) obtained by imaging the 3D model 90M from a position (virtual viewpoint) different from the position of the real camera 70 has been developed.

[0067] For example, in Figure 3 , it is assumed that a virtual image J1 (not shown) obtained when a virtual camera 72a placed at a virtual viewpoint V1 captures an image of a 3D model 90M is generated. The virtual image J1 can be generated by interpolating images of the 3D model 90M captured by the real camera 70a and the real camera 70c placed near the virtual camera 72a. That is, depth information about the subject 90 is calculated by associating the image of the 3D model 90M captured by the real camera 70a with the image of the 3D model 90M captured by the real camera 70c. Then, by projecting the texture of the area corresponding to the calculated depth information onto the virtual camera 72a, a virtual image J1 of the 3D model 90M (subject 90) viewed from the virtual camera 72a can be generated.

[0068] Similarly, a virtual image J2 (not shown) of the 3D model 90M viewed from the virtual camera 72b at the virtual viewpoint V2 placed near the real camera 70b and the real camera 70c can be generated by interpolating the image of the 3D model 90M captured by the real camera 70b and the image of the 3D model 90M captured by the real camera 70c. Note that, hereinafter, the virtual cameras 72a and 72b are collectively referred to as the virtual camera 72. In addition, the virtual viewpoints V1 and V2 are collectively referred to as virtual viewpoints V, and the virtual images J1 and J2 are collectively referred to as virtual images J (not shown).

[0069] Since the virtual image J obtained in this way is an image generated by interpolation, the image quality is deteriorated compared with the image of the 3D model 90M captured by the real camera 70 .

[0070] That is, when the virtual camera 72 is placed at the same position as the real camera 70 and captures images under the same conditions as the real camera 70, the image quality of the virtual image J obtained when the virtual camera 72 moves around the 3D model 90M is optimal. On the other hand, the image quality of the virtual image J generated when the virtual camera 72 is placed at a position away from the real camera 70 is lower than the image quality of the image captured by the real camera 70.

[0071] At this time, the degree of image quality degradation is determined by the arrangement position and optical parameters of the virtual camera device 72 and the arrangement position and optical parameters of the real camera device 70, and generally, the greater the difference between the arrangement position and optical parameters of the virtual camera device 72 and the arrangement position and optical parameters of the real camera device 70, the greater the degradation of the image quality.

[0072] Next, we will refer to Figure 4 An example of generating a virtual image J is described. Figure 4 is a diagram showing an example in which the camera path generation apparatus generates a virtual image obtained by imaging a single 3D model from a virtual viewpoint.

[0073] Assume that nine real camera devices 70 image the subject 90 from different directions to generate a 3D model 90M. Figure 4 The nine images Ia to Ig shown are captured by respective real camera devices 70. At this point, the generation of the image by placing Figure 4 A method of capturing a virtual image Ja of a 3D model 90M by the virtual camera 72a at the position shown.

[0074] In order to simplify the description, a case where a virtual image of the polygon P of the 3D model 90M is generated is described as an example. Note that it is assumed that the 3D model 90M is placed at a known position in an arbitrarily set world coordinate system XYZ.

[0075] First, the positional relationship between the 3D model 90M and the virtual camera 72a is obtained. Note that it is assumed that the arrangement position (Xv, Yv, Zv) (i.e., the virtual viewpoint) of the virtual camera 72a is known. Since the position of the polygon P of the 3D model 90M in the world coordinate system XYZ is known, the positional relationship between the polygon P and the virtual camera 72a (i.e., Figure 4 The vector B) shown is uniquely determined.

[0076] Next, for each of the nine images Ia to Ig, an image candidate to be used when generating a virtual image is selected. Specifically, it is determined whether a polygon P is shown in each of the images Ia to Ig. Whether the polygon P is shown can be determined based on determining whether occlusion occurs and determining whether the depth information about the area corresponding to the polygon P of each image matches the actual depth. Figure 4 In the example of , it is assumed that the six images Ib, Ic, Ie, If, Ih, and Ii include the polygon P. On the other hand, it is assumed that the polygon P is not shown in the images Ia, Id, and Ig.

[0077] Next, the position of each real camera 70 that captured an image candidate used when generating a virtual image is compared with the position of the virtual camera 72a, and the real cameras 70 are sorted in order of proximity to the virtual camera 72a.

[0078] Various scales may be used to determine the proximity between the real camera 70 and the virtual camera 72a. For example, it may be determined that the closer the real camera 70 and the virtual camera 72a are, the smaller the angle formed by the line segment connecting the polygon P shown in the image captured by each real camera 70 and the polygon P of the 3D model 90M and the vector B.

[0079] Furthermore, it can be determined that as the distance between the real camera 70 and the virtual camera 72a decreases, the real camera 70 and the virtual camera 72a are closer.

[0080] Furthermore, it can be determined that the real camera 70 and the virtual camera 72a are closer as the imaging magnification of the polygon P is closer. Specifically, it can be determined that the real camera 70 and the virtual camera 72a are closer as the image size p of the subject 90 calculated by expression (1) based on the subject size h of the subject 90, the distance D from each camera to the subject 90, and the focal length f of the lens used for imaging is closer.

[0081] p=h*(f / D) (1)

[0082] Furthermore, the user himself / herself can select the real camera 70 close to the virtual camera 72a.

[0083] Next, weights are given to the respective real cameras 70 in order of proximity to the virtual camera 72a, and a texture mixed based on the given weights is mapped and rendered on the virtual image J. Thus, a virtual image J of the polygon P viewed from the virtual camera 72a is generated. Note that a virtual image J of the 3D model 90M viewed from the virtual camera 72a can be generated by performing similar processing on a plurality of polygons of the 3D model 90M.

[0084] In the following, we will refer to Figure 5 A flowchart is used to describe the method of creating virtual ImageJ. Figure 5 is a flowchart illustrating an example of a process in which the camera path generation apparatus generates a virtual image.

[0085] First, a virtual viewpoint V for imaging the 3D model 90M is set (step S11 ).

[0086] Next, the positional relationship between the 3D model 90M and the virtual viewpoint V is calculated (step S12 ).

[0087] Furthermore, the real camera 70 to be used for rendering is selected (step S13).

[0088] Next, the real cameras 70 selected in step S13 are sorted in order of proximity to the virtual viewpoint V (step S14).

[0089] Then, starting from the real camera 70 closest to the virtual viewpoint V, weights are sequentially assigned to the respective real camera devices, and rendering processing is performed (step S15).

[0090] [1-3. Description of Hardware Configuration of Information Processing Device]

[0091] Figure 6 1 is a hardware block diagram showing an example of the hardware configuration of the information processing device according to the first embodiment. The information processing device 10a has a configuration in which a central processing unit (CPU) 40, a read-only memory (ROM) 41, a random access memory (RAM) 42, a storage unit 43, and an input / output controller 44 are connected via an internal bus 45.

[0092] The CPU 40 develops and executes the control program P1 stored in the storage unit 43 and various data files such as camera parameters stored in the ROM 41 on the RAM 42, thereby controlling the overall operation of the information processing device 10a. That is, the information processing device 10a has a configuration of a general computer operated by the control program P1. Note that the control program P1 can be provided via a wired transmission medium or a wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In addition, the information processing device 10a can perform a series of processes through hardware. Note that the control program P1 executed by the CPU 40 can be a program that performs processing in a time series in the order described in the present disclosure, or can be a program that performs processing in parallel or at a necessary timing (for example, when called).

[0093] The storage unit 43 includes, for example, a flash memory to store the control program P1 and the 3D model 90M executed by the CPU 40. Note that since the 3D model 90M generally has a large capacity, the 3D model may be downloaded from an external server (not shown) connected to the information processing device 10a via the Internet or the like and stored in the storage unit 43 as needed.

[0094] The storage unit 43 also stores a score map M (θ, φ) and a camera path Cp. The score map M (θ, φ) is a map obtained by quantifying a parameter related to the image quality of a virtual image J obtained by imaging the 3D model 90M with the virtual camera 72. The score map M (θ, φ) is generated for each 3D model 90M. The score map M (θ, φ) will be described in detail later. Note that the score map M (θ, φ) is an example of a map in the present disclosure. The stored camera path Cp stores, for example, a camera path Cp that is generated based on the score map M (θ, φ) and is a moving trajectory of a virtual viewpoint when the virtual camera 72 moves while maintaining an image quality of a predetermined value or more. The camera path Cp will be described in detail later.

[0095] The input / output controller 44 acquires information for operating the touch panel 50 stacked on the liquid crystal display 54, which displays information related to the information processing device 10a, via the touch panel interface 46. The input / output controller 44 displays image information on the liquid crystal display 54 via the display interface 47. In addition, the input / output controller 44 controls the operation of the real camera 70 via the camera interface 48.

[0096] In addition, although Figure 6Although not shown in the figure, the input / output controller 44 communicates with an external server (not shown) or the like through wireless communication or wired communication. Thus, the information processing device 10a receives, for example, a 3D model 90M created in another place.

[0097] [1-4. Description of Functional Configuration of Information Processing Device]

[0098] Figure 7 1 is a functional block diagram showing an example of the functional configuration of the information processing apparatus according to the first embodiment. The CPU 40 of the information processing apparatus 10a develops the control program P1 on the RAM 42 and executes the control program P1 to implement Figure 7 Each functional unit shown in .

[0099] That is, the CPU 40 of the information processing apparatus 10 a realizes the functions of the imaging control unit 81 , the 3D model generation unit 82 , and the score map generation unit 83 as the 3D model generation device 20 .

[0100] The imaging control unit 81 images the subject 90 by controlling the operation of the real imaging device 70 .

[0101] The 3D model generation unit 82 generates a 3D model 90M of the subject 90 based on the image of the subject 90 captured by the imaging control unit 81 .

[0102] The score map generation unit 83 predicts, for the 3D model 90M, the relationship between the position of the virtual viewpoint viewing the 3D model 90M and the image quality of the virtual image J obtained when the 3D model 90M is viewed from the position of the virtual viewpoint. Then, the score map generation unit 83 generates a score map M (θ, φ) that visualizes the estimated image quality of the virtual image J. Note that the score map generation unit 83 is an example of an image quality prediction unit in the present disclosure. A specific method of generating the score map M (θ, φ) will be described later.

[0103] Furthermore, the CPU 40 of the information processing device 10a implements the functions of a 3D model arrangement specifying unit 91, a camera path generating unit 92, a camera path displaying unit 93, a camera path selecting unit 94 and a rendering processing unit 95 as the camera path generating apparatus 30.

[0104] The 3D model arrangement specifying unit 91 specifies the arrangement positions of the plurality of 3D models 90M in the same space. Specifically, the operation control unit 98 specifies the arrangement positions in the three-dimensional space.

[0105] The camera path generation unit 92 generates a camera path Cp as a moving trajectory of the virtual viewpoint V based on the arrangement positions of the plurality of 3D models 90M specified by the 3D model arrangement specification unit 91 and the score map M (θ, φ) generated by the score map generation unit 83 for each 3D model 90M. Various conditions for generating the camera path Cp can be conceived. For example, a camera path Cp that can maintain an image quality equal to or higher than a predetermined value as much as possible can be generated. In addition, the camera path Cp can be generated to avoid a virtual viewpoint with a rough image quality, or the camera path set by the user can be changed to a camera path Cp with which a higher image quality can be obtained. Note that the camera path generation unit 92 is an example of a moving trajectory generation unit in the present disclosure. A specific method of generating the camera path Cp will be described later.

[0106] The camera path display unit 93 displays the camera path Cp generated by the camera path generation unit 92 on the liquid crystal display 54 .

[0107] The camera path selection unit 94 selects the camera path Cp designated by the operator from among the plurality of camera paths Cp generated by the camera path generation unit 92 .

[0108] The rendering processing unit 95 performs rendering of the plurality of 3D models 90M viewed from the virtual viewpoint V while moving the virtual viewpoint V along the camera path Cp selected by the camera path selection unit 94 , and generates a virtual image J.

[0109] Furthermore, the CPU 40 of the information processing device 10a implements an operation control unit 98 as a function of performing operation control when operating the 3D model generation device 20 and the camera path generation device 30. Furthermore, the CPU 40 implements a display control unit 99 as a function of performing display control to display information related to the 3D model generation device 20 and the camera path generation device 30.

[0110] The operation control unit 98 controls the touch panel 50 to acquire an operation instruction related to the 3D model generation device 20 and the camera path generation device 30 of the operator, and delivers the operation instruction to the relevant functional units.

[0111] The display control unit 99 controls the liquid crystal display 54 to perform display according to display instructions of various types of information from the 3D model generating apparatus 20 and the camera path generating apparatus 30 .

[0112] [1-5. Description of method for calculating image quality score]

[0113] Next, the score map M (θ, φ) used by the information processing device 10a of the present embodiment will be described. The score map M (θ, φ) is a map that quantifies the image quality of the virtual image J when the 3D model 90M is viewed for each direction of viewing the subject 90 and is generated simultaneously when the 3D model generation device 20 creates the 3D model 90M.

[0114] Figure 8 90M of the subject 90, a plurality of real cameras 70 are arranged around the subject 90. The respective real cameras 70 generally have different optical parameters of the camera (lens focal length, number of pixels, sensor size, etc.) and different formats of videos to be captured (HD (high definition), full HD, 4K, 8K, etc.).

[0115] exist Figure 8 In the example of , five real camera devices 70 (70a, 70b, 70c, 70d, 70e) are arranged around the subject 90. Note that the direction of the real camera device 70 on the XY plane is represented by an angle θ. In addition, the direction of the real camera device 70 on the XZ plane is represented by an angle φ. That is, the arrangement position of the real camera device 70 is specified by the coordinates (θ, φ).

[0116] The real camera 70a is set on the XY plane in the front direction of the subject 90 so as to face the front of the subject 90. The real camera 70b is set to face the front of the subject 90 from slightly above the subject 90. Figure 8 , the viewing angle of the camera is represented by the vertex angle of an isosceles triangle, indicating that the real camera 70b has a wide-angle lens and has a wider viewing angle than other real cameras. The real camera 70c is set in a direction looking down from above the subject 90 toward the front of the subject 90.

[0117] In addition, the real camera 70d is set on the back side of the subject 90 to face the back of the subject 90. The real camera 70e is set in a direction where the subject 90 is outside the field of view. Note that it is assumed that the real camera 70a to the real camera 70d are set at positions substantially equidistant from the subject 90, and the real camera 70e is set farther.

[0118] The camera path generation apparatus 10 generates a score map M (θ, φ) that visualizes the image quality of the image of the subject 90 captured by each real camera 70. Hereinafter, a method of quantifying the image quality of an image and a process of generating the score map M (θ, φ) will be described in sequence.

[0119] First, the information processing device 10a predicts the image quality score Iq of the image captured by each of the real camera devices 70a to 70e. It is assumed that the image quality score Iq is calculated by, for example, Expression (2).

[0120] Iq = e*(p / s)*visibility*weight (2)

[0121] In expression (2), e represents the resolution (number of pixels) of the real camera 70. For example, in the case of a real camera 70 with full HD image quality, e = 1920 (horizontal) × 1080 (vertical) pixels. In addition, in the case of a real camera 70 with 4K image quality, e = 3840 (horizontal) × 2160 (vertical) pixels. In addition, p in expression (2) is the image size of the subject 90 calculated by the above expression (1). In addition, s is the size of the image sensor included in the real camera 70. For example, in the case of a full-size image sensor, s = 36.0 (horizontal) × 24.0 (vertical) mm, and in the case of an APS-C size image sensor, s = 23.6 (horizontal) × 15.8 (vertical) mm. That is, (p / s) in expression (2) indicates the ratio of the subject 90 in the image captured by the real camera 70. Then, e*(p / s) in expression (2) represents the number of pixels of the image of the subject 90 to be captured. That is, the larger the value of e*(p / s), the higher the image quality score Iq. In this way, the information processing device 10a predicts the image quality of the virtual image J based on the optical parameters of the real camera 70 (imaging unit) used when generating the 3D model 90M (3D object) and the distance D between the real camera 70 and the 3D model 90M image.

[0122] The visibility in expression (2) is information indicating whether the subject 90 (3D model 90M) is shown in the image (i.e., whether the real camera 70 captures the subject 90). For example, in the case where the subject 90 is captured, visibility = 1, and in the case where the subject 90 is not captured, visibility = 0. The weight is an arbitrarily set weight. For example, in the case where the real camera 70 is located in front of the subject 90, that is, in the case where the subject 90 is captured from the front, the weight is set to 1.0, and in other cases, the weight is set to 0.5. Note that the formula for calculating the image quality score Iq is not limited to expression (2), and other parameters can also be considered. For example, when the subject 90 is blurred, the image quality score Iq can be reduced according to the degree of blur.

[0123] When the real camera 70 captures the subject 90 (3D model 90M) mainly from the front, that is, when the subject 90 is imaged to be completely shown on the screen from the front, the image quality score Iq calculated by expression (2) has a large value.

[0124] The 3D model generation device 20 sets the image quality score Iq calculated by expression (2) in the space (θ, φ) corresponding to the arrangement position of the real camera 70, thereby generating a score map M (θ, φ) for each direction in which the 3D model 90M (3D object) is viewed. The details will be described later (see Fig.11 ).

[0125] Next, we will refer to Fig. 9 A specific calculation example of the image quality score Iq is described. Fig. 9 is a diagram showing an example of calculating an image quality score. Specifically, Fig. 9 An example of calculating the image quality score Iq by the above expressions (1) and (2) is shown with the height of the subject 90 set to h=1.5 m. Note that the number of pixels e and the size s of the image sensor are values ​​in the vertical direction corresponding to the height direction of the subject 90 .

[0126] Note that in the case where the face of the subject 90 is not shown in the image captured by the real camera 70, the image quality score Iq is set to 0. In addition, the focus is evaluated on the captured image with 100 points when in focus. Then, for example, in the case where the evaluation score is 50 points, the image quality score Iq is multiplied by 0.5.

[0127] like Fig. 9 As shown, although the real cameras 70a, 70b, and 70c have different resolutions and different imaging conditions, by using expressions (1) and (2), the image qualities of captured images can be compared on an equal basis.

[0128] [1-6. Description of method for generating score map]

[0129] Next, we will refer to Fig.10 and Fig.11 The process of generating a score map M(θ, φ) that visualizes the image quality score Iq is described. Fig.10 is a diagram showing an example of the layout of a real imaging device. Fig.11 is a diagram showing an example of generating a score map.

[0130] As an example, Fig.10As shown, it is assumed that three real cameras 70a, 70b and 70c are set around the 3D model 90M (subject 90). Specifically, in the top view, the real camera 70a is set in the direction of θ=θ1 relative to the reference direction of the angle θ. The real camera 70b is set in the direction of θ=θ2. The real camera 70c is set in the direction of θ=θ3. Note that the 3D model 90M faces the direction of the real cameras 70a and 70b. That is, the real cameras 70a and 70b image the 3D model 90M from the front direction. On the other hand, the real camera 70c is set to face the back side of the 3D model 90M. Note that the real camera 70a includes a lens with a focal length f longer than that of the real cameras 70b and 70c.

[0131] In addition, if Fig.10 As shown, it is assumed that the real camera devices 70a, 70b and 70c are set at a position where the 3D model 90M (subject 90) is overlooked from above. Specifically, in the side view, the real camera device 70a is set in the direction of φ=φ1 with respect to the reference direction of the angle φ. The real camera device 70b is set in the direction of φ=φ2. Then, the real camera device 70c is set in the direction of φ=φ3.

[0132] In the case where the image quality of the image captured by the real camera 70 at the position of the angle θ and the distance D from the 3D model 90M is good, when the virtual camera 72a is brought close to the 3D model 90M while maintaining the angle θ, the 3D model 90M is in a state of being imaged in a further magnified manner, and therefore, even when the real camera is close to a position where the 3D model 90M can be imaged on the full screen, the image quality of the image captured by the real camera 70a is maintained. Then, when the virtual camera 72a is brought close to the 3D model 90M beyond the closest point 74a (see Fig.10 ), the 3D model 90M is framed, and thus the 3D model 90M cannot be imaged, i.e., the image quality of the image captured by the real camera 70a is not maintained. On the other hand, in the case where the virtual camera 72a is moved away from the position of the real camera 70a while maintaining the angle θ, assuming that the focal length f of the virtual camera 72a and the focal length f of the real camera 70a are the same, the 3D model 90M is simply captured small, and thus the image quality of the image captured by the real camera 70a is maintained.

[0133] Similarly, the image quality of the images of the 3D model 90M captured by the real cameras 70b and 70c is maintained away from Fig.10Note that the specific positions of the closest points 74a, 74b, and 74c can be calculated based on the imaging conditions of the real cameras 70a, 70b, and 70c (specifically, the imaging magnification of the 3D model 90M).

[0134] Score map generation unit 83 (see Figure 7 )according to Fig.11 The flowchart shown is used to create a score map M(θ, φ) that visualizes the image quality scores Iq of images captured by the real cameras 70a, 70b and 70c set in such a positional relationship.

[0135] First, the score map generation unit 83 calculates the image quality score Iq for each of the real cameras 70a, 70b, and 70c (step S21). Note that the calculation process of the image quality score Iq is as described above.

[0136] Next, the score map generation unit 83 plots the calculated image quality score Iq on the (θ, φ) plane (step S22 ). Fig.11 An example of actually plotting the image quality score Iq is shown in the upper half of FIG. On the (θ, φ) plane, the values ​​of the image quality score Iq are plotted at respective positions of coordinates (θ1, φ1), (θ2, φ2), and (θ3, φ3) corresponding to the arrangement positions of the real camera devices 70a, 70b, and 70c.

[0137] Next, the score map generation unit 83 generates a score map for each real camera 70 by diffusing the values ​​of the image quality scores Iq plotted on the (θ, φ) plane around the (θ, φ) plane (step S23 ).

[0138] In the case where the virtual camera 72 is set near the real camera 70, when the angle difference between the virtual camera 72 and the real camera 70 is small and the virtual camera 72 captures the 3D model 90M in the image, the value of the image quality score Iq can be considered to be substantially equal. Therefore, even when there is a slight angle difference between the real camera 70 and the virtual camera 72 in the score map of each real camera 70, the drawn image quality score Iq is diffused to the surroundings to allow for the difference.

[0139] Specifically, the image quality score Iq can be diffused to the surroundings by performing a smoothing operation on the (θ, φ) plane. For the smoothing operation, for example, a known Gaussian filter can be used. By diffusing the image quality score Iq to the surroundings, a mountain-shaped score distribution centered on the arrangement position of each real camera 70 is obtained, such as Fig.11 Notice that Fig.11 The discrete values ​​are shown to be distributed for easy viewing, but in fact a unimodal distribution storing continuous values ​​is formed. That is, the image quality score Iq has a distribution in which the value is highest at the installation position of the real camera 70 and decreases monotonically as the distance from the installation position of the real camera 70 increases.

[0140] Note that the size of the filter to be operated when performing the smoothing operation can be changed according to the value of the image quality score Iq. That is, in the case where the image quality score Iq is high, it is considered that even if the angle difference between the real camera 70 and the virtual camera 72 is slightly large, high image quality can be maintained. Therefore, it is desirable to apply a smoothing filter with a large filter size. In this case, the smoothed image quality score Iq forms a gentle unimodal distribution.

[0141] On the contrary, when the image quality score Iq is low, the image quality is expected to degrade rapidly when the angle difference between the real camera 70 and the virtual camera 72 increases. Therefore, it is desirable to apply a smoothing filter with a small filter size. In this case, the smoothed image quality score Iq forms a steep unimodal distribution.

[0142] Then, the score map generation unit 83 integrates the score maps of the respective real camera devices 70a, 70b, and 70c into one score map M(θ, φ) (step S24). When integrating the score maps of the real camera devices 70a, 70b, and 70c, when a plurality of real camera devices 70 have image quality scores Iq at the same coordinates (θ, φ), the maximum value thereof is stored in the score map M(θ, φ).

[0143] Note that the score map M (θ, φ) is not limited to the score map drawn on the above-mentioned (θ, φ) plane. That is, the image quality score Iq according to the distance between the real camera 70 and the 3D model 90M may be drawn. In addition, the image quality score Iq may be drawn according to the focal length f of the real camera 70. In addition, the image quality score Iq corresponding to the resolution of the real camera 70 may be drawn. In addition, the image quality score Iq corresponding to the size of the 3D model 90M captured by the real camera 70 may be drawn.

[0144] Furthermore, the score map generation unit 83 may also generate a map storing the distances between the nearest points 74 a , 74 b , and 74 c and the 3D model 90M.

[0145] [1-7. Description of a method for generating a camera path]

[0146] Next, we will refer to Fig.12 A method of generating the camera path Cp is described. Fig.12is a diagram showing a method of generating a camera path.

[0147] When multiple 3D models 90M are set in the same space, the camera path generation unit 92 generates a camera path Cp as a movement trajectory of the virtual viewpoint V and can maintain a predetermined value of image quality or greater as much as possible when imaging multiple 3D models 90M, for example.

[0148] Fig.12 1 is a plan view showing a state where two 3D models 90Ma (subject 90a) and 90Mb (subject 90b) are arranged in the same space.

[0149] Note that Fig.12 As shown, it is assumed that score maps Ma(θ, φ) and Mb(θ, φ) of 3D models 90Ma and 90Mb have been generated.

[0150] First, the camera path generation unit 92 refers to the score maps Ma(θ, φ) and Mb(θ, φ) as image quality prediction results, and reconstructs the virtual camera 72 in three-dimensional space so as to be estimated to be able to capture the areas of the 3D models 90Ma and 90Mb with high image quality.

[0151] For example, focus on cluster C1 that is predicted to be able to obtain high image quality in the score map Ma (θ, φ) of the 3D model 90Ma. In cluster C1, areas with particularly high image quality are distributed within the angle range θa centered at the angle θ=θ1. In addition, areas estimated to have high image quality are distributed in the angle range θb centered at the angle θ=θ1.

[0152] That is, the area with particularly high image quality is reconstructed as a conical area extending from the position of the 3D model 90Ma in the direction of the angle θ1 and extending over the angle range θa. Similarly, the area estimated to have high image quality is reconstructed as a conical area extending from the position of the 3D model 90Ma in the direction of the angle θ1 and extending over the angle range θb.

[0153] Similarly, for other clusters in the score map Ma(θ, φ) and the score map Mb(θ, φ), the region estimated to have high image quality can be reconstructed into the three-dimensional space. Note that in practice, the conical region is reconstructed in the three-dimensional space considering the φ direction, but for simplicity, it is described only in the θ direction.

[0154] refer to Fig.12As shown in the reconstruction result, the camera path generation unit 92 generates the camera path Cp under the condition that the camera path Cp passes through an area with as high an image quality as possible for a long time. Note that the camera path generation unit 92 considers as a constraint condition that the camera path is not brought closer to a position closer than the above-mentioned closest point.

[0155] Note that since the camera path Cp can be infinitely generated, it is desirable to first indicate the start point of the camera path Cp or the start point and end point of the camera path Cp.

[0156] Fig.12 An example of generating the camera path Cp when the start point 110 and the end point 112 of the camera path Cp are specified is shown. The camera path Cp is ideally estimated to be an area capable of imaging with high image quality through both the plurality of 3D models 90Ma (subject 90a) and the plurality of 3D models 90Mb (subject 90b). Fig.12 As shown, the start point 110 and the end point 112 are set as areas where both the plurality of 3D models 90Ma and the plurality of 3D models 90Mb are estimated to be imaged with high image quality. Note that the start point 110 and the end point 112 can be set by the operator of the information processing device 10a using the touch panel 50 while viewing the reconstruction result of the high image quality area displayed on the liquid crystal display 54.

[0157] Then, the camera path generation unit 92 searches for a camera path Cp from the set start point 110 to the end point 112 while tracking a position where the image quality score Iq is as high as possible. Note that the camera path generation unit 92 also sets the orientation of the virtual camera 72 at each point through which the camera path Cp passes. The orientation of the virtual camera 72 is set so that all of the 3D models 90Ma and 90Mb can be imaged together. When all of the 3D models 90Ma and 90Mb cannot be imaged, it is possible to Fig.12 As in the virtual camera 72c in FIG. 7 , only a portion of the subject 90a is captured.

[0158] In this manner, the camera path generation unit 92 generates a camera path Cp from the virtual camera 72a installed at the start point 110 to the virtual camera 72d installed at the end point 112 in the order of the virtual camera 72b and the virtual camera 72c.

[0159] For example, when searching for the camera path Cp, a known dynamic programming method may be applied. That is, a camera path Cp in which the sum of weights of the camera path Cp from the start point 110 to the end point 112 is minimized may be generated by weighting each of a plurality of candidates for the camera path Cp searched in the micro area.

[0160] The camera path generation unit 92 may also set the moving speed of the virtual camera 72. Specifically, the virtual camera 72 may move slowly when placed in an area with high image quality, and may move quickly when placed in an area with low image quality.

[0161] Furthermore, in the case where the camera path Cp has to pass through an area with a low image quality score Iq, the camera path generation unit 92 may set the virtual camera 72 to a direction not facing the 3D model 90M.

[0162] Furthermore, when generating the camera path Cp, the score map M (θ, φ) may be adjusted so that the camera path Cp is easily generated. For example, a weight may be added to the score map M (θ, φ) to correct the image quality score Iq of the 3D model 90M of the object of interest 90 to a high direction. This makes it possible to generate a camera path Cp that preferentially passes through an area where the 3D model 90M can be imaged with higher image quality.

[0163] Furthermore, the image quality score Iq when captured by the real camera 70 with high resolution can be corrected to a high direction in consideration of the resolution of the real camera 70. Therefore, when an image captured by a lens with a long focal length f has the same image quality score Iq as an image captured with high resolution, a high image quality area captured by the real camera 70 with a higher resolution can be preferentially searched when searching for the camera path Cp.

[0164] Furthermore, depending on the arrangement state of the plurality of 3D models 90M, the plurality of 3D models 90M may overlap when viewed from the virtual camera 72. Since it is not preferable that the camera path Cp passes through such a position, the camera path Cp may be searched in a state where the installation position of the virtual camera 72 where shielding occurs is removed. Note that the installation position of the virtual camera 72 where shielding occurs may be obtained by calculation based on the installation position of each 3D model 90M.

[0165] In addition, it is explained that the virtual camera device 72 can maintain high image quality when the virtual camera device 72 moves away from the 3D model 90M in the same angle θ direction, but an upper limit value can be set for the distance between the 3D model 90M and the virtual camera device 72 to improve the efficiency of searching the camera device path Cp.

[0166] Camera path display unit 93 (see Figure 7) causes the display control unit 99 to display the camera path Cp generated by the camera path generation unit 92 on the liquid crystal display 54. Note that the display form of the camera path Cp is not limited. For example, the generated camera path Cp may be drawn in a three-dimensional space in which a plurality of 3D models 90M are provided, or the camera path Cp may be drawn in the form of three views.

[0167] Furthermore, the camera path generation unit 92 can generate a plurality of camera paths Cp. The plurality of generated camera paths Cp are displayed on the liquid crystal display 54 by the operation of the camera path display unit 93 and the display control unit 99. Then, the operator of the information processing device 10a selects one camera path Cp by the operation of the camera path selection unit 94 (see Figure 7 Specifically, when one camera path Cp is selected from a plurality of camera paths Cp displayed on the liquid crystal display 54 by the action of the camera path selection unit 94 on the touch panel 50 by changing the display form of color etc., one camera path Cp is selected by the action of the operation control unit 98 that detects the selection operation.

[0168] [1-8. Description of Camera Device Path Generation Assistance Method]

[0169] Next, we will refer to Fig.13 A method of auxiliary camera path Cp generation is described. Fig.13 1 is a first diagram showing an example of a user interface when generating a camera path.

[0170] As described above, since the camera path Cp can be infinitely generated, it is desirable to impose some constraints to efficiently generate the camera path Cp. Fig.13 and Fig.14 A user interface for efficiently generating camera paths Cp is described. Fig.13 is a first diagram illustrating an example of a user interface for predicting a camera path. Fig.14 is a second diagram illustrating an example of a user interface for predicting a camera path.

[0171] Fig.13 An example of displaying the arrangement state of the plurality of subjects 90 a and 90 b , the arrangement state of the real camera 70 , and the imaging range 120 of each real camera 70 on the liquid crystal display 54 is shown.

[0172] like Fig.13 As shown, the imaging range 120 includes the installation position 121 of the real camera device, the range 122 where high image quality can be obtained (for example, corresponding to Fig.12The angle range θa) shown and the information of the focal length 123 of the real camera device 70.

[0173] The operator of the information processing device 10a can Fig.13 The position of the actual camera 70 is sequentially tracked on the liquid crystal display 54 of the information shown to specify the camera path Cp. Fig.13 An example of setting the camera path Cp by sequentially tracking the installation positions 121b, 121c, 121d, and 121e from the installation position 121a is shown. By using such a method, since the arrangement of the real camera 70 is clearly visualized, it is easy to intuitively generate the camera path Cp.

[0174] Note that since the position of the real camera device 70 can be moved forward and backward, Fig.13 In the embodiment, the real camera 70 can be tracked in a direction approaching or moving away from the subjects 90a and 90b.

[0175] Fig.14 An example of displaying the arrangement state of a plurality of subjects 90a and 90b, the arrangement state of the real camera 70, and the imaging range 120 of each real camera 70 on the liquid crystal display 54 is shown. Fig.13 The difference is that information indicating which object is captured by each real camera is added to the imaging range of the real camera 70 .

[0176] Specifically, in Fig.14 In the figure, the 3D model of the object 90a is generated by five real camera devices 70, which image the respective imaging ranges 120a1, 120a2, 120a3, 120a4 and 120a5. In addition, the 3D model of the object 90b is generated by six real camera devices 70 that image the respective imaging ranges 120b1, 120b2, 120b3, 120b4, 120b5 and 120b6.

[0177] The operator of the information processing device 10a can Fig.14 The position of the real camera 70 is sequentially tracked on the LCD 54 of the information shown to specify the camera path Cp. At this time, one of the subject 90a and the subject 90b, that is, the camera path Cp focused on the specific subject can be specified.

[0178] Note that you can pass Fig.13 or Fig.14The camera path Cp specified by the method shown is passed to the camera path generation unit 92, and the camera path generation unit 92 can perform recalculation so that the specified camera path Cp is the camera path Cp with which the virtual image J with higher image quality can be obtained.

[0179] Then, the camera path Cp generated by the camera path generation unit 92 or by Fig.13 and Fig.14 The camera path Cp specified by the method shown is transferred to the rendering processing unit 95. Then, the rendering processing unit 95 performs rendering of the plurality of subjects 90 while moving the virtual viewpoint V along the camera path Cp, and generates a virtual image J. Then, the virtual image J subjected to the rendering process is displayed on the liquid crystal display 54 by the action of the display control unit 99.

[0180] [1-9. Description of Flow of Processing Executed by Camera Path Generation Device]

[0181] Next, we will refer to Fig.15 The flow of processing performed by the camera path generation device 30 is described. Fig.15 is a flowchart showing an example of the flow of processing performed by the camera path generation apparatus.

[0182] The 3D model arrangement specifying unit 91 specifies the arrangement positions of the plurality of subjects 90 (step S31 ).

[0183] The camera path generation unit 92 acquires the score map M (θ, φ) of each subject 90 from the 3D model generation device 20 (step S32 ).

[0184] The camera path generation unit 92 generates a camera path Cp that is a moving trajectory of the virtual viewpoint V and can maintain an image quality equal to or higher than a predetermined value as much as possible (step S33). A specific method of generating the camera path Cp is as described above. Note that in addition to generating a camera path Cp that can maintain a (high) image quality equal to or higher than a predetermined value, the camera path generation unit 92 can also generate a camera path Cp to avoid a virtual viewpoint with a rough image quality, or can change the camera path set by the user to a camera path Cp with which a higher image quality can be obtained.

[0185] The camera path display unit 93 causes the display control unit 99 to display the camera path Cp on the liquid crystal display 54 (step S34).

[0186] The rendering processing unit 95 performs rendering processing on the subject 90 while moving the virtual viewpoint of the virtual camera 72 along the camera path Cp (step S35 ).

[0187] The drawing processing unit 95 also causes the display control unit 99 to display the result of the rendering process on the liquid crystal display 54 (step S36 ).

[0188] As described above, according to the information processing device 10a of the first embodiment, the score map generation unit 83 (image quality prediction unit) included in the information processing device 10a predicts, for the 3D model 90M (3D object), the relationship between the position of the virtual viewpoint V from which the 3D model 90M is viewed and the image quality of the virtual image J obtained when the 3D model 90M is viewed from the position of the virtual viewpoint V.

[0189] Therefore, the image quality of the virtual image J obtained when the 3D model 90M is imaged from an arbitrary direction can be estimated and visualized.

[0190] In addition, according to the information processing device 10a of the first embodiment, the camera path generation unit 92 (movement trajectory generation unit) generates a camera path Cp as a movement trajectory of the virtual viewpoint V based on the image quality of the virtual image J predicted by the score map generation unit 83 (image quality prediction unit) for each 3D model 90M and the arrangement positions of multiple 3D models 90M (3D objects).

[0191] Therefore, when a plurality of 3D models 90M generated under different conditions are arranged in the same space, it is possible to simultaneously image the plurality of 3D models 90M and generate a camera path Cp in which a virtual image J having the highest possible image quality can be generated.

[0192] In addition, according to the information processing device 10a of the first embodiment, the score mapping generation unit 83 (image quality prediction unit) predicts the image quality of the virtual image J based on the optical parameters (number of pixels e, size s of the image sensor, and focal length f) of the real camera device 70 (imaging unit) used when generating the 3D model 90M (3D object) and the distance D between the real camera device 70 and the 3D model 90M.

[0193] Therefore, image quality can be predicted through simple operations.

[0194] Furthermore, according to the information processing device 10 a of the first embodiment, the optical parameter is a parameter related to the ratio of the 3D model 90M (3D object) to the image captured by the real camera 70 (imaging unit).

[0195] Therefore, even when the signal format of the real camera 70 used when generating the 3D model 90M is different, the image quality can be compared on the same scale.

[0196] In addition, according to the information processing device 10a of the first embodiment, the direction of viewing the 3D model 90M (3D object) includes at least one of information indicating whether the real camera 70 (imaging unit) captures the 3D model 90M and information indicating whether the 3D model 90M is captured from the front.

[0197] Therefore, the image quality may be predicted taking into account the orientation of the 3D model 90M.

[0198] In addition, according to the information processing device 10a of the first embodiment, the score map generation unit 83 (image quality prediction unit) generates a score map M(θ, φ) (map), which is obtained by quantifying parameters related to the image quality of the image obtained by imaging the 3D model 90M for each direction in which the 3D model 90M (3D object) is viewed.

[0199] This makes it possible to visualize the image quality taking into account the imaging direction of the 3D model 90M.

[0200] In addition, according to the information processing device 10a of the first embodiment, the score map generation unit 83 (image quality prediction unit) generates a score map M (θ, φ) (map) by drawing quantized parameters related to the image quality of the virtual image J at a position corresponding to the imaging position of the virtual image J and diffusing the drawn parameters related to the image quality of the virtual image J from the drawn position to the surroundings.

[0201] Therefore, when predicting the image quality of the virtual image J, a deviation between the position of the real camera 70 and the position of the virtual camera 72 can be allowed.

[0202] In addition, according to the information processing device 10a of the first embodiment, the score map generation unit 83 (image quality prediction unit) performs a smoothing operation on the image quality of the virtual image J plotted on the score map M(θ, φ) (map) to diffuse the image quality to the surroundings.

[0203] Therefore, the magnitude of the deviation between the position of the real camera 70 and the position of the virtual camera 72 can be determined by a simple operation.

[0204] Furthermore, in the information processing device 10 a of the first embodiment, the 3D model 90M (3D object) has texture information related to the viewpoint position.

[0205] Therefore, since the texture varies according to the viewpoint position, rendering can be performed on the virtual image J with higher image quality.

[0206] Furthermore, in the information processing apparatus 10 a of the first embodiment, the 3D model generation device 20 independently generates 3D models 90M (3D objects) of a plurality of subjects 90 .

[0207] Therefore, it is possible to set the camera path Cp of the virtual camera 72 capable of imaging a plurality of subjects 90 with high image quality in a state where a plurality of subjects 90 exist.

[0208] (2. Second Embodiment)

[0209] In addition to using the above-mentioned score map M(θ, φ) in the case of generating the camera path Cp, there is another effective method of using the score map M(θ, φ). Hereinafter, other uses of the score map M(θ, φ) will be described.

[0210] The information processing device 10b according to the second embodiment of the present disclosure is an example of the information processing device 10b that evaluates the generation condition of the 3D model 90M based on the score map M (θ, φ) of the 3D model 90M of the subject 90 generated by the 3D model generation apparatus 20 described in the first embodiment.

[0211] Fig.16 1 is a block diagram showing an example of a basic configuration of an information processing apparatus according to the present embodiment. The information processing apparatus 10b includes a 3D model generating device 20 and a mobile terminal 32b.

[0212] The 3D model generation device 20 has the same functions as those described in the first embodiment. Note that the 3D model generation device 20 described here includes a plurality of 3D model generation devices 20a, 20b, ... This assumes a situation where different 3D model generation devices 20a, 20b, ... are placed at a plurality of different places. Each of the 3D model generation devices 20a, 20b, ... generates a 3D model 90M and a score map M (θ, φ) of the subject 90. Then, the plurality of 3D models 90M and the score maps M (θ, φ) generated at different places are connected via a communication line and transmitted to the mobile terminal 32b.

[0213] The mobile terminal 32b acquires the score map M (θ, φ) generated by the 3D model generation device 20a, 20b, ... Then, the mobile terminal 32b analyzes the score map M (θ, φ) to evaluate the installation position of the real camera 70 when generating the 3D model 90M. Then, the mobile terminal 32b outputs the evaluation result. Note that the mobile terminal 32b is, for example, a smartphone, a mobile PC, etc.

[0214] Note that the hardware configuration of the 3D model generating device 20 is as described in the first embodiment. Furthermore, the mobile terminal 32b has a general-purpose computer configuration.

[0215] Fig.17 32 b is a functional block diagram showing an example of a functional configuration of a mobile terminal according to the second embodiment. The mobile terminal 32 b includes a score map acquisition unit 33 , a score map analysis unit 34 , and a real camera position proposal unit 35 .

[0216] The score map acquisition unit 33 acquires the score map M(θ, φ) from each of the 3D model generating devices 20 a , 20 b , . . . .

[0217] The score map analysis unit 34 evaluates the arrangement conditions of the real camera 70 (imaging unit) used when generating the 3D model 90M (3D object) based on the score map M (θ, φ) acquired from each 3D model generating device 20a, 20b, ... Note that the score map analysis unit 34 is an example of an arrangement evaluation unit in the present disclosure.

[0218] The real camera position proposing unit 35 sends the analysis result performed by the score map analyzing unit 34 to the corresponding 3D model generating devices 20a, 20b, . . . . .

[0219] Fig.18 is a diagram for explaining the operation of the mobile terminal according to the second embodiment. The mobile terminal 32b obtains the 3D model generating device 20 (to be precise, any one of the 3D model generating devices 20a, 20b, ...) Fig.18 The score map M(θ, φ) shown in . Fig.18 As shown, the score map M (θ, φ) represents the image quality of the 3D model 90M generated in a state where two real cameras 70 a and 70 b are arranged for the subject 90 .

[0220] The score map analysis unit 34 refers to the score map M (θ, φ) to analyze whether there is an area with missing information (i.e., an area not covered by the real camera 70), whether there is an area with low image quality in the angle θ direction and the angle φ direction, etc. Fig.18 In the example of FIG. 1 , the score map analysis unit 34 determines that there is a region w1 with missing information in the angle θ direction. Fig.18 As shown, by installing a new real camera 70c in the information-missing area w1 and updating the 3D model 90M, a 3D model 90M covering a wider range can be generated.

[0221] Then, the real camera position proposing unit 35 transmits the analysis result to any of the 3D model generating devices 20a, 20b, ... that have transmitted the score map M(θ, φ). The operator of the 3D model generating device that receives the analysis result updates the 3D model 90M according to the received result.

[0222] As described above, according to the information processing device 10b of the second embodiment, the score mapping analysis unit 34 (arrangement evaluation unit) evaluates the arrangement conditions of the real camera 70 (imaging unit) used when generating the 3D model 90M (3D object) based on the score mapping M (θ, φ) (mapping map).

[0223] Therefore, when the 3D model 90M is generated by visualizing the installation position using the score map M (θ, φ), the installation position of the real camera 70 can be easily and reliably estimated.

[0224] (3. Third Embodiment)

[0225] The information processing device 10c (not shown) according to the third embodiment of the present disclosure includes a mobile terminal 32c (not shown) instead of the mobile terminal 32b in the information processing device 10b described in the second embodiment. Note that the hardware configuration of the information processing device 10c is the same as that of the information processing device 10b.

[0226] The mobile terminal 32c included in the information processing device 10c receives the 3D model 90M of the subject 90 generated by the 3D model generation device 20 described in the first embodiment. Then, the mobile terminal 32c performs rendering processing on the 3D model 90M from the direction specified by the operator, and presents the reconstructed 3D model 90M. At this time, the mobile terminal 32c refers to the score map M (θ, φ) and controls the display state to avoid browsing from a direction with low image quality or a direction where information is missing.

[0227] Fig.19 32c includes a score map acquisition unit 33, a virtual camera installation prohibited area calculation unit 60, an observation direction detection unit 61, a virtual image generation unit 62, a virtual image display unit 63, and a prohibited area video processing unit 64.

[0228] The score map acquisition unit 33 acquires the score map M(θ, φ) from each of the 3D model generating devices 20 a , 20 b , . . . .

[0229] The virtual camera installation prohibition area calculation unit 60 calculates the position (the direction indicated by the coordinates (θ, φ)) where the virtual viewpoint V is not installed based on the score map M (θ, φ). Note that the position where the virtual viewpoint V is not installed is, for example, an area where information is missing (i.e., an area not covered by the real camera 70) or an area with low image quality. Note that the virtual camera installation prohibition area calculation unit 60 is an example of a browsing prohibition area calculation unit in the present disclosure.

[0230] The viewing direction detection unit 61 detects the viewing direction of the rendering result, that is, the virtual image J, instructed by the operator of the mobile terminal 32c. Specifically, although it differs depending on the operating system included in the mobile terminal 32c, for example, in the case where the viewing direction is changed by changing the orientation of the mobile terminal 32c, the viewing direction detection unit 61 detects the viewing direction by monitoring, for example, the output of a gyro sensor included in the mobile terminal 32c. In addition, in the case where the viewing direction is changed by sliding the screen of the mobile terminal 32c, the viewing direction detection unit 61 detects the viewing direction by monitoring the sliding operation performed.

[0231] The virtual image generation unit 62 generates a virtual image J viewed from the viewing direction indicated by the operator of the mobile terminal 32c. Note that the mobile terminal 32c performs rendering processing on the generated virtual image J.

[0232] The virtual image display unit 63 displays the virtual image J generated by the virtual image generation unit 62 on the display unit of the mobile terminal 32 c .

[0233] When the viewing direction detected by the viewing direction detection unit 61 matches the virtual viewpoint V calculated by the virtual camera installation prohibited area calculation unit 60 (i.e., the position (direction indicated by the coordinates (θ, φ)) where the virtual camera 72 is not installed, the virtual image J of the 3D model 90M (3D object) is not generated. Specifically, the virtual image J is not generated by fading out the display of the mobile terminal 32c, blurring the display of the mobile terminal 32c, applying an effect to the display of the mobile terminal 32c, etc. By presenting such a video, the operator is guided to set the viewing direction of the virtual viewpoint V that generates a good virtual image J. Note that the prohibited area video processing unit 64 is an example of a video processing unit according to the present disclosure.

[0234] Fig. 20 3 is a diagram for explaining the operation of the mobile terminal according to the third embodiment. Assume that when a 3D model 90M of a subject 90 is created by real camera devices 70a and 70b, Fig. 20At this time, the virtual camera installation prohibited area calculation unit 60 analyzes the score map M (θ, φ) to calculate the virtual camera installation prohibited area as a position (direction indicated by the coordinates (θ, φ)) where the virtual camera 72 is not installed. That is, the installation position of the virtual camera 72 is limited to the range of the area w2 indicated in the score map M (θ, φ) (that is, the range from the angle θmin to the angle θmax).

[0235] When reconstructing these pieces of information in real space, Fig. 20 The area w2 shown in FIG. 7 is limited to the installation position of the virtual camera 72 , that is, the virtual viewpoint V.

[0236] As described above, according to the information processing device 10c of the third embodiment, the virtual camera installation prohibition area calculation unit 60 (browsing prohibition area calculation unit) calculates the position where the virtual viewpoint V is not installed based on the score map M (θ, φ) (map) when generating the virtual image J based on the 3D model 90M (3D object). Then, when the virtual viewpoint V of the 3D model 90M matches the position of the virtual viewpoint V calculated by the virtual camera installation prohibition area calculation unit 60, the prohibition area video processing unit 64 (video processing unit) does not generate the virtual image J of the 3D model 90M based on the position of the virtual viewpoint V.

[0237] Therefore, it is possible to avoid setting the virtual viewpoint V at a position where the virtual image J cannot be generated or the image quality of the generated virtual image J is low.

[0238] (4. Fourth Embodiment)

[0239] The information processing device 10d (not shown) according to the fourth embodiment of the present disclosure includes a mobile terminal 32d (not shown) instead of the mobile terminal 32b in the information processing device 10b described in the second embodiment. Note that the hardware configuration of the information processing device 10d is the same as that of the information processing device 10b.

[0240] The mobile terminal 32d included in the information processing apparatus 10d receives the 3D model 90M of the subject 90 generated by the 3D model generation device 20 described in the first embodiment. Then, the mobile terminal 32d generates a virtual image J while viewing the 3D model 90M from a virtual viewpoint V. When a CG object is also set in the virtual image J generated in this way, the mobile terminal 32d refers to the score map M (θ, φ) and sets the CG objects collectively in an area based on low image quality.

[0241] Fig.2132 d is a functional block diagram showing an example of the functional configuration of the mobile terminal according to the fourth embodiment. The mobile terminal 32 d includes a score map acquisition unit 33 , a CG object acquisition unit 65 , a CG object arrangement area calculation unit 66 , and a rendering processing unit 95 .

[0242] The score map acquisition unit 33 acquires the score map M(θ, φ) from each of the 3D model generating devices 20 a , 20 b , . . . .

[0243] The CG object acquisition unit 65 acquires a CG object to be set in the virtual image J from a server device (not shown) or the like.

[0244] The CG object arrangement area calculation unit 66 calculates an area suitable for setting a CG object with reference to the score map M (θ, φ). An area suitable for setting a CG object is, for example, an area where information is missing (i.e., an area not covered by the real camera 70) or an area with low image quality. Note that the CG object arrangement area calculation unit 66 is an example of an arrangement position calculation unit in the present disclosure.

[0245] The rendering processing unit 95 performs rendering of the 3D model 90M viewed from the virtual viewpoint V and generates a virtual image J. Furthermore, the rendering processing unit 95 sets a CG object in the virtual image J.

[0246] As described above, according to the information processing device 10d of the fourth embodiment, the CG object layout area calculation unit 66 (layout position calculation unit) calculates the layout position when setting another image on the generated virtual image J based on the score mapping map M(θ, φ) (mapping map).

[0247] Therefore, in a case where another image is set on the virtual image J, the 3D model 90M displayed with good image quality can be not disturbed by centrally setting the other image in the area with low image quality of the virtual image J.

[0248] (5. Fifth Embodiment)

[0249] An information processing apparatus 10 e (not shown) according to the fourth embodiment of the present disclosure is an apparatus that adds another function to the 3D model generating device 20 in the information processing apparatus 10 b described in the second embodiment.

[0250] In the case where the 3D model 90M has a texture associated with the viewpoint position, it is necessary to send a plurality of images captured by the real camera 70 to a device that performs rendering. At this time, reducing the amount of data to be sent as much as possible is an issue to be solved.

[0251] When a plurality of images of a subject 90 captured by a real camera 70 are transmitted to a mobile terminal 32b, a 3D model generation device 20 included in an information processing device 10e generates a packed image in which a plurality of images are stored in one image to transmit the packed image, thereby improving transmission efficiency. Specifically, the 3D model generation device 20 according to the present disclosure packs images with high image quality in a larger size based on a score map M (θ, φ) generated by the 3D model generation device 20.

[0252] Fig. 22 3D model generating device 20 includes a score map generating unit 67 , a packed image generating unit 68 , and a packed image transmitting unit 69 .

[0253] The score map generation unit 67 generates a score map M(θ, φ) of the generated 3D model 90M.

[0254] When generating a packed image in which a plurality of images of the subject 90 captured by the real camera 70 are stored in one image, the packed image generation unit 68 performs packing with higher resolution as the image quality is higher based on the score map M(θ, φ). Note that the packed image generation unit 68 is an example of an image generation unit in the present disclosure.

[0255] The packaged image transmission unit 69 transmits the packaged image generated by the packaged image generation unit 68 to the mobile terminal 32 b .

[0256] Fig.23 3D model generation device according to the fifth embodiment. Fig.23 The score map M(θ, φ) is shown in the upper part of .

[0257] In M(θ, φ), clusters C1, C2, and C3 predicted to have high image quality are found. In this case, when generating a packed image, the packed image generation unit 68 packs images captured in the areas of clusters C1, C2, and C3 at high resolution.

[0258] Specifically, Fig.23 In the packing image 130 shown, the image 130a captured in the area of ​​cluster C1, the image 130b captured in the area of ​​cluster C2, and the image 130c captured in the area of ​​cluster C3 are packed with high resolution. Note that in order to perform packing in a high-resolution state, the size of the image is increased in the present embodiment. Then, the images (130d to 130i) other than these three images are packed with low resolution, that is, reduced size.

[0259] After receiving such a packed image 130, the mobile terminal 32b generates a virtual image J having high image quality and performs rendering processing in the areas of the clusters C1, C2, and C3 having high image quality. In addition, in areas other than the clusters C1, C2, and C3, the mobile terminal 32b generates a virtual image J having an image quality corresponding to the image stored in the received packed image 130 and performs rendering processing.

[0260] As described above, according to the 3D model generating device 20 of the fifth embodiment, when generating a packed image 130 in which a plurality of images obtained by imaging the 3D model 90M (3D object) are stored in one image, the packed image generating unit 68 (image generating unit) performs packing based on the score mapping map M (θ, φ) (mapping map) with a larger size, that is, with a higher resolution, as the image quality is higher.

[0261] Therefore, the transmission efficiency of the 3D model 90M having the texture associated with the viewpoint position can be improved.

[0262] In addition, the effects in each embodiment described in this specification are merely examples and are not limited, and other effects may exist. In addition, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications may be made without departing from the gist of the present disclosure.

[0263] For example, the present disclosure may also have the following configurations. (1)

[0265] An information processing device, comprising:

[0266] An image quality prediction unit predicts, for a 3D object, a relationship between a position of a virtual viewpoint from which the 3D object is viewed and an image quality of a virtual image obtained when the 3D object is viewed from the position of the virtual viewpoint. (2)

[0268] The information processing device according to (1), further comprising:

[0269] A movement trajectory generating unit generates a movement trajectory of the virtual viewpoint based on arrangement positions of a plurality of 3D objects and the image quality of the virtual image predicted by the image quality predicting unit for each of the 3D objects. (3)

[0271] The information processing device according to (1) or (2), wherein:

[0272] The image quality prediction unit predicts the image quality based on an optical parameter of an imaging unit used when generating the 3D object and a distance between the imaging unit and the 3D object. (4)

[0274] The information processing device according to claim (3), wherein:

[0275] The optical parameter is a parameter related to a ratio of the 3D object to an image captured by the imaging unit. (5)

[0277] The information processing device according to (3) or (4), wherein:

[0278] The direction in which the 3D object is viewed includes at least one of information indicating whether the imaging unit captures the 3D object and information indicating whether the imaging unit captures the 3D object from the front. (6)

[0280] The information processing device according to any one of (1) to (5), wherein:

[0281] The image quality prediction unit generates, for the 3D object, a map in which a parameter related to an image quality of an image obtained by imaging the 3D object is quantified for each direction in which the 3D object is viewed. (7)

[0283] The information processing device according to any one of (1) to (6), wherein:

[0284] The image quality prediction unit plots a quantized parameter related to the image quality of the image at a position corresponding to an imaging position of the image; and

[0285] The map is generated by diffusing the mapped parameter related to the image quality of the image from a mapped position to surroundings. (8)

[0287] The information processing device according to claim (7), wherein:

[0288] The image quality prediction unit performs a smoothing operation on the image quality of the image plotted on the map to diffuse the image quality to surroundings. (9)

[0290] The information processing device according to any one of (1) to (8), wherein:

[0291] The 3D object has texture information related to the viewpoint position. (10)

[0293] The information processing device according to any one of (6) to (9), further comprising:

[0294] An arrangement evaluation unit evaluates arrangement conditions of imaging units used when generating the 3D object based on the map. (11)

[0296] The information processing device according to any one of (6) to (9), further comprising:

[0297] a browsing prohibited area calculation unit that calculates, based on the map, a position where a virtual viewpoint is not set when a virtual image is generated from the 3D object; and

[0298] A video processing unit is configured to generate a virtual image of the 3D object not according to the position of the virtual viewpoint when the virtual viewpoint of the 3D object matches the position of the virtual viewpoint calculated by the browsing forbidden area calculation unit. (12)

[0300] The information processing device according to any one of (6) to (9), further comprising:

[0301] An arrangement position calculation unit calculates an arrangement position when arranging other images in the generated virtual image based on the map. (13)

[0303] The information processing device according to any one of (6) to (9), further comprising:

[0304] An image generating unit performs packing of images having higher image quality at higher resolution based on the map when generating a packed image storing a plurality of images obtained by imaging the 3D object in one image. (14)

[0306] The information processing device according to any one of (2) to (13), wherein:

[0307] The multiple 3D objects are independently generated. (15)

[0309] A video generation method, comprising:

[0310] For a 3D object, predicting a relationship between a position of a virtual viewpoint for viewing the 3D object and an image quality of a virtual image obtained when viewing the 3D object from the position of the virtual viewpoint; and

[0311] A video image is generated based on a movement trajectory of the virtual viewpoint generated using a relationship between an arrangement position of the 3D object and an image quality of the virtual image. (16)

[0313] A computer-readable storage medium stores a program for causing a computer to:

[0314] An image quality prediction unit predicts, for a 3D object, a relationship between a position of a virtual viewpoint for viewing the 3D object and an image quality of a virtual image obtained when viewing the 3D object from the position of the virtual viewpoint.

[0315] Reference numerals list

[0316] 10a Information processing device

[0317] 20 3D model generation equipment

[0318] 30 Camera path generation device

[0319] 32b Mobile Terminal

[0320] 34 Score Map Analysis Unit (Placement Evaluation Unit)

[0321] 60 Virtual camera installation prohibited area calculation unit (browsing prohibited area calculation unit)

[0322] 64 Prohibited Area Video Processing Unit (Video Processing Unit)

[0323] 66 CG Object placement area calculation unit (placement position calculation unit)

[0324] 70, 70a, 70b, 70c, 70d, 70e real camera device (imaging unit imaging device)

[0325] 72, 72a, 72b, 72c, 72d virtual camera device

[0326] 83 score map generation unit (image quality prediction unit)

[0327] 90, 90a, 90b Subject

[0328] 90M、90Ma、90Mb 3D Model (3D Object)

[0329] 91 3D model layout specified unit

[0330] 92 Camera device path generation unit (movement trajectory generation unit)

[0331] 93 Camera path display unit

[0332] Cp Camera path

[0333] D Distance

[0334] e Number of pixels

[0335] f Focal length

[0336] h Subject size

[0337] Iq Image Quality Score

[0338] J, J1, J2 virtual images

[0339] M(θ, φ) Score Mapping Graph (Mapping Graph)

[0340] p Image size

[0341] s Image sensor size

[0342] V, V1, V2 virtual viewpoint

Claims

1. An information processing device, comprising: an image quality prediction unit configured to predict, for a 3D object, a relationship between a position of a virtual viewpoint for viewing the 3D object and an image quality of a virtual image obtained when viewing the 3D object from the position of the virtual viewpoint, Wherein, the image quality prediction unit is further configured to generating a map for the 3D object in which a parameter related to the image quality of an image obtained by imaging the 3D object is quantified for each direction from which the 3D object is viewed, and Wherein, the image quality prediction unit is further configured to plotting a quantified parameter related to image quality of the image at a location corresponding to an imaging location of the image, and The map is generated by diffusing the mapped parameter related to the image quality of the image from a mapped position to surroundings.

2. The information processing device according to claim 1, further comprising: A movement trajectory generating unit is configured to generate a movement trajectory of the virtual viewpoint based on arrangement positions of a plurality of 3D objects and the image quality of the virtual image predicted by the image quality predicting unit for each of the 3D objects.

3. The information processing device according to claim 1, wherein: The image quality prediction unit is further configured to predict the image quality based on optical parameters of an imaging unit used when generating the 3D object and a distance between the imaging unit and the 3D object.

4. The information processing device according to claim 3, wherein: The optical parameter is a parameter related to a ratio of the 3D object to an image captured by the imaging unit.

5. The information processing device according to claim 3, wherein: The direction in which the 3D object is viewed includes at least one of information indicating whether the imaging unit captures the 3D object and information indicating whether the imaging unit captures the 3D object from the front.

6. The information processing device according to claim 1, wherein: The image quality prediction unit is further configured to perform a smoothing operation on the image quality of the image plotted on the map to diffuse the image quality to surroundings.

7. The information processing device according to claim 1, wherein: The 3D object has texture information related to the viewpoint position.

8. The information processing device according to claim 1, further comprising: An arrangement evaluation unit is configured to evaluate arrangement conditions of imaging units used when generating the 3D object based on the map.

9. The information processing device according to claim 1, further comprising: a browsing prohibited area calculation unit configured to calculate, based on the map, a position where a virtual viewpoint is not set when a virtual image is generated according to the 3D object; as well as A video processing unit is configured to not generate a virtual image of the 3D object according to the position of the virtual viewpoint when the virtual viewpoint of the 3D object matches the position of the virtual viewpoint calculated by the browsing forbidden area calculation unit.

10. The information processing device according to claim 1, further comprising: An arrangement position calculation unit calculates an arrangement position when arranging other images in the generated virtual image based on the map.

11. The information processing device according to claim 1, further comprising: An image generating unit is configured to perform packing of images having higher image quality at higher resolution based on the map when generating a packed image storing a plurality of images obtained by imaging the 3D object in one image.

12. The information processing device according to claim 2, wherein: The multiple 3D objects are independently generated.

13. A video generation method, comprising: For a 3D object, predicting a relationship between a position of a virtual viewpoint for viewing the 3D object and an image quality of a virtual image obtained when viewing the 3D object from the position of the virtual viewpoint; as well as generating a video image based on a movement trajectory of the virtual viewpoint generated using a relationship between an arrangement position of the 3D object and an image quality of the virtual image, Wherein, the method further comprises: generating a map for the 3D object in which a parameter related to the image quality of an image obtained by imaging the 3D object is quantified for each direction from which the 3D object is viewed, and Wherein, the method further comprises: plotting a quantified parameter related to image quality of the image at a location corresponding to an imaging location of the image, and The map is generated by diffusing the mapped parameter related to the image quality of the image from a mapped position to surroundings.

14. A computer-readable storage medium storing a program for causing a computer to: an image quality prediction unit configured to predict, for a 3D object, a relationship between a position of a virtual viewpoint for viewing the 3D object and an image quality of a virtual image obtained when viewing the 3D object from the position of the virtual viewpoint, in, The image quality prediction unit is further configured to generating a map for the 3D object in which a parameter related to the image quality of an image obtained by imaging the 3D object is quantified for each direction from which the 3D object is viewed, and Wherein, the image quality prediction unit is further configured to plotting a quantified parameter related to image quality of the image at a location corresponding to an imaging location of the image, and The map is generated by diffusing the mapped parameter related to the image quality of the image from a mapped position to surroundings.

Images