Subject modeling device, rendering device and program

Abstract

To achieve viewpoint dependency rendering with little artifact and obtain a rendering result having natural texture.SOLUTION: A constitution point selection section 11 of a subject modeling device 1 selects a constitution point position Qn from a point group; a parameter selection section 13 selects a camera parameter pm from a camera parameter group; and an image selection section 14 selects an image Im from a photographed image group. A direction vector calculation section 15 calculates a direction vector Λm,n from the constitution point position Qn to a camera position Wm. A projection section 16 projects the constitution point position Qn to image coordinates ξm,n using the camera parameter pm. A pixel value selection section 17 selects a pixel value Vm,n from the image Im. A first multiplexing section 18 obtains observation ray information Ln from the direction vector Λ1,n to ΛM,n and the pixel value V1,n to VM,n, and a second multiplexing section 19 obtains a point group K with ray information from the constitution point position Q1 to QN and the observation ray information L1 to LN.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for recording and drawing the shape and light rays of an object, and in particular to an object modeling device that models an object for three-dimensional computer graphics, a rendering device that renders a model of an object according to a viewpoint, and a program. [Background technology]

[0002] Conventionally, a point cloud is known as a model for describing the shape of an object. This point cloud is a group of points that expresses the shape as a set of points having three-dimensional coordinates, and may record color information (e.g., red, green, and blue components) for each constituent point of the point cloud.

[0003] Light fields are also known as a method for describing 3D images. Light fields can be expressed in four degrees of freedom, consisting of two-dimensional image coordinates on the image plane and two-dimensional directions of light rays toward the image coordinates, or in five degrees of freedom, describing the two-dimensional directions of light rays originating from points (three-dimensional) on the surface or inside of an object.

[0004] Also, a technique has been disclosed for an immersive video format in which a point cloud representing a subject is obtained from an image, a parametric surface of the subject is extracted from a light field, and a texture map is obtained in relation to the parametric surface (see, for example, Patent Document 1).

[0005] A method for reconstructing a point cloud from light field data has also been disclosed (see, for example, Patent Document 2).

[0006] Also, a method has been disclosed for improving the recognition rate of character information by combining a light field and a point cloud to remove specular reflection components (see, for example, Patent Document 3).

[0007] Furthermore, in order to obtain the texture observed from a certain viewpoint CE for each triangular patch that constitutes a 3D model of a subject, a method has been proposed in which an image captured from a direction closest to the viewpoint CE is used from among images captured from multiple viewpoints (see, for example, non-patent document 1). [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Special Publication No. 2019-534500 [Patent Document 2] Special Publication No. 2019-534606 [Patent Document 3] JP 2018-45464 A [Non-patent literature]

[0009] [Non-Patent Document 1] Yusuke Uno, Yuta Nakajima, Norihiko Kawai, Tomokazu Sato, Naokazu Yokoya, "Restoring the Contour Shape of Virtualized Real Objects Based on Viewpoint-Dependent Texture and Geometry for Augmented Reality", Technical Report of the Institute of Electronics, Information and Communication Engineers, IEICE Technical Report 112(385), 223-228, 2013-01-23 Summary of the Invention [Problem to be solved by the invention]

[0010] In point clouds that record the aforementioned color information, it is common for only one piece of color information that is independent of the viewpoint to be recorded. For this reason, objects cannot be accurately described when the apparent color or brightness changes when the viewpoint is changed, such as in specular reflection and anisotropic reflection. As a result, the rendering result may show an unnatural texture depending on the viewpoint.

[0011] In addition, the aforementioned 4-degree-of-freedom light field records color information for each pixel of a given image plane for each direction of incident light (2 degrees of freedom). As a result, within a limited range of viewpoint movement that does not cause occlusion within or between subjects, it is possible to reproduce something close to the actual light, making it possible to present a three-dimensional effect or texture. However, as mentioned above, since the range of viewpoint movement is limited, its application is limited to so-called 3DoF+.

[0012] In order to correctly reproduce the light rays from the subject while supporting the so-called 6DoF, which allows the viewpoint to be moved freely, it is necessary to record color information in association with the direction of the light rays emitted from each small area on the subject (light field with 5 degrees of freedom). In other words, it is necessary to record color information for each direction (2 degrees of freedom) of each 3D spatial position (3 degrees of freedom) where the subject exists. In this case, there is a problem that a huge amount of storage capacity is required.

[0013] Furthermore, as described in the aforementioned Non-Patent Document 1, according to a method of adaptively selecting texture images depending on the viewpoint, it is only necessary to store images for the texture images for the number of viewpoints, which makes it possible to reduce the storage capacity.

[0014] However, the optimal texture image to be selected may vary for each region of the object or for each viewpoint position, and the rendering result may have spatial and temporal discontinuities in pixel values ​​due to switching of texture images. These discontinuities are perceived as artifacts, resulting in a degradation of subjective image quality.

[0015] Therefore, the present invention has been made to solve the above-mentioned problems, and its object is to provide a subject modeling device, a rendering device, and a program that can realize viewpoint-dependent rendering with fewer artifacts and obtain rendering results that exhibit a natural texture. [Means for solving the problem]

[0016] In order to solve the above problem, the object modeling device of claim 1 is an object modeling device that inputs a point cloud consisting of positional information of a plurality of constituent points representing a shape of an object, a camera parameter group consisting of camera parameters of a plurality of cameras, and an image group consisting of images of the object captured by each of the plurality of cameras, and models the point cloud to generate a point cloud with light ray information, in which an index for sequentially selecting the constituent points that constitute the point cloud is n, and an index for sequentially selecting the camera parameters that constitute the camera parameter group and the images that constitute the image group is m, and a direction calculation is performed to calculate directional information regarding the direction of a ray from the constituent point to the camera based on the camera position included in the camera parameters of index m and the position information of the constituent point of index n. a projection unit that projects the position information of the component point of index n onto image coordinates of the image of index m using the camera parameters of index m; a selection unit that selects, from the image of index m, pixel value information of the image coordinates projected by the projection unit; a first multiplexing unit that generates observation ray information of index n by multiplexing, for index n, the direction information of all of the indexes m calculated by the direction calculation unit and the pixel value information of all of the indexes m selected by the selection unit; and a second multiplexing unit that generates the point group with light ray information by multiplexing the position information of all of the indexes n and the observation ray information of all of the indexes n generated by the first multiplexing unit.

[0017] Moreover, the subject modeling device of claim 2 is the subject modeling device of claim 1, wherein the selection unit selects color information of the image coordinates from the image, and the first multiplexing unit multiplexes, for the index n, the direction information of all of the indexes m calculated by the direction calculation unit and the color information of all of the indexes m selected by the selection unit, thereby generating the observation ray information of the index n.

[0018] Further, a rendering device according to claim 3 uses a point cloud with light ray information, in which positional information and light ray information of a plurality of constituent points representing a shape of a subject are multiplexed, and a camera parameter p of a virtual camera, to render the point cloud with light ray information and generate a composite image photographed by the virtual camera, the light ray information being information generated based on the camera parameters of a plurality of cameras and images of the subject photographed by each of the plurality of cameras, and including directional information regarding the direction of a ray from the constituent point to the camera and pixel value information regarding pixels onto which the constituent point is projected, an index for sequentially selecting the constituent points constituting the point cloud with light ray information is n, and an index for sequentially selecting the directional information and the pixel value information included in the light ray information is m, and the positional information of the constituent point of index n is rendered using the camera parameter p. a direction calculation unit which calculates virtual direction information regarding a direction of a ray from the component point to the virtual camera based on a camera position of the virtual camera and the position information of the component point of the index n included in the camera parameter p; a weight value calculation unit which calculates a weight for the index n based on the direction information of the index m included in the ray information and the virtual direction information calculated by the direction calculation unit; a synthesis unit which weights and synthesizes, for the index n, the pixel value information of all the indexes m included in the ray information by using the weight values ​​of all the indexes m calculated by the weight value calculation unit, to obtain a representative value; and an image generation unit which places, for all the indexes n, the representative values ​​obtained by the synthesis unit at the image coordinates projected by the projection unit, and generates the synthetic image.

[0019] The rendering device of claim 4 is the rendering device according to claim 3, characterized in that the light ray information is information including the direction information and color information related to a pixel onto which the component point is projected, an index for sequentially selecting the direction information and the color information included in the light ray information is m, and the synthesis unit weights and synthesizes the color information of all the indexes m included in the light ray information for the index n using the weight values ​​of all the indexes m to obtain the representative value.

[0020] Further, the rendering device of claim 5 is characterized in that, in the rendering device described in claim 3 or 4, the weight value calculation unit calculates the weight value for the index n by applying a broad monotonically decreasing function to the absolute value of the angle between the direction information of the index m included in the ray information and the virtual direction information.

[0021] The program of claim 6 further comprises a computer constituting an object modeling device that receives as input a point cloud consisting of positional information of a plurality of constituent points representing a shape of an object, a camera parameter group consisting of camera parameters of a plurality of cameras, and an image group consisting of images of the object captured by each of the plurality of cameras, and models the point cloud to generate a point cloud with light ray information, the computer comprising: a direction calculation unit that calculates directional information relating to the direction of a ray from the constituent point to the camera based on the camera position included in the camera parameters of index m and the position information of the constituent point of index n, where n is an index for sequentially selecting the constituent points that constitute the point cloud and m is an index for sequentially selecting the camera parameters that constitute the camera parameter group and the images that constitute the image group; the camera parameters of the index m being used to project the position information of the component point of the index n onto image coordinates of the image of the index m; a selection unit being used to select, from the image of the index m, pixel value information of the image coordinates projected by the projection unit; a first multiplexing unit being used to generate observation ray information of the index n by multiplexing, for the index n, the direction information of all of the indexes m calculated by the direction calculation unit and the pixel value information of all of the indexes m selected by the selection unit; and a second multiplexing unit being used to generate a point group with light ray information by multiplexing the position information of all of the indexes n and the observation ray information of all of the indexes n generated by the first multiplexing unit.

[0022] A program according to claim 7 provides a rendering device that uses a point cloud with light ray information, in which positional information and light ray information of a plurality of constituent points representing a shape of a subject are multiplexed, and a camera parameter p of a virtual camera, to render the point cloud with light ray information and generate a composite image photographed by the virtual camera, the light ray information being information generated based on the camera parameters of a plurality of cameras and images of the subject photographed by each of the plurality of cameras, and including directional information regarding the direction of a ray from the constituent point to the camera and pixel value information regarding pixels onto which the constituent point is projected, an index for sequentially selecting the constituent points constituting the point cloud with light ray information is n, and an index for sequentially selecting the directional information and the pixel value information included in the light ray information is m, and the position information of the constituent point of the index n is calculated using the camera parameter p. a direction calculation unit which calculates virtual direction information regarding the direction of a ray from the constituent point to the virtual camera based on the camera position of the virtual camera and the position information of the constituent point of the index n included in the camera parameter p; a weight value calculation unit which calculates a weighting value for the index n based on the direction information of the index m included in the ray information and the virtual direction information calculated by the direction calculation unit; a synthesis unit which weights and synthesizes, for the index n, the pixel value information of all the indexes m included in the ray information using the weighting values ​​of all the indexes m calculated by the weight value calculation unit, to obtain a representative value; and an image generation unit which places, for all the indexes n, the representative values ​​obtained by the synthesis unit in the image coordinates projected by the projection unit, and generates the synthetic image. Effect of the Invention

[0023] As described above, according to the present invention, it is possible to realize viewpoint-dependent rendering with few artifacts and obtain rendering results that exhibit a natural texture. [Brief description of the drawings]

[0024] [Figure 1] 1 is a block diagram showing an example of the configuration of a subject modeling device according to an embodiment of the present invention. [Diagram 2] 13 is a flowchart showing an example of processing performed by the object modeling device. [Diagram 3] FIG. 2 is a conceptual diagram illustrating an example of processing performed by the object modeling device. [Figure 4] FIG. 2 is a conceptual diagram illustrating a point group K with ray information. [Diagram 5] 1 is a block diagram showing an example of the configuration of a rendering device according to an embodiment of the present invention. [Figure 6] 13 is a flowchart illustrating an example of processing performed by a rendering device. [Figure 7] FIG. 2 is a conceptual diagram illustrating an example of processing by a rendering device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. [Subject Modeling Device] First, an object modeling device according to an embodiment of the present invention will be described below. Fig. 1 is a block diagram showing an example of the configuration of an object modeling device according to an embodiment of the present invention.

[0026] This object modeling device 1 calculates the position vector (position of each point) Q of each point constituting the object. n A point cloud including each camera, camera parameters p m A set of camera parameters, and an image I m Then, subject modeling device 1 generates and outputs a point group K with light ray information based on the point group, the camera parameter group, and the captured image group.

[0027] A point cloud describes the shape of a subject as a set of points in a three-dimensional space. That is, a point cloud is made up of one or more constituent points, and each constituent point has a three-dimensional coordinate constituent point position Q as position information.n The number of constituent points of the point cloud (number of constituent points) is N (N is an integer greater than or equal to 1), and the position of the nth constituent point (n is an integer greater than or equal to 1 and less than or equal to N) is Q n Let us assume that.

[0028] The captured image group consists of multiple images I taken of the subject by cameras with different viewpoints. m Let the number of rays (number of images, number of cameras) be M (M is an integer greater than or equal to 1), and the mth image (m is an integer greater than or equal to 1 and less than or equal to M) be I m Let us assume that.

[0029] The camera parameters are set for each image I m The m-th camera parameter that constitutes the camera parameter group is the parameter of the optical system (camera and lens) when the image I m Let p be the camera parameters when the image was taken. m Let us assume that.

[0030] Camera parameters p m are parameters necessary for associating the three-dimensional coordinates of the subject scene (hereinafter referred to as "world coordinates") with the image coordinates of the image, and consist of information such as the position of the optical principal point of the lens, the attitude of the camera, the size and arrangement of the pixels of the image sensor, and the focal length of the lens.

[0031] In the following, the coordinates of the space in which the subject and multiple cameras are arranged are called world coordinates X, and the coordinates of image I m Let the above coordinates be image coordinates x, and the transformation from world coordinates X to image coordinates x using camera parameters p is expressed by the following equation using function h.

number

[0032] As shown in FIG. 1, the subject modeling device 1 includes a first counter 10, a point selection unit 11, a second counter 12, a parameter selection unit 13, an image selection unit 14, a direction vector calculation unit (direction calculation unit) 15, a projection unit 16, a pixel value selection unit 17, a first multiplexing unit 18, and a second multiplexing unit 19.

[0033] The first counter 10 counts sequentially from 1 to N to generate an index n for sequentially selecting each constituent point of the input point cloud, and outputs the count value as index n to the constituent point selection unit 11 and the second multiplexing unit 19.

[0034] The component point selection unit 11 selects the component point positions Q1, . . . , Q N The composing point selection unit 11 inputs a group of points including the nth composing point corresponding to the index n from the group of points, and also inputs an index n from the first counter 10. Then, the composing point selection unit 11 selects the composing point position Q n Select the point position Q n to the direction vector calculation unit 15, the projection unit 16, and the second multiplexing unit 19.

[0035] The second counter 12 counts the number of images I1, . . . , I M and each of the camera parameters p1, . . . , p M In order to generate an index m for sequentially selecting the images, the second counter 12 counts from 1 to M. Then, the second counter 12 outputs the count value as the index m to the parameter selection unit 13, the image selection unit 14, and the first multiplexing unit 18.

[0036] The parameter selection unit 13 selects the camera parameters p1, . . . , p M The parameter selection unit 13 inputs a camera parameter group consisting of the camera parameters p m Select the camera parameters p m to the direction vector calculation unit 15 and the projection unit 16.

[0037] The image selection unit 14 selects images I1,...,I MThe image selection unit 14 inputs a group of captured images consisting of the image I, and also inputs an index m from the second counter 12. The image selection unit 14 then selects the m-th image I corresponding to the index m from the group of captured images. m Select the image I m to the pixel value selection unit 17.

[0038] The direction vector calculation unit 15 receives the component point position Q from the component point selection unit 11. n and at the same time, the camera parameter p m Enter the camera parameters p m The world coordinates of the optical principal point of the lens (hereinafter referred to as the "camera position") W m The following shall be included:

[0039] The direction vector calculation unit 15 calculates the position of the component point Q by the following formula: n From camera position W m The direction (hereinafter referred to as the "direction vector") Λ, which is the direction information of the vector of the light ray reaching m,n Calculate the direction vector Λ m,n to the first multiplexing unit 18.

number

[0040] The projection unit 16 receives the composition point position Q from the composition point selection unit 11. n and at the same time, the camera parameter p m Then, the projection unit 16 inputs the camera parameter p m Using the point position Q n Let ξ be the image coordinate. m,n and project it to the image coordinates ξ m,n to the pixel value selection unit 17.

number

[0041] That is, the projection unit 16 uses the camera parameter p mUsing this, the nth point in the world coordinate system is m By projecting the image coordinates of n Let ξ be the image coordinate. m,n Convert to.

[0042] The pixel value selection unit 17 selects the image I from the image selection unit 14. m and at the same time, the image coordinates ξ m,n Then, the pixel value selection unit 17 inputs the image I m From the image coordinate ξ m,n The pixel value I(ξ m,n ) and use this as the pixel value V m,n to the first multiplexing unit 18.

[0043] The pixel value selection unit 17 selects the pixel value of the image I m From pixel value I(ξ m,n ) when selecting the image coordinate ξ m,n If there is no pixel value information corresponding to the image coordinate ξ m,n Alternatively, the pixel value that is closest to the image coordinate ξ and has pixel value information may be selected. m,n When there is no pixel value information corresponding to the image coordinate ξ m,n If the coordinate value indicating the image coordinate ξ is non-integer, m,n indicates a position that does not match the pixel arrangement on the image sensor.

[0044] Furthermore, the pixel value selection unit 17 selects the image coordinate ξ m,n If there is no pixel value information corresponding to the image coordinate ξ m,n The pixel value may be calculated by interpolation or extrapolation, taking into consideration the pixel values ​​of the pixels in the vicinity (for example, up to a predetermined number of pixels from the closest position) and their positional relationships. In this case, for example, an interpolation method such as bilinear interpolation, bicubic interpolation, Lanczos interpolation, or spline interpolation may be used for the interpolation.

[0045] The first multiplexer 18 receives the index m from the second counter 12 and the direction vector Λ from the direction vector calculator 15. m,nThe pixel value V m,n Enter the following respectively.

[0046] The first multiplexing unit 18 multiplexes the direction vector Λ of all indexes m for index n. 1,n ,···,Λ M,n and pixel value V 1,n ,···,V M,n The resulting matrix (hereinafter referred to as the "observation ray information" at the nth component point) L n Then, the first multiplexing unit 18 obtains the observation light information L n to the second multiplexing unit 19.

[0047] That is, the first multiplexing unit 18 multiplexes the m-th direction vector Λ for the n-th component point. m,n and pixel value V m,n The set of results is the observation light information L n is required.

[0048] For example, the observation light information L n is expressed by the following formula:

number

[0049] As shown in the above formula (4), the first multiplexing unit 18 has a direction vector Λ 1,n ,···,Λ M,n Including observation light information L n The first multiplexing unit 18 may obtain a direction vector Λ 1,n ,···,Λ M,n is converted into a unit direction vector, and the observation light information L n Furthermore, the first multiplexing unit 18 may obtain a direction vector Λ 1,n ,···,Λ M,n is converted into two-degree-of-freedom angle information (e.g., azimuth angle and elevation angle), and the observation light information L n It is also possible to obtain the following.

[0050] For example, the first multiplexing unit 18 has a direction vector Λ 1,n ,···,Λ M,n is the azimuth angle φ 1,n , ,φ M,n and the elevation angle θ 1,n , ,θ M,n The observation light information L including these angles is converted into n Request.

number

[0051] The second multiplexing unit 19 receives the index n from the first counter 10 and the component point position Q from the component point selection unit 11. n The first multiplexing unit 18 outputs the observation light information L n Enter the following respectively.

[0052] The second multiplexing unit 19 multiplexes the component point positions Q1, . . . , Q N and the observation light information L1, ,L N and obtains a resultant matrix (hereinafter, referred to as a “point group with ray information”) K. Then, the second multiplexing unit 19 outputs the point group K with ray information.

[0053] That is, the second multiplexing unit 19 multiplexes the component point position Q n and observation light information L n The set of results is calculated as a point group K with ray information.

[0054] For example, the point group K with ray information is expressed by the following equation.

number

[0055] (process) Next, a detailed description will be given of the flow of processing by the object modeling device 1 shown in Fig. 1. Fig. 2 is a flowchart showing an example of processing by the object modeling device 1, and Fig. 3 is a conceptual diagram explaining the example of processing by the object modeling device 1.

[0056] In FIG. 3, the position Q of the nth point in the point group reflecting the shape of the subject 3 is n The camera parameters of the cameras 4-1, 4-m, and 4-M are respectively p1, p m ,p M Let these camera positions be W1, W m ,W M Let us assume that.

[0057] The images taken by the cameras 4-1, 4-m, and 4-M are I1, I m ,I M Then, the position of the component point Q n The constituent points of images I1,I m ,I M The image coordinates projected onto the coordinates of ξ 1,n ,ξ m,n ,ξ M,n In addition, the image coordinates ξ 1,n ,ξ m,n ,ξ M,n The pixel values ​​of V 1,n ,V m,n ,V M,n Let us assume that.

[0058] First, the first counter 10 generates an index n=1 with a count value of 1 (step S201, n←1). Then, the composing point selection unit 11 selects the composing point position Q of the n-th composing point corresponding to the index n from the point group. n (Step S202). As shown in FIG. 3, the position Q of the n-th component point is selected from the point group reflecting the shape of the subject 3. n is selected.

[0059] The second counter 12 generates an index m with a count value of 1 (step S203, m←1). Then, the parameter selection unit 13 selects the camera parameter p m (step S204), the image selection unit 14 selects the image I m (Step S205).

[0060] As shown in FIG. 3, when index m=1, camera parameter p1 and image I1 are selected, and when index m=M, camera parameter p M and Image I M That is, for index m, the camera parameter p m and Image I m is selected.

[0061] The direction vector calculation unit 15 calculates the position of the component point Q n From camera position W m Direction vector Λ leading to m,n (step S206), and the projection unit 16 calculates the camera parameter p m Using the point position Q n Let ξ be the image coordinate. m,n (Step S207). Then, the pixel value selection unit 17 projects the image I m From the image coordinate ξ m,n Pixel value V m,n (Step S208).

[0062] As shown in Figure 3, for index m=1, the direction vector Λ 1,n , image coordinate ξ 1,n and pixel value V 1,n For index m=M, the direction vector Λ M,n , image coordinate ξ M,n and pixel value V M,n That is, for index m, the direction vector Λ m,n , image coordinate ξ m,n and pixel value V m,n is obtained.

[0063] The second counter 12 determines whether or not the index m is less than the number of rays M (step S209). If the second counter 12 determines in step S209 that the index m is less than the number of rays M (step S209: Y), it adds 1 to the index m (step S210, m←m+1) and proceeds to step S204. As a result, the processes of steps S204 to S208 are performed for the index m.

[0064] On the other hand, if the second counter 12 determines in step S209 that the index m is not less than the number of rays M (step S209: N), the process proceeds to step S211.

[0065] In this case, for the index n (initially n=1) and all the indices m=1, . . . , M, the direction vector Λ 1,n ,···,Λ M,n and pixel value V 1,n ,···,V M,n is obtained.

[0066] The first multiplexing unit 18 proceeds from step S209(N) to obtain the direction vectors Λ of all indexes m for the index n. 1,n ,···,Λ M,n and pixel value V 1,n ,···,V M,n The observation light information L n is calculated (step S211).

[0067] The first counter 10 judges whether the index n is less than the number of constituent points N (step S212). If the first counter 10 judges in step S212 that the index n is less than the number of constituent points N (step S212: Y), the first counter 10 adds 1 to the index n (step S213, n←n+1) and proceeds to step S202. As a result, the processes of steps S202 to S211 are performed for the index n.

[0068] On the other hand, when the first counter 10 determines in step S212 that the index n is not less than the number of constituent points N (step S212: N), the process proceeds to step S214. In this case, for all indexes n=1, . . . , N, the constituent point positions Q1, . . . , Q N and the observation light information L1, ,L N is obtained.

[0069] The second multiplexing unit 19 moves on from step S212(N) and calculates the position of each of the constituent points Q1, . . . , Q N and the observation light information L1, ,L N are multiplexed to obtain a point group K with ray information (step S214).

[0070] FIG. 4 is a conceptual diagram for explaining a point group K with light ray information, and shows an example in which the number of cameras is five (number of light rays M=5).

[0071] Focus on one of the points 101, and define the position of this point 101 as Q n For one point 101, the position Q n and the observed light ray information L consisting of components l1, . . . , l5 of the light ray information observed by the five cameras 4-1, . . . , 4-5. n and are multiplexed, and the components of the point group K with ray information (Q n ,L n ) is obtained.

[0072] In the example of the above formula (5), for the point 101, the position Q n In addition, the observation light information L n =(φ 1,n ,θ 1,n ,V 1,n ),···,(φ 5,n ,θ 5,n ,V 5,n ) are multiplied, and the components (Q n ,L n ) is obtained.

[0073] That is, for all the points of index n, the positions of the points Q1, . . . , Q N and the observation light information L1, ,L N and are multiplexed to obtain a point group K with ray information.

[0074] As described above, according to the subject modeling device 1 of the embodiment of the present invention, the point selection unit 11 selects the point position Q of the index n from the point group. n The parameter selection unit 13 selects the camera parameter p of the index m from the camera parameter group. m The image selection unit 14 selects an image I m Select .

[0075] The direction vector calculation unit 15 calculates the position of the component point Q n From camera position W m Direction vector Λ leading to m,n The projection unit 16 calculates the camera parameters p m Using the point position Q n Let ξ be the image coordinate. m,n and the pixel value selection unit 17 projects the image I m From image coordinate ξ m,n Pixel value V m,n Select .

[0076] Thus, for index n and all indices m, the direction vector Λ 1,n ,···,Λ M,n and pixel value V 1,n ,···,V M,n is obtained.

[0077] The first multiplexing unit 18 multiplexes the direction vector Λ of all indexes m for index n. 1,n ,···,Λ M,n and pixel value V 1,n ,···,V M,n The observation light information L n Request.

[0078] As a result, for all indices n=1,...,N, the component point positions Q1,...,Q N and the observation light information L1, ,L N is obtained.

[0079] The second multiplexing unit 19 multiplexes the component point positions Q1, . . . , Q N and the observation light information L1, ,L N and obtain a point group K with ray information.

[0080] In this way, multiple observation ray information L n Therefore, it is possible to provide rendering that changes the appearance (e.g., color) when viewed from a different viewpoint. In other words, a rendering device 2 shown in Fig. 5 described later can generate an image when viewed from a different viewpoint by using a point group K with ray information generated by the subject modeling device 1.

[0081] When an image with a different viewpoint is generated by the rendering device 2, the position Q of each of the constituent points reflecting the shape of the subject 3 is n and a direction vector Λ obtained with respect to the configuration point based on the viewpoints of the multiple cameras 4. m,n and pixel value V m,n Observation light information L n A point group K with ray information is used, which consists of the above.

[0082] In the method of the above-mentioned non-patent document 1, the rendering result has spatial and temporal discontinuity of pixel values ​​due to switching of texture images. In contrast, in the embodiment of the present invention, for rendering, a directional vector Λ based on multiple viewpoints is used. m,n and pixel value V m,n Therefore, the rendering result does not have spatial and temporal discontinuities in pixel values.

[0083] Therefore, it is possible to avoid the spatial and temporal discontinuities that are a problem when generating an image by switching between multiple light rays, resulting in viewpoint-dependent rendering with fewer artifacts and rendering results that exhibit a natural texture.

[0084] In addition, the observation light information L n For each point, the position Q of the point is calculated based on the number of viewpoints of the multiple cameras 4. n , directional vector Λ m,n and pixel value V m,n Therefore, the amount of data can be reduced.

[0085] Below, we compare the amount of data generated by the conventional technology using the above-mentioned light field description method with the amount of data generated by the embodiment of the present invention. In this conventional technology, we assume that pixel values ​​are recorded in a total of 4-5 dimensional space consisting of position (2-3 degrees of freedom) and light direction (2 degrees of freedom), that is, that the 4-5 dimensional space is divided into small pieces and pixel values ​​are recorded at each lattice point.

[0086] In the prior art, for example, if color information is recorded for 1,000,000 points in space at intervals of 360 in the azimuth direction and 180 in the elevation direction, pixel values ​​(e.g., 24-bit values) must be recorded for 1,000,000 x 360 x 180 = 64,800,000,000 addresses. In this case, the memory capacity required is 1.5552 x 10 12 It is a bit.

[0087] In contrast, in an embodiment of the present invention, in the case of a point cloud consisting of 1,000,000 points in space, this value is multiplied by the number of cameras (e.g., the number of rays M=100), resulting in 1,000,000×100=100,000,000 addresses, for which a 40-bit value consisting of the direction (360×180=64,800≒65,536, so a 16-bit value) and pixel value (e.g., a 24-bit value) must be recorded. In this case, the memory capacity required is 4.0×10 9 It is a bit.

[0088] Thus, the amount of data in the conventional technology is 1.5552 × 10 12 bits, whereas the data amount according to the embodiment of the present invention is 4.0×10 9 Therefore, in the embodiment of the present invention, when the number of cameras is not extremely large (for example, about 100 cameras), the data reduction effect is remarkable. In this example, the data amount is reduced to 1 / 388.8.

[0089] [Rendering Device] Next, a rendering device according to an embodiment of the present invention will be described below with reference to Fig. 5, which is a block diagram showing an example of the configuration of a rendering device according to an embodiment of the present invention.

[0090] 1 and camera parameters p of a virtual camera 5. Based on the point group K with light ray information and the camera parameters p, the rendering device 2 generates a computer graphics (CG) image observed by the virtual camera 5 indicated by the camera parameters p, and outputs the CG image as a composite image J.

[0091] As shown in FIG. 5, the rendering device 2 includes a first counter 20, a component point selection unit 21, a direction vector calculation unit (direction calculation unit) 22, a projection unit 23, a second counter 24, a ray selection unit 25, a weighting value calculation unit 26, a pixel value synthesis unit 27, and an image generation unit 28.

[0092] The first counter 20 counts from 1 to N in order to generate an index n for sequentially selecting each point of the point group K with ray information, and outputs the count value as index n to the point selection unit 21 and the image generation unit 28.

[0093] The point selection unit 21 selects a group of points with ray information K={(Q1, L1), . . . , (Q N ,L N)} and also inputs an index n from the first counter 20. The point selection unit 21 then selects a group of points with ray information K={(Q1, L1), . . . , (Q N ,L N )}, the component of the nth component point corresponding to index n (Q n ,L n The constituent point selection unit 21 selects the constituent point position Q n to the direction vector calculation unit 22 and the projection unit 23, and also outputs the observation light ray information L n is output to the light ray selection unit 25.

[0094] The direction vector calculation unit 22 receives the component point position Q from the component point selection unit 21. n is input, and the camera parameters p of the virtual camera 5 are also input. The camera parameters p include the camera position W.

[0095] The direction vector calculation unit 22 calculates the position of the component point Q by the following formula: n A unit direction vector (hereinafter referred to as a "direction vector") Γ is virtual direction information indicating the direction of a ray from n Calculate the direction vector Γ n to the weight value calculation unit 26.

number

[0096] The projection unit 23 receives the composition point position Q from the composition point selection unit 21. n The projection unit 23 inputs the camera parameter p of the virtual camera 5 and the position of the configuration point Q n image coordinate q n and project it onto image coordinate q n is output to the image generating unit 28.

number

[0097] That is, the projection unit 23 projects the n-th component point on the world coordinate system onto the image coordinate system of the image (synthetic image J) captured by the virtual camera 5 using the camera parameter p, thereby obtaining the component point position Q n image coordinate q n Convert to.

[0098] The second counter 24 receives the observation light information L n from the direction vector Λ m,n and pixel value V m,n are selected in sequence, and the pixel value V m,n and weight value β m,n In order to generate an index m for sequentially associating the first and second pixels with each other, the second counter 24 counts from 1 to M. The second counter 24 then outputs the count value to the light ray selection unit 25 and the pixel value synthesis unit 27 as the index m.

[0099] The ray selection unit 25 receives the observation ray information L n The light ray selection unit 25 inputs the observation light ray information L n The m-th component corresponding to index m (observation light information L shown in the above formula (4)) n In this case, the components of the column vector that constitutes the m-th column (Λ m,n ,V m,n The ray selection unit 25 selects a direction vector Λ m,n The unit vector of (hereinafter, for convenience of explanation, the direction vector Λ m,n ) is output to the weight value calculation unit 26, and the pixel value V m,n to the pixel value synthesis unit 27.

[0100] The light ray selection unit 25 selects the observation light ray information L n In the case of , the observation light information L n The m-th component corresponding to index m (the component of the column vector constituting the m-th column (φ m,n ,θ m,n ,V m,n )).

[0101] The light ray selection unit 25 calculates the azimuth angle φ m,n and the elevation angle θ m,n Let us define the direction vector Λ m,n , and the direction vector Λ m,n to the weighting value calculation unit 26, and the pixel value V m,n to the pixel value synthesis unit 27.

number

[0102] The weight value calculation unit 26 receives the direction vector Λ from the light ray selection unit 25. m,n is input, and the direction vector Γ is output from the direction vector calculation unit 22. n Then, the weight value calculation unit 26 inputs the direction vector Λ m,n and the direction vector Γ n Based on the direction vector Λ m,n and the direction vector Γ n The narrower the angle between the two, the larger the weight value β m,n The larger the angle, the smaller the weight value β m,n The weight value calculation unit 26 calculates, for example, a direction vector Λ m,n and the direction vector Γ n The weight value β is calculated by applying a monotonically decreasing function to the absolute value of the angle between m,n Request.

[0103] During rendering, the pixel value V observed by the camera in the direction as close as possible to the virtual viewpoint of the virtual camera 5 among the cameras used when the point group K with light ray information was generated by the object modeling device 1 shown in FIG. m,n Therefore, it is preferable to use the weight value β m,n As the direction vector Λ m,n and the direction vector Γ n The angle α m,n The narrower the gap, the more preferable it is to set a value that is not small.

[0104] Therefore, the weight value calculation unit 26 calculates a direction vector Λ, which is a unit direction vector, as shown in the following formula, for example.m,n and the direction vector Γ n (i.e., cosα m,n ) function f, the weight value β m,n The following may be calculated.

number

[0105] For example, the function f shown in the following formula may be used.

number

number

[0106] In the formulas (11) and (12), the larger the constant s, the more uniform the direction (angle α m,n (narrow) direction vector Λ m,n and the direction vector Γ n In other words, the combination of the direction vector Λ m,n The pixel value V m,n is reflected in the composite image J. The constant s is preferably, for example, equal to or greater than 1 and equal to or less than 100.

[0107] In addition, the weight value calculation unit 26 uses the observation light information L n A variable s that changes according to the number of light rays M may be used. n The more the number of rays M, the more uniform the direction vector Λ m,n and the direction vector Γ n Focusing on just the combination of these two tends to result in a clearer final result.

[0108] Therefore, the weight value calculation unit 26 sets the variable s to a larger value as the number of rays M increases, and sets the variable s to a smaller value as the number of rays M decreases (or sets the variable s to a value equal to the number of rays M).The weight value calculation unit 26 then calculates the weight value β in the equation (10) using the function f in the equation (11) or (12). m,n Calculate the following.

[0109] In addition, the weight value calculation unit 26 calculates the direction vector Γ using the following formula: n For the direction vector Λ 1,n ,···,Λ M,n The weight value β m,n The following may be calculated.

number

[0110] The pixel value synthesis unit 27 receives the index m from the second counter 24 and the pixel value V m,n The weight value calculation unit 26 outputs the weight value β m,n Enter the following respectively.

[0111] The pixel value synthesis unit 27 synthesizes the pixel value V m,n and weight value β m,n While matching all pixel values ​​V 1,n ,···,V M,n For all weight values ​​β 1,n , ,β M,n Then, the pixel value synthesis unit 27 synthesizes the weighting results for all indexes m to obtain a representative value U n Then, the pixel value synthesis unit 27 calculates the representative value U n is output to the image generating unit 28.

[0112] For example, the pixel value synthesis unit 27 calculates the weighted average value as a representative value U n It is calculated as follows.

number

[0113] In addition, the pixel value synthesis unit 27 uses the weight value β m,n Depending on the pixel value V m,n (e.g., the weight value β m,n The larger the pixel value V m,n (increase the number of samples) and take the median of the resampled pixel value sample sequence as the representative value U n It may be calculated as follows.

[0114] The image generating unit 28 receives the index n from the first counter 20 and the image coordinate q from the projection unit 23. n The pixel value synthesis unit 27 outputs the representative value U n Then, the image generating unit 28 calculates the representative value U n image coordinate q n The operation of placing the images in the matrix is ​​repeated to generate and output a composite image J.

number

[0115] In addition, the image generating unit 28 calculates the image coordinate q for the index n. n is outside the image of the composite image J, or the projection unit 23 calculates the image coordinate q n If the position of the point Q n is behind the virtual camera 5), ​​the representative value U n image coordinate q n Alternatively, the process of placing the image in the area 104 may be skipped.

[0116] In addition, the image generator 28 calculates the image coordinate q n If the component of is a non-integer value, the value is rounded to an integer, and then the representative value U of index n is calculated using the above formula (15). n image coordinate q n Alternatively, an operation of placing the image in the image area may be performed.

[0117] The image generator 28 also calculates a representative value U n Let q be the image coordinate. n The interpolation process may be spread to surrounding pixels of the pixel. Furthermore, the image generating unit 28 may generate (oversample) a provisional composite image with a resolution higher than the resolution of the composite image J to be finally output, and may generate the composite image J to be finally output by interpolating and reducing this provisional composite image.

[0118] (process) Next, a detailed description will be given of the flow of processing by the rendering device 2 shown in Fig. 5. Fig. 6 is a flowchart showing an example of processing by the rendering device 2, and Fig. 7 is a conceptual diagram explaining the example of processing by the rendering device 2.

[0119] In FIG. 7, the position Q of the nth point in the point group reflecting the shape of the subject 3 is n Regarding the representative value U n is the image coordinate q in the composite image J captured by the virtual camera 5. n The equipment shall be located at

[0120] Camera parameters p1, p2 of cameras 4-1, 4-m, and 4-M m ,p M and camera positions W1,W m ,W M , and images I1,I m ,I M , image coordinate ξ 1,n ,ξ m,n ,ξ M,n , pixel value V 1,n ,V m,n ,V M,n and the direction vector Λ 1,n ,Λ m,n ,Λ M,n is the same as in Figure 3.

[0121] First, the first counter 20 generates an index n=1 with a count value of 1 (step S601, n←1). Then, the composition point selection unit 21 selects the composition point position Q of the n-th composition point corresponding to the index n from the point group K with ray information.n and observation light information L n (Step S602).

[0122] As shown in FIG. 7, the position Q of the nth point in the point group reflecting the shape of the subject 3 is n is selected, and the observation light information L n A direction vector Λ 1,n ,···,Λ M,n and pixel value V 1,n ,···,V M,n is selected.

[0123] The direction vector calculation unit 22 calculates the position of the component point Q n Direction vector Γ from to camera position W n (step S603), and the projection unit 23 calculates the configuration point position Q n image coordinate q n (step S604).

[0124] As shown in FIG. 7, the position of the composition point Q n directional vector Γ from the virtual camera 5 to the camera position W n is obtained, and the image coordinate q in the composite image J is n is obtained.

[0125] The second counter 24 generates an index m with a count value of 1 (step S605, m←1). Then, the light ray selection unit 25 selects the observation light ray information L n from the direction vector Λ of index m m,n and pixel value V m,n (step S606), and the weight value calculation unit 26 calculates the direction vector Λ m,n and the direction vector Γ n From the weight value β m,n is calculated (step S607).

[0126] As shown in Fig. 7, when index m=1, the direction vector Λ 1,n and pixel value V 1,nis selected, and when index m=M, the observation ray information L M from the direction vector Λ M,n and pixel value V M,n In other words, for index m, the observation ray information L m from the direction vector Λ m,n and pixel value V m,n is selected.

[0127] The second counter 24 determines whether or not the index m is less than the number of rays M (step S608). If the second counter 24 determines in step S608 that the index m is less than the number of rays M (step S608: Y), it adds 1 to the index m (step S609, m←m+1) and proceeds to step S606. As a result, the processes of steps S606 and S607 are performed for the index m.

[0128] On the other hand, if the second counter 24 determines in step S608 that the index m is not less than the number of rays M (step S608: N), the process proceeds to step S610.

[0129] In this case, for index n (initially n=1) and all indexes m=1, . . . , M, the pixel value V 1,n ,···,V M,n and weight value β 1,n , ,β M,n is obtained.

[0130] The pixel value synthesis unit 27 proceeds from step S608(N) to synthesize pixel values ​​V 1,n ,···,V M,n For each, the corresponding weight value β 1,n , ,β M,n The representative value U n is calculated (step S610).

[0131] The first counter 20 determines whether or not the index n is less than the number of constituent points N (step S611). If the first counter 20 determines in step S611 that the index n is less than the number of constituent points N (step S611: Y), it adds 1 to the index n (step S612, n←n+1) and proceeds to step S602. As a result, the processes of steps S602 to S610 are performed for the index n.

[0132] On the other hand, when the first counter 20 determines in step S611 that the index n is not less than the number of component points N (step S611: N), the process proceeds to step S613. In this case, for all indexes n=1, . . . , N, the image coordinates q1, . . . , q N and representative values ​​U1, ,U N is obtained.

[0133] The image generating unit 28 proceeds from step S611(N) to calculate the representative values ​​U1, . . . , U N The corresponding image coordinates q1, ,q N and generate a composite image J (step S613).

[0134] As described above, according to the rendering device 2 of the embodiment of the present invention, the configuration point selection unit 21 selects the configuration point position Q of the index n from the point group K with light ray information generated by the object modeling device 1 shown in FIG. n and observation light information L n In addition, the direction vector calculation unit 22 selects the component point position Q n directional vector Γ from the virtual camera 5 to the camera position W n Calculate the following.

[0135] The projection unit 23 calculates the position of the configuration point Q using the camera parameter p. n image coordinate q n This allows us to project the image coordinates q1, ,q N is obtained.

[0136] The light ray selection unit 25 selects the observation light ray information L n to the direction vector Λ of index m m,n and pixel value V m,n and the weight value calculation unit 26 selects the direction vector Λ m,n and the direction vector Γ n Based on the weight value β m,n Calculate the following.

[0137] This allows the pixel value V 1,n ,···,V M,n and weight value β 1,n , ,β M,n is obtained.

[0138] The pixel value synthesis unit 27 synthesizes pixel values ​​V 1,n ,···,V M,n For the weight value β 1,n , ,β M,n The representative value U n This allows us to calculate the representative values ​​U1, ,U N is obtained.

[0139] The image generator 28 calculates representative values ​​U1, . . . , U N Let q1, ,q be the image coordinates. N and generate a composite image J.

[0140] As a result, a direction vector Γ calculated from the relative positions of the virtual viewpoint, which is the camera position W of the virtual camera 5, and each constituent point of the point cloud is n and the direction vector Λ of the ray associated with each component point m,n In other words, a composite image J viewed from different viewpoints can be generated by using the point group K with light ray information generated by the subject modeling device 1.

[0141] When the composite image J is generated, the position Q of each component point reflecting the shape of the subject 3 is determined. n and a direction vector Λ obtained with respect to the configuration point based on the viewpoints of the multiple cameras 4. m,n and pixel value V m,n Observation light information L n A point group K with ray information is used, which consists of the above.

[0142] In the method of the above-mentioned non-patent document 1, the rendering result has spatial and temporal discontinuity of pixel values ​​due to switching of texture images. In contrast, in the embodiment of the present invention, for rendering, a directional vector Λ based on multiple viewpoints is used. m,n and pixel value V m,n Therefore, the rendering result does not have spatial and temporal discontinuities in pixel values.

[0143] Therefore, it is possible to avoid the spatial and temporal discontinuities that are a problem when generating an image by switching between multiple light rays, resulting in viewpoint-dependent rendering with fewer artifacts and rendering results that exhibit a natural texture.

[0144] The rendering device 2 may use a Z-buffer method to generate the composite image J. The Z-buffer method is a method of projecting each of the constituent points of a point cloud onto an image plane in sequence and superimposing the images to construct an image, in which a Z-buffer that records the distance (or depth value (Z value)) from the camera position to each constituent point is also used.

[0145] When overlaying the image of a constituent point on an image, the Z value already recorded at the pixel position of the image is compared with the Z value of the constituent point to be overlaid, and if the Z value of the constituent point to be overlaid is smaller or if the two Z values ​​are equal, the pixel value of the constituent point to be overlaid is overwritten.

[0146] Specifically, in the rendering device 2 shown in FIG. 5, the projection unit 23 projects an image coordinate q nIn addition, the following equation is used to calculate the position of the camera point Q from the camera position W included in the camera parameter p. n Distance d between n Then, the projection unit 23 calculates the image coordinate q n and distance d n is output to the image generating unit 28.

number

[0147] The image generating unit 28 receives the index n from the first counter 20 and the image coordinate q from the projection unit 23. n and distance d n The pixel value synthesis unit 27 outputs the representative value U n The image generator 28 inputs the image coordinates q1, . . . , q for all indexes n. N and distances d1,...,d N and representative values ​​U1, ,U N While associating these, the composite image J is calculated using the following formula.

number

[0148] First, the image generating unit 28 calculates the initial value Z of the Z buffer as shown in the above formula (17-1). (0) Set all to infinity and set the initial value of the temporary composite image J (0) Set to any value (e.g. 0).

[0149] The image generating unit 28 calculates the image coordinate q n Z value in Z (n-1) (qn ) and distance d n Then, the image generating unit 28 compares the distance d n If is smaller, the representative value U n The image coordinate q n The pixel value J of the provisional synthetic image in (n) (q n On the other hand, the image generating unit 28 sets the distance d n is not smaller than the n-1th pixel value J (n-1) (q n ) is the pixel value J (n) (q n ) to maintain (copy).

[0150] The image generating unit 28 calculates the image coordinate q n Z value in Z (n-1) (q n ) and distance d n Then, the image generating unit 28 compares the distance d n If is smaller, the distance d n The image coordinate q n Z value in Z (n) (q n On the other hand, the image generating unit 28 sets the distance d n is not smaller than the n-1th Z value Z (n-1) (q n ) as Z value Z (n) (q n ) to maintain (copy).

[0151] The image generating unit 28 calculates the image coordinate q n For image coordinates x other than (n) (x) and Z value Z (n) In both cases, (x) maintains (copies) each n-1th value as is.

[0152] Finally, the image generating unit 28 applies the formulas (17-2) to (17-4) to the (N) constituent points of all indexes n=1, . . . , N as shown in the formula (17-5) to generate a provisional composite image J (N)(x) is calculated and this is designated as the composite image J.

[0153] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0154] For example, the pixel value selection unit 17 of the object modeling device 1 shown in FIG. m From image coordinate ξ m,n Pixel value V m,n and the first multiplexing unit 18 selects the direction vector Λ 1,n ,···,Λ M,n and pixel value V 1,n ,···,V M,n The observation light information L n In this case, the pixel value V m,n Here is the image I m The pixel value V m,n The value may be color information such as RGB values, infrared intensity, or a distance value measured by a distance sensor. The same applies to the rendering device 2 shown in FIG.

[0155] A normal computer can be used as the hardware configuration of the object modeling device 1 and the rendering device 2 according to the embodiment of the present invention. The object modeling device 1 and the rendering device 2 are configured by a computer including a CPU, a volatile storage medium such as a RAM, a non-volatile storage medium such as a ROM, an interface, etc.

[0156] Each function of first counter 10, composition point selection unit 11, second counter 12, parameter selection unit 13, image selection unit 14, direction vector calculation unit 15, projection unit 16, pixel value selection unit 17, first multiplexing unit 18 and second multiplexing unit 19 provided in subject modeling device 1 is realized by causing a CPU to execute a program describing these functions. These programs are stored in the storage medium, and are read out and executed by the CPU.

[0157] Furthermore, each function of the first counter 20, the composition point selection unit 21, the direction vector calculation unit 22, the projection unit 23, the second counter 24, the ray selection unit 25, the weight value calculation unit 26, the pixel value synthesis unit 27, and the image generation unit 28 provided in the rendering device 2 is also realized by having the CPU execute a program describing these functions. These programs are stored in the storage medium, and are read out and executed by the CPU.

[0158] In addition, these programs can be distributed by storing them on storage media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memories, and can also be transmitted and received via a network. [Explanation of symbols]

[0159] 1. Subject modeling device 2. Rendering Device 3. Subject 4. Camera 5. Virtual Camera 10,20 First counter 11,21 Point selection section 12,24 Second counter 13 Parameter selection section 14 Image selection section 15,22 Direction vector calculation unit (direction calculation unit) 16,23 Projection section 17 Pixel value selection section 18 First multiplexing section 19 Second multiplexing section 25 Ray Selection Section 26 Weighting value calculation unit 27 Pixel value synthesis unit 28 Image Generation Unit 101 constituent points n,m index N Number of components M Number of rays (number of images, number of cameras) Q n Composition point position p m ,p camera parameters W m ,W Camera position I m image J Composite Image Λ m,n ,Γ n Direction Vector ξ m,n ,q n Image Coordinates V m,n Pixel Value U n Representative value L n Observation beam information Point cloud with K-ray information β m,n Weight Value

Claims

1. 1. An object modeling device that inputs a point cloud consisting of positional information of a plurality of constituent points representing a shape of an object, a camera parameter group consisting of camera parameters of a plurality of cameras, and an image group consisting of images of the object captured by each of the plurality of cameras, and models the point cloud to generate a point cloud with light ray information, Let n be an index for sequentially selecting the constituent points constituting the point cloud, and let m be an index for sequentially selecting the camera parameters constituting the camera parameter group and the images constituting the image group, a direction calculation unit that calculates direction information regarding a direction of a ray from the configuration point to a camera based on a camera position included in the camera parameters of the index m and the position information of the configuration point of the index n; a projection unit that projects the position information of the component point of the index n onto image coordinates of the image of the index m using the camera parameters of the index m; a selection unit that selects pixel value information of the image coordinates projected by the projection unit from the image of the index m; a first multiplexing unit that generates observation ray information for the index n by multiplexing, for the index n, the direction information for all of the indexes m calculated by the direction calculation unit and the pixel value information for all of the indexes m selected by the selection unit; a second multiplexing unit that generates the point group with light ray information by multiplexing the position information of all of the indexes n and the observation light ray information of all of the indexes n generated by the first multiplexing unit; An object modeling device comprising:

2. 2. The object modeling apparatus according to claim 1, The selection unit is Selecting color information for the image coordinates from the image; The first multiplexing unit the direction information of all of the indexes m calculated by the direction calculation unit and the color information of all of the indexes m selected by the selection unit are multiplexed for the index n to generate the observation ray information of the index n.

3. 1. A rendering device for rendering a point cloud with light ray information, in which position information and light ray information of a plurality of constituent points representing a shape of a subject are multiplexed, using a camera parameter p of a virtual camera, and generating a composite image captured by the virtual camera, comprising: The light ray information is information generated based on camera parameters of a plurality of cameras and images of the subject photographed by each of the plurality of cameras, and includes directional information regarding the direction of a light ray from the composition point to the camera, and pixel value information regarding a pixel onto which the composition point is projected, Let n be an index for sequentially selecting the constituent points constituting the point group with ray information, and m be an index for sequentially selecting the direction information and the pixel value information included in the ray information, a projection unit that projects the position information of the component point of the index n onto image coordinates using the camera parameter p; a direction calculation unit that calculates virtual direction information regarding a direction of a ray from the configuration point to the virtual camera based on a camera position of the virtual camera and the position information of the configuration point of the index n included in the camera parameter p; a weight value calculation unit that calculates a weight value for the index n based on the direction information of the index m included in the light ray information and the virtual direction information calculated by the direction calculation unit; a synthesis unit that performs weighting synthesis on the pixel value information of all the indexes m included in the light ray information for the index n using the weight values ​​of all the indexes m calculated by the weight value calculation unit to obtain a representative value; an image generating unit that places the representative values ​​calculated by the synthesis unit for all the indexes n at the image coordinates projected by the projection unit and generates the synthetic image; A rendering device comprising:

4. 4. The rendering device according to claim 3, The light ray information includes the direction information and color information related to a pixel onto which the configuration point is projected, Let m be an index for sequentially selecting the direction information and the color information included in the light ray information, The synthesis unit is a weighting synthesis step of weighting the color information of all of the indexes m included in the light ray information for the index n using the weighting values ​​of all of the indexes m to obtain the representative value,

5. 5. A rendering device according to claim 3, further comprising: The weight value calculation unit A rendering device characterized in that, for the index n, the weight value is calculated by applying a broad monotonically decreasing function to the absolute value of the angle between the direction information of the index m contained in the ray information and the virtual direction information.

6. a computer constituting an object modeling device which receives a point cloud consisting of position information of a plurality of constituent points representing the shape of an object, a camera parameter group consisting of camera parameters of a plurality of cameras, and an image group consisting of images of the object captured by each of the plurality of cameras, and models the point cloud to generate a point cloud with light ray information; Let n be an index for sequentially selecting the constituent points constituting the point cloud, and let m be an index for sequentially selecting the camera parameters constituting the camera parameter group and the images constituting the image group, a direction calculation unit that calculates direction information regarding a direction of a ray from the component point to a camera based on a camera position included in the camera parameters of the index m and the position information of the component point of the index n; a projection unit that projects the position information of the component point of the index n onto image coordinates of the image of the index m using the camera parameters of the index m; a selection unit that selects pixel value information of the image coordinates projected by the projection unit from the image of the index m; a first multiplexing unit that generates observation ray information for the index n by multiplexing the direction information for all of the indexes m calculated by the direction calculation unit and the pixel value information for all of the indexes m selected by the selection unit for the index n; and A program for functioning as a second multiplexing unit that generates a point cloud with light ray information by multiplexing the position information of all of the indexes n and the observation ray information of all of the indexes n generated by the first multiplexing unit.

7. a computer constituting a rendering device that renders a point cloud with light ray information, in which position information and light ray information of a plurality of constituent points representing a shape of a subject are multiplexed, using the point cloud with light ray information and a camera parameter p of a virtual camera, and generates a composite image captured by the virtual camera; The light ray information is information generated based on camera parameters of a plurality of cameras and images of the subject photographed by each of the plurality of cameras, and includes directional information regarding the direction of a light ray from the composition point to the camera, and pixel value information regarding a pixel onto which the composition point is projected, Let n be an index for sequentially selecting the constituent points constituting the point group with ray information, and m be an index for sequentially selecting the direction information and the pixel value information included in the ray information, a projection unit that projects the position information of the component point of the index n onto image coordinates using the camera parameter p; a direction calculation unit that calculates virtual direction information regarding a direction of a ray from the constituent point to the virtual camera based on a camera position of the virtual camera included in the camera parameter p and the position information of the constituent point of the index n; a weight value calculation unit that calculates a weight value for the index n based on the direction information of the index m included in the light ray information and the virtual direction information calculated by the direction calculation unit; a synthesis unit that performs weighting synthesis on the pixel value information of all the indexes m included in the light ray information for the index n using the weight values ​​of all the indexes m calculated by the weight value calculation unit to obtain a representative value; and A program for causing an image generating unit to function as an image generating unit that generates the composite image by arranging the representative value obtained by the synthesis unit for all of the indexes n at the image coordinates projected by the projection unit.

Images