Information multiplexing device, its program, and three-dimensional model generation system

Abstract

To provide an information multiplexing device capable of multiplexing sensor information onto three-dimensional shape data of an object.SOLUTION: An information multiplexing device 4 includes: vertex selection means 40 for selecting a vertex of three-dimensional shape data; pixel value selection means 41 for selecting a pixel value of corresponding image coordinates from a sensor image, in which pixel values are measurement values of an object acquired from multiple sensors, using projection transformation information to project the measurement values onto the sensor image when the selected vertex is closest to a viewpoint position; weight calculation means 42 for calculating, using the projection transformation information, a weight that increases as the resolution of the sensor image becomes higher; synthesis means 43 for synthesizing the pixel value selected by the pixel value selection means 41 based on the weight calculated by the weight calculation means 42; and modeling means 44 for associating the synthesized pixel value with the selected vertex.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information multiplexing device and a program for multiplexing sensor information onto three-dimensional shape data, and a three-dimensional model generation system. [Background technology]

[0002] Conventionally, there is a thermography technique that outputs the temperature distribution of each local region in a three-dimensional object as a two-dimensional image. There is also photogrammetry technology, which estimates the three-dimensional shape of an object from camera images taken with multiple cameras. There is also a volumetric capture technology that estimates an object as three-dimensional time-series data directly from camera images taken with multiple cameras. Patent Document 1 discloses a head-mounted temperature distribution recognition device that includes a thermosensor that detects infrared rays and a spatial recognition sensor for recognizing three-dimensional shapes, and generates image data based on the measurement data of these sensors and displays it on a display. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-092521 Summary of the Invention [Problem to be solved by the invention]

[0004] Thermography technology visualizes and outputs the temperature distribution of an object as an image, and is not linked to the three-dimensional shape of the object. Furthermore, while photogrammetry and volumetric capture technologies can correlate and acquire the three-dimensional shape and color pattern of an object, they cannot link non-visible light information such as temperature. In addition, the invention described in Patent Document 1 associates the three-dimensional shape of an object with the temperature distribution and displays it on a display. However, the temperature distribution corresponds to the two-dimensional image displayed on the display. As described above, the conventional technology does not associate the three-dimensional shape itself of the object with information that can be acquired by a sensor such as temperature. On the other hand, when generating a three-dimensional model of an object, there is a desire to associate information that can be acquired by a sensor with the three-dimensional shape in addition to the three-dimensional shape data.

[0005] Therefore, an object of the present invention is to provide an information multiplexing device and its program capable of multiplexing sensor information with three-dimensional shape data of an object, and a three-dimensional model generation system.

Means for Solving the Problems

[0006] In order to solve the above problems, an information multiplexing device according to the present invention is an information multiplexing device that multiplexes the measured values with the three-dimensional shape data from the three-dimensional shape data of an object, a sensor image having the two-dimensional measured values of the object acquired from a plurality of sensors as pixel values, and projection conversion information for projecting the measured values of the object onto the sensor image, and includes vertex selection means, pixel value selection means, weight calculation means, composition means, and modeling means.

[0007] In such a configuration, the information multiplexing device sequentially selects vertices of the three-dimensional shape data by the vertex selection means. Then, the information multiplexing device calculates, for each sensor, the image coordinates of the sensor image corresponding to the selected vertex using the projection conversion information by the pixel value selection means, and when the selected vertex is the vertex of the three-dimensional shape data closest to the viewpoint position of the projection conversion among the vertices corresponding to the calculated image coordinates, selects the pixel value of the image coordinates in association with the selected vertex. Thereby, it is possible to prevent associating incorrect pixel values with vertices existing on the back side even at the same image coordinates.

[0008] Further, the information multiplexing device calculates, for each sensor, a weight that is larger as the resolution of the sensor image is higher, using the projection conversion information, by means of the weight calculation means. Then, the information multiplexing device synthesizes, for each selected vertex, the pixel values for each sensor selected by the pixel value selection means, based on the weights calculated by the weight calculation means, by means of the synthesis means. Then, the information multiplexing device associates the pixel values synthesized by the synthesis means with the selected vertices, by means of the modeling means. As a result, the information acquired by the sensor is associated with each vertex of the three-dimensional shape data.

[0009] Also, when a normal is associated with a vertex of the three-dimensional shape data, the weight calculation means may calculate a weight that is larger as the resolution of the sensor image is higher and as the angle formed by the normal and the vector from the vertex to the viewpoint position of the projection conversion is smaller, using the projection conversion information. Note that the information multiplexing device can be operated by a program for causing a computer to function as the information multiplexing device.

[0010] Further, in order to solve the above problems, a three-dimensional model generation system according to the present invention is a three-dimensional model generation system that generates a three-dimensional model in which the measurement values of sensors are multiplexed on three-dimensional shape data of an object, and includes a plurality of cameras, a plurality of sensors, a three-dimensional shape estimation device, and an information multiplexing device.

[0011] In such a configuration, the three-dimensional model generation system photographs an object with a plurality of cameras. Also, the three-dimensional model generation system acquires a sensor image having two-dimensional measurement values of the object as pixel values, with a plurality of sensors. Also, the three-dimensional model generation system estimates three-dimensional shape data of the object from the camera images photographed by the plurality of cameras, by means of the three-dimensional shape estimation device. Then, the three-dimensional model generation system multiplexes the measured values onto the three-dimensional shape data by the information multiplexing device, using the three-dimensional shape data estimated by the three-dimensional shape estimation device, a plurality of sensor images acquired from a plurality of sensors, and projection conversion information for projecting the measured values of the object onto the sensor images. As a result, the three-dimensional model generation system can multiplex the information acquired from the sensors at the vertices of the three-dimensional shape data of the object.

Advantages of the Invention

[0012] According to the present invention, sensor information can be multiplexed at each vertex of the three-dimensional shape data of the object.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. ≪Three-Dimensional Model Generation System≫ With reference to FIG. 1, the configuration of a three-dimensional model generation system 100 according to an embodiment of the present invention will be described.

[0015] The three-dimensional model generation system 100 generates a three-dimensional model in which measurement values of sensors are multiplexed with three-dimensional shape data of an object (including a living thing) H. The object H is a three-dimensional object that is a target for generating a three-dimensional model, such as a person. The three-dimensional model generation system 100 includes a plurality of cameras 1, a plurality of sensors 2, a three-dimensional shape estimation device 3, and an information multiplexing device 4. Here, although FIG. 1 shows six cameras (11,..., 16) and three sensors (21,..., 23), these numbers are arbitrary.

[0016] The camera 1 photographs the object H. The cameras 1 are arranged so as to surround the object H and photograph the object H with at least two or more cameras. For example, the cameras 1 are arranged at predetermined positions such as on a circumference or on a hemisphere centered on the object H in a volumetric studio. The camera 1 outputs a camera image obtained by photographing the object H to the three-dimensional shape estimation device 3. For example, as shown in FIG. 2, the camera 1 outputs a camera image G1 obtained by photographing the object H as a two-dimensional image to the three-dimensional shape estimation device 3.

[0017] The sensor 2 measures a two-dimensional physical quantity of the object H and outputs a sensor image having the measurement value as a pixel value. The sensor 2 visualizes a predetermined physical quantity (sensor information) of the object H as a two-dimensional image (sensor image). At least two or more sensors 2 are arranged at positions where the physical quantity of the object H can be measured. The arrangement position of the sensor 2 may be any position as long as it can measure the physical quantity of the object H. For example, in FIG. 1, the sensors 2 are arranged so as to surround the object H at an equal distance, similar to the cameras 1, but may be arranged closer to or farther from the object H than the cameras 1.

[0018] As the sensor 2, a thermographic camera, a color camera, a camera that captures non-visible light (ultraviolet rays, infrared rays, X-rays, millimeter waves, muons, etc.), or a general device that irradiates and detects light rays (ultrasonic waves) to visualize and output the texture information (subsurface scattering, diffuse reflection coefficient, specular reflection coefficient, normal direction, surface roughness, etc.) of the object H as an image of two-dimensional data can be used. This sensor 2 may be one corresponding to the information to be multiplexed as a three-dimensional model. The plurality of sensors 2 shall measure the same kind of physical quantity.

[0019] The sensor 2 outputs a sensor image, which is sensor information obtained as two-dimensional data, to the information multiplexing device 4. Here, the sensor image output by the sensor 21 is defined as the first sensor image. Also, the sensor image output by the sensor 22 is defined as the second sensor image. Further, the sensor image output by the sensor 23 is defined as the third sensor image. For example, as shown in FIG. 3, the sensor 2 outputs a sensor image G2, which visualizes the sensor information obtained from the object H as a two-dimensional image, to the information multiplexing device 4. Note that the sensor image G2 does not necessarily have to be data of one channel and may be data of multiple channels. For example, when a color camera is used as the sensor 2, it may be composed of two-dimensional data of three channels of RGB as the sensor image. In this way, by arranging a plurality of sensors 2, it is possible to reduce the noise of the measured values for the three-dimensional region measured by the plurality of sensors 2. Also, it is possible to reduce the regions (dead angles) where the measured values cannot be multiplexed.

[0020] The three-dimensional shape estimation device 3 estimates the three-dimensional shape data of the object H using a plurality of camera images captured by the plurality of cameras 1 for the object H. The three-dimensional shape data is data representing the shape of the object H as a numerical sequence. The three-dimensional shape data is, for example, data of a point cloud (point group) model that represents the surface or the interior of an object as a set of points (hereinafter referred to as vertices). In this case, as shown in FIG. 4, the three-dimensional shape data is a sequence of vector values representing the three-dimensional coordinates (x, y, z) of each vertex on the object surface.

[0021] Alternatively, the three-dimensional shape data may be data of a polygon mesh model. In this case, the three-dimensional shape data is composed of three or more vertices (vertices), line (edge) information that is connection information between vertices, and face (face) information that is connection information between lines. Furthermore, the three-dimensional shape data may be data of a voxel model. In this case, the three-dimensional shape data is obtained by dividing the three-dimensional space into a grid (for example, a grid on a rectangular parallelepiped), and is composed of information indicating whether or not it is inside the object for each element (voxel) of the grid.

[0022] Here, the three-dimensional shape data is assumed to be data of a point cloud model. Let the number of vertices in the point cloud model be N (N is a natural number), and the three-dimensional coordinates of the nth vertex (n is an integer from 1 to N) be V n When this is the case, the three-dimensional shape data S can be represented by the following formula (1).

[0023]

Equation

[0024] When the three-dimensional shape data is data of a polygon mesh model, only the vertex information needs to be focused on, and the set can be regarded as data of a point cloud model. When the three-dimensional shape data is data of a voxel model, the center coordinates of the voxels inside the object can be regarded as vertices, and the set can be regarded as data of a point cloud model. Note that, for the method in which the three-dimensional shape estimation device 3 estimates three-dimensional shape data from a plurality of camera images, for example, conventional general methods such as photogrammetry technology and volumetric capture technology may be used, and thus the description is omitted here. The three-dimensional shape estimation device 3 outputs the estimated three-dimensional shape data to the information multiplexing device 4.

[0025] The information multiplexing device 4 multiplexes the measurement values (sensor information) onto the three-dimensional shape data from the three-dimensional shape data estimated by the three-dimensional shape estimation device 3, the sensor image having the two-dimensional measurement values of the object H acquired from the plurality of sensors 2 as pixel values, and the projection conversion information for projecting the measurement values of the object H onto the sensor image.

[0026] Here, the projection conversion information corresponding to the first sensor image output by the sensor 21 is defined as the first projection conversion information. Also, the projection conversion information corresponding to the second sensor image output by the sensor 22 is defined as the second projection conversion information. Further, the projection conversion information corresponding to the third sensor image output by the sensor 23 is defined as the third projection conversion information.

[0027] The projection conversion information (first to third projection conversion information) is information for specifying the functions h(X) and g(X) shown in the following formula (2) for associating the three-dimensional coordinates X in the three-dimensional space where the object H exists with the two-dimensional coordinates x and the depth value d of the sensor images (first to third sensor images).

[0028]

Equation

[0029] Here, the functions h(X) and g(X) are each modeled by a predetermined projection conversion (for example, perspective projection), and their parameters are used as the projection conversion information. For example, the functions h(X) and g(X) are modeled as shown in the following formula (3), and their parameters are used as the projection conversion information.

[0030]

Equation

[0031] Here, T is the viewpoint position in the projective transformation, R is the rotation transformation matrix indicating the orientation of sensor 2, f is the focal length of the lens of the projective transformation considering sensor 2 as a camera, (r x , r y ) is the pixel size in the horizontal and vertical directions of sensor 2, and (c x , c y ) are the image coordinates of the point where sensor 2 intersects the optical axis of the lens. The projective transformation information may discretize the three-dimensional coordinates X at predetermined intervals and implement the functions h(X) and g(X) as look-up tables (LUTs), respectively. Note that the projective transformation information may be the functions h(X) and g(X) themselves. The detailed configuration and operation of this information multiplexing device 4 will be described later.

[0032] With the configuration described above, the three-dimensional model generation system 100 can estimate three-dimensional shape data from the camera images of the object H captured by the plurality of cameras 1, and generate a three-dimensional model in which the sensor images (sensor information) output from the plurality of sensors 2 are multiplexed into the three-dimensional shape data.

[0033] <Configuration of Information Multiplexing Device> With reference to FIG. 5 (and appropriately FIG. 1), the configuration of the information multiplexing device 4 according to the embodiment of the present invention will be described. The information multiplexing device 4 includes vertex selection means 40, pixel value selection means 41, weight calculation means 42, synthesis means 43, and modeling means 44.

[0034] The vertex selection means 40 sequentially selects the coordinates (vertex coordinates) of the vertices constituting the three-dimensional shape data. Here, the vertex selection means 40 sequentially selects the N vertices constituting the three-dimensional shape data S shown in Equation (1) and outputs them to the pixel value selection means 41, the weight calculation means 42, and the modeling means 44. Hereinafter, the vertex coordinates selected at the nth time when sequentially selecting the N vertex coordinates are denoted as V n .

[0035] The pixel value selection means 41 calculates the two-dimensional coordinates of the sensor image corresponding to the vertex selected by the vertex selection means 40 using the projection conversion information, and selects the pixel value of the two-dimensional coordinates of the sensor image when the selected vertex is the foremost among the vertices corresponding to the calculated two-dimensional coordinates. The pixel value selection means 41 outputs the selected pixel value to the composition means 43. However, when the selected vertex is not the foremost, the pixel value selection means 41 outputs an identifier indicating a predetermined NaN to the composition means 43. A plurality of pixel value selection means 41 are provided according to the number of sensors 2. Here, the number of sensors 2 is three, and three pixel value selection means 411, 412, and 413 are provided as a configuration. The pixel value selection means 411, 412, and 413 have the same function, differing only in the input sensor images (first to third sensor images) and projection conversion information (first to third projection conversion information).

[0036] (Configuration of Pixel Value Selection Means) Here, with reference to FIG. 6, the configuration of the pixel value selection means 41 will be described in more detail. The pixel value selection means 41 includes a depth map generation means 410, a projection means 411, a depth value reading means 412, a foremost determination means 413, and a pixel value reading means 414.

[0037] The depth map generation means 410 generates a depth map from the three-dimensional shape data using the projection conversion information. That is, the depth map generation means 410 generates a depth map indicating the depth when observing the three-dimensional shape data from the viewpoint, posture, and field angle specified by the projection conversion information. The depth map generation means 410 inputs the functions h(X) and g(X) shown in Equation (2) as the projection conversion information and generates a depth map D(x). Here, x is the two-dimensional coordinate of the sensor image. For example, the depth map generation means 410 generates a depth map D(x) according to the recurrence formula shown in the following Equation (4).

[0038]

Number

[0039] That is, the depth map generation means 410 sets, for all pixel positions x of the sensor image, a value indicating an infinite distance (+∞) predetermined for the initial value D0(x) of the depth map. Then, for each vertex coordinate V where n ∈ {1, 2, …, N} of the depth map generation means 410 n when the depth value g(V n ) obtained by the function g(X) corresponding to is smaller than the (n - 1) depth value D n corresponding to the vertex coordinate V n at the image coordinate h(V n ), the depth value D n-1 (h(V n )) is updated with g(V n ), and in other cases, the depth value D n (h(V n )) is set to D n (h(V n )). Note that the inequality sign "<" for determining the magnitude in Equation (4) may be an inequality sign "≦" including the equal sign. n-1 (h(V n )) is set to D

[0040] Then, the depth map generation means 410 generates the depth map D(x) by setting D N (x) to D(x) for all pixel positions x of the sensor image. As a result, the minimum value (the one closest to the viewpoint position) among the depth values projected onto a certain pixel position x is set in the depth map. The depth map generation means 410 outputs the generated depth map to the depth value reading means 412.

[0041] The projection means 411 calculates the depth value of the vertex (vertex coordinate) selected by the vertex selection means 40 and the image coordinate of the sensor image using the projection conversion information. That is, as shown in the following equation (5), the projection means 411 uses the functions h(X) and g(X) shown in equation (2) as projection conversion information to obtain the vertex coordinates V n corresponding depth value d n and image coordinates x n are calculated.

[0042]

Equation

[0043] The projection means 411 outputs the calculated depth value d n to the front determination means 413, and outputs the calculated image coordinates x n to the depth value reading means 412 and the pixel value reading means 414.

[0044] The depth value reading means 412 reads the depth value corresponding to the image coordinates calculated by the projection means 411 from the depth map generated by the depth map generation means 410. That is, the depth value reading means 412 reads the depth value D(x n corresponding to the image coordinates x from the depth map D(x) which is the output value of equation (4). n ) The depth value reading means 412 outputs the read depth value D(x n ) to the front determination means 413.

[0045] The front determination means 413 compares the depth value calculated by the projection means 411 with the depth value read from the depth map, and determines whether the vertex (vertex coordinates) selected by the vertex selection means 40 is the closest to the viewpoint position of the projection transformation. Here, the front determination means 413 uses the depth value d n corresponding to the vertex coordinates V calculated by the projection means 411, n and the depth value D(x n corresponding to the image coordinates x read by the depth value reading means 412, n ) nDetermines whether it is the foremost when viewed from the viewpoint position of the projective transformation in the sensor image. For example, as shown in the following formula (6), the foremost determination means 413 determines the vertex coordinates V n When it is the foremost when viewed from the viewpoint position of the sensor image, that is, when d n = D(x n ), the value "1" is used as the determination result c, and in other cases, the value "0" is used as the determination result c.

[0046]

Equation

[0047] The foremost determination means 413 outputs the determination result to the pixel value reading means 414. Note that the foremost determination means 413 may use the following formula (7) instead of the formula (6) assuming the occurrence of errors due to sampling of pixel positions.

[0048]

Equation

[0049] Here, ε is a predetermined depth tolerance. Note that the inequality sign "<" for performing the magnitude determination in formula (7) may be an inequality sign "≦" including the equal sign. Also, in formula (7), when d n ≦ 0, c = 0. This is because when d n ≦ 0, it means that the vertex coordinates V n are on the image plane of the sensor image or on the opposite side of the object H with respect to the image plane.

[0050] The pixel value reading means 414 selects the pixel of the image coordinates of the sensor image corresponding to the vertex coordinates V n and reads the pixel value when it is determined by the foremost determination means 413 that it is the closest to the viewpoint position. Here, as shown in the following formula (8), the pixel value reading means 414 reads the pixel value E(x n ) from the sensor image E and outputs it as the pixel value e n for the synthesis means 43 only when the determination result of the front determination means 413 is c = 1 (front).

[0051]

Equation

[0052] In addition, when the determination result is other than c = 1 (here, c = 0), the pixel value reading means 414 sets an identifier NaN (Not a Number) indicating a non - number to the output pixel value e n and outputs it to the synthesis means 43. Also, when the image coordinate x n is outside the sensor image E, the pixel value reading means 414 outputs e n = NaN to the synthesis means 43. For NaN, "NaN defined in the IEEE 754 floating - point standard" may be used. Also, for NaN, numerical values that do not occur within the sensor image E (for example, numerical values outside the value range, +∞, -∞, etc.) may be used. Also, for NaN, a predetermined specific numerical value (for example, 0, -1, etc.) may be used. With the configuration described above, the pixel value selection means 41 can select the pixel value of the sensor image corresponding to the vertex only when the vertex exists at the front. Returning to FIG. 5, the description of the configuration of the information multiplexing device 4 will be continued.

[0053] The weight calculation means 42 calculates the weight of each pixel value for synthesizing the pixel values selected by the plurality of pixel value selection means 41 (411, 412, 413). A plurality of weight calculation means 42 are provided according to the number of sensors 2. Here, the number of sensors 2 is three, and three weight calculation means 421, 422, and 423 are provided as the configuration. The weight calculation means 421, 422, and 423 have the same function, differing only in the input projection conversion information (first to third projection conversion information). The weight calculation means 42 calculates a larger weight as the resolution of the sensor image is higher using the projection conversion information. Here, the weight calculation means 42 calculates a weight proportional to the number of pixels (area) in which the small sphere at the vertex coordinates is imaged on the sensor image based on the projection conversion information.

[0054] Specifically, the weight calculation means 42 calculates the weight based on the vertex coordinates, the depth value obtained from the function related to the depth of the projection conversion information, and the focal length and pixel size included in the projection conversion information. Here, let the input vertex coordinates be V n , the depth value obtained from the function g(X) related to the depth of the projection conversion information be g(V n ), and the focal length and pixel size included in the projection conversion information be f and (r x , r y ) respectively. Then, the weight calculation means 42 calculates the weight w n by the following formula (9).

[0055]

Equation

[0056] Thus, the weight calculation means 42 can increase the weight as the resolution when projecting the vertex onto the sensor image is higher. The weight calculation means 42 outputs the calculated weight to the synthesis means 43.

[0057] The synthesis means 43 synthesizes a plurality of pixel values of the sensor image corresponding to the vertices of the vertex coordinates V n selected by the pixel value selection means 41 (411, 412, 413) based on the respective weights calculated by the weight calculation means 42 (421, 422, 423). Here, the synthesis means 43 calculates the weighted average of the plurality of pixel values based on the weights to obtain the vertex coordinates V nObtain the pixel value on the sensor image corresponding thereto. In addition, when the pixel value input from the pixel value selection means 41 includes a not-a-number (NaN), the pixel value is excluded and a weighted average is taken. Also, when all the pixel values input from the pixel value selection means 41 are not-a-number (NaN), or when the total sum of the weights is "0", the combining means 43 sets the output pixel value as not-a-number (NaN).

[0058] Here, assume that the number of sensor images is M (M is an integer of 1 or more; in the example of FIG. 5, M = 3), and pixel values e n (m) (m = 1, 2,..., M) are input from each of the pixel value selection means 41, and weights w n (m) (m = 1, 2,..., M) are input from each of the weight calculation means 42. In this case, the combining means 43 calculates the pixel value e obtained by taking the weighted average according to the following formula (10). n Calculate.

[0059]

Equation

[0060] The combining means 43 outputs the pixel value after combination (weighted average) to the modeling means 44. The modeling means 44 associates the vertex coordinates selected by the vertex selection means 40 with the pixel value combined by the combining means 43 corresponding to the vertex coordinates. That is, the modeling means 44 forms a three-dimensional model M shown in the following formula (11) in pairs with the vertex coordinates V n (n = 1,..., N) sequentially selected by the vertex selection means 40 and the pixel values e n (n = 1,..., N) output from the combining means 43. 3D Generate.

[0061]

Equation

[0062] With the configuration described above, the information multiplexing device 4 can multiplex sensor information onto three-dimensional shape data. In addition, the information multiplexing device 4 can generate a three-dimensional model with high accuracy because it synthesizes the three-dimensional shape data by weighting more the pixel values of the higher-resolution images from the sensor images acquired by the plurality of sensors 2. Note that the information multiplexing device 4 can be operated by a program (information multiplexing program) for causing a computer (not shown) to function as each of the above-described means.

[0063] <Operation of Information Multiplexing Device> Next, with reference to FIG. 7 (refer to FIGS. 5 and 6 as appropriate for the configuration), the operation of the information multiplexing device 4 will be described. Note that steps S1, S3 to S8 are performed in parallel for each sensor 2.

[0064] In step S1, the depth map generation means 410 generates a depth map when observing the three-dimensional shape data from the viewpoints, postures, and field angles specified by the projection conversion information. Here, the depth map generation means 410 of the pixel value selection means 411, 412, and 413 generates depth maps corresponding to different viewpoints from the three-dimensional shape data, respectively.

[0065] In step S2, the vertex selection means 40 sequentially selects the coordinates (vertex coordinates) of the vertices constituting the three-dimensional shape data. In step S3, the projection means 411 calculates the depth value and the image coordinates of the vertex selected in step S2 using the projection conversion information. Here, the projection means 411 of the pixel value selection means 411, 412, and 413 calculates the depth value of the vertex corresponding to each viewpoint and the image coordinates of the sensor image.

[0066] In step S4, the depth value reading means 412 reads the depth value corresponding to the image coordinates calculated in step S3 from the depth map generated in step S1. Here, the depth value reading means 412 of the pixel value selection means 411, 412, 413 reads the depth value corresponding to the image coordinates of the vertex from the depth maps corresponding to different viewpoints respectively. In step S5, the front determination means 413 compares the depth value of the vertex calculated using the projection conversion information in step S3 with the depth value of the image coordinates of the vertex read from the depth map in step S4, and determines whether the vertex is in the front. Here, the front determination means 413 of the pixel value selection means 411, 412, 413 performs the front determination of the vertex when viewed from different viewpoints of the sensor image.

[0067] If the vertex is determined to be in the front as a result of this determination (Yes in step S5), in step S6, the pixel value reading means 414 reads the pixel value of the image coordinates calculated in step S3 from the sensor image. On the other hand, if the vertex is not determined to be in the front (No in step S5), in step S7, the pixel value reading means 414 sets an identifier indicating a non - number predetermined for the pixel value. In steps S6 and S7, the pixel value reading means 414 of the pixel value selection means 411, 412, 413 reads the pixel value only when the vertex is in the front in each sensor image, and sets the pixel value as a non - number if it is not in the front.

[0068] In step S8, the weight calculation means 42 calculates the weight of the pixel value read in step S6. Here, the weight calculation means 421, 422, 423 calculate weights of magnitudes proportional to the number of pixels (area) in which the virtual small sphere at the vertex coordinates is imaged on the sensor image based on the respective projection conversion information. The weight calculation means 42 increases the weight as the number of these pixels is larger, that is, as the resolution is higher.

[0069] In step S9, the synthesizing means 43 synthesizes the pixel values read out for each of the pixel value selecting means 411, 412, and 413 in step S6 by taking a weighted average based on the weights calculated for each of the weight calculating means 421, 422, and 423 in step S8. However, the synthesizing means 43 excludes from the weighted average the pixel values for which NaN (Not a Number) was set in step S7.

[0070] In step S10, the modeling means 44 associates the coordinates of the vertices selected in step S2 with the pixel values synthesized in step S9. In step S11, the vertex selection means 40 determines whether all the vertices of the three-dimensional shape data have been selected. Here, if not all the vertices have been selected yet (No in step S11), the information multiplexing device 4 returns to step S2 and continues the operation. On the other hand, if all the vertices have been selected (Yes in step S11), the information multiplexing device 4 terminates the operation.

[0071] Through the above operations, the information multiplexing device 4 can multiplex the sensor information onto the three-dimensional shape data. As described above, the configuration and operation of the information multiplexing device 4 according to the embodiment of the present invention have been explained, but the present invention is not limited to this embodiment.

[0072] ≪Modification Example 1≫ Here, the three-dimensional shape data has been described by way of an example of a sequence of vector values representing the three-dimensional coordinates (x, y, z) of each vertex. However, the three-dimensional shape data may include, in addition to the three-dimensional coordinates of the vertices, the normal (normal vector) of each vertex. In this case, the weight calculating means 42 of the information multiplexing device 4 may calculate the weights taking the normal into account.

[0073] With reference to FIG. 8, the configuration of an information multiplexing device 4B, which is a configuration of a modification example of the information multiplexing device 4, will be described. It is assumed that the three-dimensional shape data input to the information multiplexing device 4B has a normal (normal vector) further associated with each vertex of the three-dimensional shape data input to the information multiplexing device 4 (FIG. 5). The information multiplexing device 4B includes a vertex selection means 40B, a pixel value selection means 41, a weight calculation means 42B, a synthesis means 43, and a modeling means 44. Since the configurations other than the vertex selection means 40B and the weight calculation means 42B are the same as those of the information multiplexing device 4, the same reference numerals are given and the description thereof is omitted.

[0074] The vertex selection means 40B sequentially selects the coordinates (vertex coordinates) of the vertices constituting the three-dimensional shape data and the corresponding normals. Here, the vertex selection means 40B sequentially selects the N vertices constituting the three-dimensional shape data S shown in the formula (1) and outputs them to the pixel value selection means 41, the weight calculation means 42B, and the modeling means 44. Hereinafter, the vertex coordinates selected at the n-th time when sequentially selecting the N vertex coordinates are V n is defined as In addition, the vertex selection means 40B sequentially selects the normals associated with the respective vertices of the three-dimensional shape data S in the same order as the vertices and outputs them to the weight calculation means 42B. Hereinafter, the normal selected at the n-th time when sequentially selecting the N normals corresponding to the vertices is H n is defined as

[0075] The weight calculation means 42B calculates the weights of the respective pixel values for synthesizing the pixel values selected by the plurality of pixel value selection means 41 (411, 412, 413). A plurality of weight calculation means 42B are provided according to the number of sensors 2. Here, as in the information multiplexing device 4 (FIG. 5), the number of sensors 2 is three, and three weight calculation means 42B1, 42B2, and 42B3 are provided as a configuration. The weight calculation means 42B1, 42B2, and 42B3 have the same functions except that the input projection conversion information (first to third projection conversion information) is different.

[0076] The weight calculation means 42B calculates a weight according to the degree to which the normal line faces the viewpoint direction in the sensor image. For example, the weight calculation means 42B calculates the weight based on the normal line H n and the vector T-V n from the vertex coordinates V n to the viewpoint position T included in the projection conversion information, and increases the weight as the angle formed therewith approaches 0°. Specifically, the weight calculation means 42B calculates the weight w n by the following formula (12).

[0077]

Equation

[0078] The weight calculation means 42B outputs the calculated weight to the synthesis means 43. Note that the weight calculation means 42B may calculate a weight considering both the weight based on the resolution shown in formula (9) and the weight based on the degree of facing in the viewpoint direction shown in formula (12). Specifically, the weight calculation means 42B calculates the weight w n by the following formula (13).

[0079]

Equation

[0080] According to this formula (13), for pixels corresponding to a viewpoint position with a high resolution when the vertex is projected onto the sensor image and a large degree of facing the vertex, the weight can be made heavier. In this way, by considering the resolution and the direction from the viewpoint position, the information multiplexing device 4B can generate a three-dimensional model with higher accuracy than when considering either one alone. Note that the information multiplexing device 4B can be operated by a program (information multiplexing program) for causing a computer (not shown) to function as each of the above-described means.

[0081] In addition, the operation of the information multiplexing device 4B differs only in that the vertex selection means 40B selects vertex coordinates and a normal vector at step S2, and the weight calculation means 42B calculates a weight based on the normal vector at step S8, compared with the information multiplexing device 4 described with reference to FIG. 7. Since the other operations are the same as those of the information multiplexing device 4, the description thereof is omitted.

[0082] <<Modification Example 2>> Here, it is assumed that a plurality of sensors 2 are provided to multiplex sensor information onto three-dimensional shape data. However, only one sensor 2 may be used.

[0083] With reference to FIG. 9, the configuration of an information multiplexing device 4C, which is a modified example of the information multiplexing device 4 when only one sensor 2 is used, will be described. Since there is only one sensor 2, a sensor image and projection conversion information corresponding to the single sensor 2 are input to the information multiplexing device 4C. In addition, it is assumed that the three-dimensional shape data input to the information multiplexing device 4C has a normal vector (normal line vector) associated with each vertex, similar to the information multiplexing device 4B (FIG. 8). The information multiplexing device 4C includes a vertex selection means 40B, an acceptance / rejection determination means 45, a pixel value selection means 41, and a modeling means 44B.

[0084] The vertex selection means 40B sequentially selects the coordinates of the vertices (vertex coordinates) constituting the three-dimensional shape data and the corresponding normal vectors. The vertex selection means 40B has the same function as the vertex selection means 40B of the information multiplexing device 4B (FIG. 8). However, here, the vertex selection means 40B sequentially outputs the normal vectors associated with the respective vertices of the three-dimensional shape data to the acceptance / rejection determination means 45. In addition, the vertex selection means 40B sequentially outputs the coordinates of the vertices (vertex coordinates) constituting the three-dimensional shape data to the pixel value selection means 41.

[0085] The acceptance / rejection determination means 45 determines whether or not the vertex at the vertex coordinates selected by the vertex selection means 40B should be adopted as a vertex of the three-dimensional model based on the direction of the normal vector of the vertex selected by the vertex selection means 40B. The acceptance / rejection determination means 45, for example, uses the selected normal line H n and the vertex coordinate V n to calculate the vector T-V from the vertex coordinate V to the viewpoint position T included in the projection conversion information. If the angle formed by the vector T-V and the normal line H is less than 45°, the vertex is regarded as having its normal line facing the direction of the viewpoint position, and the vertex is adopted as a vertex of the three-dimensional model. n Specifically, the acceptance / rejection determination means 45 calculates the determination result b of vertex acceptance / rejection by the following formula (14). Here, when the selected vertex is adopted, b = 1, and when it is not adopted, b = 0.

[0086]

Equation

[0087] Note that the inequality sign “>” for the magnitude determination in formula (14) may be the inequality sign “≧” including the equal sign. That is, vertices with an angle of 45° or less may be adopted. The acceptance / rejection determination means 45 outputs the determination result to the modeling means 44B.

[0088] The pixel value selection means 41 uses the projection conversion information to calculate the two-dimensional coordinates of the sensor image corresponding to the vertex selected by the vertex selection means 40B, and selects the pixel value of the two-dimensional coordinates of the sensor image when the selected vertex is the foremost among the vertices corresponding to the calculated two-dimensional coordinates. The pixel value selection means 41 has the same function as the pixel value selection means 41 of the information multiplexing device 4 (Fig. 5) and the same configuration as the pixel value selection means 41 described in Fig. 6. However, here, the pixel value selection means 41 outputs the selected pixel value to the modeling means 44B.

[0089] The modeling means 44B multiplexes the vertex coordinates selected by the vertex selection means 40B and the pixel values selected by the pixel value selection means 41 corresponding to the vertex coordinates. However, for vertices not adopted as vertices according to the determination result of the acceptance / rejection determination means 45, no association with pixel values is made. As a result, the information multiplexing device 4C can accurately generate a three-dimensional model by using the pixel values of the pixels corresponding to the viewpoint positions with a large degree of facing the vertex. Note that the information multiplexing device 4C can be operated by a program (information multiplexing program) for causing a computer (not shown) to function as each of the above-described means.

Explanation of Signs

[0090] 100 Three-dimensional model generation system 1 Camera 2 Sensor 3 Three-dimensional shape estimation device 4 Information multiplexing device 40 Vertex selection means 41 Pixel value selection means 410 Depth map generation means 411 Projection means 412 Depth value reading means 413 Front determination means 414 Pixel value reading means 42 Weight calculation means 43 Composition means 44 Modeling means 45 Acceptance / rejection determination means

Claims

1. An information multiplexing device that multiplexes the measurement values onto the three-dimensional shape data from the three-dimensional shape data of the object, a sensor image having the two-dimensional measurement values of the object obtained from a plurality of sensors as pixel values, and projection conversion information for projecting the measurement values of the object onto the sensor image, comprising: vertex selection means for sequentially selecting vertices of the three-dimensional shape data; for each sensor, using the projection conversion information, calculating the image coordinates of the sensor image corresponding to the selected vertex, and when the selected vertex is the vertex closest to the viewpoint position of the projection conversion among the vertices of the three-dimensional shape data corresponding to the image coordinates, pixel value selection means for selecting the pixel value of the image coordinates in association with the selected vertex; for each sensor, using the projection conversion information, weight calculation means for calculating a weight that is larger as the resolution of the sensor image is higher; for each selected vertex, combining means for combining the pixel values for each sensor selected by the pixel value selection means based on the weight calculated by the weight calculation means; modeling means for associating the pixel value combined by the combining means with the selected vertex; An information multiplexing device characterized by comprising.

2. The pixel value selection means includes: depth map generation means for generating a depth map corresponding to the sensor image from the three-dimensional shape data using the projection conversion information corresponding to the sensor; projection means for calculating the image coordinates and depth value of the sensor image corresponding to the vertex selected by the vertex selection means using the projection conversion information; depth value reading means for reading the depth value corresponding to the image coordinates calculated by the projection means from the depth map; frontmost determination means for comparing the depth value calculated by the projection means with the depth value read from the depth map and determining whether the vertex selected by the vertex selection means is the closest to the viewpoint position of the projection conversion; pixel value reading means for selecting the pixel of the image coordinates of the selected vertex and reading the pixel value from the sensor image when it is determined that it is the closest to the viewpoint position; The information multiplexing device according to claim 1, characterized by comprising.

3. The projection conversion information includes information specifying a function g(X) that associates three-dimensional coordinates X in the three-dimensional space where the object exists with depth values, a focal length f, and the pixel size (r ｘ , r ｙ ) of the sensor, The weight calculation means sets the vertex coordinates of the vertex selected by the vertex selection means as V ｎ When (n is an integer from 1 to N, and N is the number of vertices), the weight w ｎ is 【Number 1】 The information multiplexing device according to claim 1, characterized by calculating according to.

4. In the three-dimensional shape data, a normal is associated with the vertex, The vertex selection means selects the vertex in association with the normal. The weight calculation means calculates a weight that is larger as the resolution of the sensor image is higher and as the angle formed by the normal line and the vector from the vertex to the viewpoint position of the projection transformation is smaller, using the projection transformation information. The information multiplexing device according to claim 1, characterized in that.

5. The projection conversion information includes information specifying a function g(X) that associates the three-dimensional coordinates X in the three-dimensional space where the object exists with the depth value, a viewpoint position T, a focal length f, and the pixel size (r ｘ , r ｙ ) of the sensor, The weight calculation means sets the vertex coordinates of the vertex selected by the vertex selection means as V ｎ , the normal as H ｎ (where n is an integer from 1 to N, and N is the number of vertices), and the weight w ｎ as 【Number 2】 The information multiplexing device according to claim 4, characterized in that it is calculated by.

6. A program for causing a computer to function as the information multiplexing device according to any one of claims 1 to 5.

7. A three-dimensional model generation system for generating a three-dimensional model in which the measurement values of a sensor are multiplexed with the three-dimensional shape data of an object, A plurality of cameras for photographing the object, A plurality of sensors for outputting a sensor image having the two-dimensional measurement value of the object as a pixel value, A three-dimensional shape estimation device for estimating the three-dimensional shape data of the object from the camera images photographed by the plurality of cameras, From the three-dimensional shape data estimated by the three-dimensional shape estimation device, the plurality of sensor images acquired from the plurality of sensors, and the projection transformation information for projecting the measurement value of the object onto the sensor image, the information multiplexing device according to any one of claims 1 to 5 for multiplexing the measurement value onto the three-dimensional shape data, A three-dimensional model generation system, characterized in that it comprises.

Images