Image processing device, image processing method, and program

Abstract

The present disclosure pertains to an image processing device, an image processing method, and a program with which it is possible to easily generate a 3D model provided with spectral information. A spectral 3D model is generated based on the group of RGB images, and spectral information of the group of spectral images is mapped to generate a spectral 3D model in which the spectral information is mapped to the 3D model. The present disclosure can be applied to a technique for generating a spectral 3D image in which spectral information is superimposed on a 3D model.

Description

Technical Field

[0001] The present disclosure relates to an image processing device, an image processing method, and a program, and particularly to an image processing device, an image processing method, and a program capable of easily generating a three-dimensional model including spectral information. Background Art

[0002] The following technology has been proposed: generating a three-dimensional image including spectral information by using a two-dimensional RGB camera to generate a three-dimensional model from an RGB image group obtained by capturing images from multiple angles, and then superimposing spectral information captured by a spectral camera on the three-dimensional model (see Patent Document 1).

[0003] [Citation List]

[0004] [Patent Document]

[0005] [Patent Document 1]

[0006] WO 2011 / 054040 Summary of the invention

[0007] [Technical issues]

[0008] However, in the technology described in Patent Document 1, since the spectral camera only captures one spectral image, spectral information can only be superimposed on the three-dimensional model from a specific viewpoint where the spectral image is captured. Therefore, a three-dimensional image including spectral information from a desired viewpoint cannot be generated.

[0009] Furthermore, the technology described in Patent Document 1 cannot generate a three-dimensional model including spectral information required to generate a three-dimensional image including spectral information from a desired viewpoint, and thus cannot use an application requiring a three-dimensional model including spectral information.

[0010] Given that this has been achieved, the present disclosure enables easy generation of three-dimensional models including, among other things, spectral information.

[0011] [Solution to the problem]

[0012] The image processing device and program involved in the first aspect of the present invention are equipped with a spectral three-dimensional model generation unit, which generates a spectral three-dimensional model that maps the above-mentioned spectral information to the above-mentioned three-dimensional model based on a spectral image group composed of a plurality of spectral images having spectral information of a predetermined space and a spatial image group composed of a plurality of spatial images used to generate a three-dimensional model of the predetermined space.

[0013] The image processing method involved in the first aspect of the present invention includes: a step of generating a spectral three-dimensional model, which maps the above-mentioned spectral information to the above-mentioned three-dimensional model based on a spectral image group composed of a plurality of spectral images having spectral information of a predetermined space and a spatial image group composed of a plurality of spatial images used to generate the three-dimensional model of the above-mentioned predetermined space.

[0014] In a first aspect of the present invention, a spectral three-dimensional model is generated, wherein the spectral information has been mapped onto the three-dimensional model based on a spectral image group consisting of a plurality of spectral images having spectral information of a predetermined space and a spatial image group consisting of a plurality of spatial images for generating a three-dimensional model of the predetermined space.

[0015] The image processing device involved in the second aspect of the present invention comprises: a spectral three-dimensional model generating unit, which generates a spectral three-dimensional model, which maps the above-mentioned spectral information to the above-mentioned three-dimensional model based on a spectral image group composed of a plurality of spectral images having spectral information of a predetermined space and a spatial image group composed of a plurality of spatial images used to generate the three-dimensional model of the above-mentioned predetermined space; and a spectral information extraction unit of an object of interest, which extracts spectral information of an object of interest from the spectral information.

[0016] According to a second method of the present invention, a spectral three-dimensional model is generated, which maps the above-mentioned spectral information to the above-mentioned three-dimensional model based on a spectral image group composed of a plurality of spectral images having spectral information of a predetermined space and a spatial image group composed of a plurality of spatial images used to generate the three-dimensional model of the above-mentioned predetermined space; and spectral information of an object of interest is extracted from the spectral information. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 : is a diagram showing an overview of a first embodiment of the present disclosure.

[0018] Figure 2 is a diagram showing a first configuration example of the first embodiment of the image processing device of the present disclosure.

[0019] Figure 3 : is a diagram showing the format of input / output data in the image processing apparatus of the present disclosure.

[0020] Figure 4 It is shown by Figure 2 FIG. 1 is a flowchart of a spectral three-dimensional model generation process of the image processing device shown in FIG.

[0021] Figure 5 is a diagram showing a second configuration example of the first embodiment of the image processing device of the present disclosure.

[0022] Figure 6 It is shown by Figure 5 FIG. 1 is a flowchart of a spectral three-dimensional model generation process of the image processing device shown in FIG.

[0023] Figure 7 is a diagram showing a third configuration example of the first embodiment of the image processing device of the present disclosure.

[0024] Figure 8 It is shown by Figure 7 Flowchart of a spectral three-dimensional model generation process of the image processing device shown in FIG.

[0025] Fig. 9 : is a diagram showing a fourth configuration example of the first embodiment of the image processing device of the present disclosure.

[0026] Fig.10 It is shown by Fig. 9 Flowchart of a spectral three-dimensional model generation process of the image processing device shown in FIG.

[0027] Fig.11 is a diagram showing a fifth configuration example of the first embodiment of the image processing device of the present disclosure.

[0028] Fig.12 It is shown by Fig.11 FIG. 1 is a flowchart of a spectral three-dimensional model generation process of the image processing device shown in FIG.

[0029] Fig.13 It is a diagram showing a sixth configuration example of the first embodiment of the image processing device of the present invention.

[0030] Fig.14 It is shown by Fig.13 FIG. 1 is a flowchart of a spectral three-dimensional model generation process of the image processing device shown in FIG.

[0031] Fig.15 It is a diagram showing a seventh configuration example of the first embodiment of the image processing device of the present invention.

[0032] Fig.16 It is shown Fig.15 Flowchart of a spectral three-dimensional model generation process of the image processing device shown in FIG.

[0033] Fig.17 It is a diagram showing an eighth configuration example of the first embodiment of the image processing device of the present invention.

[0034] Fig.18 It is shown by Fig.17 FIG. 1 is a flowchart of a spectral three-dimensional model generation process of the image processing device shown in FIG.

[0035] Fig.19It is a diagram showing a ninth configuration example of the first embodiment of the image processing device of the present disclosure.

[0036] Fig. 20 It is shown Fig.19 FIG. 1 is a flowchart of a spectral three-dimensional model generation process of the image processing device shown in FIG.

[0037] Fig.21 It is a diagram showing a tenth configuration example of the first embodiment of the image processing device of the present disclosure.

[0038] Fig. 22 It is shown Fig.21 FIG. 4 is a flowchart of a spectral three-dimensional model generation process of the image processing device shown in FIG.

[0039] Fig.23 It is a diagram showing an outline of a second embodiment of the present disclosure.

[0040] Fig.24 is a diagram showing a first configuration example of the second embodiment of the image processing device of the present invention.

[0041] Fig.25 It is shown by Fig.24 Flowchart of a process for generating a spectral three-dimensional model of an object of interest by the image processing device shown in FIG.

[0042] Fig.26 1 is a schematic diagram showing a second configuration example of the second embodiment of the image processing device of the present disclosure.

[0043] Fig. 27 Yes means Fig.26 FIG. 1 is a flowchart of a process for generating a three-dimensional model of a spectrum of an object of interest by an image processing device.

[0044] Fig.28 It is a diagram showing a third configuration example of the second embodiment of the image processing device of the present disclosure.

[0045] Fig.29 It is shown by Fig.28 FIG. 1 is a flowchart of a process of generating a three-dimensional model of a spectrum of an object of interest by the image processing device shown in FIG.

[0046] Fig.30 It is a diagram showing a fourth configuration example of the second embodiment of the image processing device of the present disclosure.

[0047] Fig.31 It is shown by Fig.30 Flowchart of the process of generating a spectral three-dimensional model of an object of interest by the image processing device shown in FIG.

[0048] Fig.32is a diagram showing a fifth configuration example of the second embodiment of the image processing device of the present disclosure.

[0049] Fig.33 It is shown by Fig.32 Flowchart of a process for generating a spectral three-dimensional model of an object of interest by the image processing device shown in FIG.

[0050] Fig.34 It is a diagram showing a sixth configuration example of the second embodiment of the image processing device of the present disclosure.

[0051] Fig.35 It is shown by Fig.34 Flowchart of a process for generating a spectral three-dimensional model of an object of interest by the image processing device shown in FIG.

[0052] Fig.36 It is a diagram showing a seventh configuration example of the second embodiment of the image processing device of the present disclosure.

[0053] Fig.37 Yes means Fig.36 FIG. 1 is a flowchart of a process of generating a three-dimensional model of a spectrum of an object of interest by an image processing device.

[0054] Fig.38 It is a diagram showing an eighth configuration example of the second embodiment of the image processing device of the present disclosure.

[0055] Fig.39 Yes means Fig.38 FIG. 1 is a flowchart of a process for generating a three-dimensional model of a spectrum of an object of interest by an image processing device.

[0056] Fig.40 It is a diagram showing a ninth configuration example of the second embodiment of the image processing device of the present disclosure.

[0057] Fig.41 It is shown by Fig.40 Flowchart of the process of generating a spectral three-dimensional model of an object of interest by the image processing device shown in FIG.

[0058] Fig.42 It is a diagram showing a tenth configuration example of the second embodiment of the image processing device of the present invention.

[0059] Fig.43 It is shown by Fig.42 FIG. 1 is a flowchart of a process of generating a three-dimensional model of a spectrum of an object of interest by the image processing device shown in FIG.

[0060] Fig.44 An example of the configuration of a general-purpose computer is shown. DETAILED DESCRIPTION

[0061] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that in this specification and the drawings, components having substantially the same functional configuration will be denoted by the same reference numerals, and repeated descriptions thereof will be omitted.

[0062] Hereinafter, an embodiment for carrying out the present technology will be described. The description will be given in the following order.

[0063] 1. Overview of the Invention

[0064] 2. First Implementation

[0065] 3. Second Implementation

[0066] 4. Examples of software execution

[0067] <<1. Overview of the Invention>>

[0068] <Generation of Three-Dimensional Image Including Spectral Information>

[0069] The present disclosure enables easy generation of three-dimensional models including, among other things, spectral information.

[0070] In order to describe the technology of the present disclosure, in order to easily generate a three-dimensional model including spectral information, a technology for generating a three-dimensional image including spectral information will first be described.

[0071] Hereinafter, a three-dimensional image including spectral information is also referred to as a “spectral three-dimensional image”, and a three-dimensional model including spectral information is also referred to as a “spectral three-dimensional model”.

[0072] Figure 1 The upper part shows an example of the configuration of an image processing device that generates a spectral three-dimensional image, Figure 1 The lower part of FIG. 1 shows an example of a configuration of an image processing device that generates a spectral three-dimensional model for describing the overview of the present disclosure.

[0073] Figure 1 The upper image processing device 11 generates a spectral three-dimensional image from a group of a plurality of RGB images and a single spectral image.

[0074] Specifically, the image processing device 11 includes an RGB camera 21 , a spectral camera 22 , a three-dimensional image constructing unit 23 , and a spectral information mapping unit 24 .

[0075] The RGB camera 21 generates an RGB image group composed of a plurality of RGB images by photographing the subject from a plurality of angles, and outputs the generated RGB image group to the three-dimensional image constructing unit 23 .

[0076] The spectral camera 22 is, for example, a camera including a multispectral sensor or the like, and captures a single spectral image of an object and outputs the image to the spectral information mapping unit 24 .

[0077] The 3D image constructing unit 23 constructs an RGB 3D image of the subject viewed from the viewpoint of the spectral camera 22 from the RGB image group, for example, by using structure from motion (SfM), and outputs the obtained image to the spectral information mapping unit 24 .

[0078] The spectral information mapping unit 24 maps spectral information from corresponding positions of the spectral image to an RGB 3D image corresponding to the shooting angle of the spectral image based on the 3D image of the subject, thereby generating and outputting a spectral 3D image including spectral information.

[0079] Although the spectral three-dimensional image is generated by the above processing, Figure 1 The spectral three-dimensional image generated by the upper image processing device 11 corresponds only to the acquisition angle of the spectral image acquired by the spectral camera 22, and therefore cannot generate a spectral three-dimensional image from a desired viewpoint position.

[0080] Therefore, in the present disclosure, in addition to the RGB image, a spectral image group is obtained by capturing spectral images from multiple angles, such as in Figure 1 The image processing device is shown in the lower part.

[0081] Then, spectral information from the RGB image and the spectral image captured at the corresponding angle is mapped to the RGB image to generate a spectral information mapping image, and a spectral three-dimensional model is generated based on the spectral information mapping image group.

[0082] More specifically, in Figure 1 The lower image processing device 31 includes an RGB camera 41 , a spectral camera 42 , a spectral information mapping processing unit 43 and a three-dimensional reconfiguration unit 44 .

[0083] The RGB camera 41 has the same structure as the RGB camera 21 , and forms an RGB image group by photographing a subject from multiple angles, and outputs the RGB image group to the spectral information mapping processing unit 43 .

[0084] The basic function of the spectral camera 42 is the same as that of the spectral camera 22 , but a spectral image group is formed by taking spectral images from a plurality of angles and outputting the spectral images to the spectral information mapping processing unit 43 .

[0085] When obtaining the RGB image group and the spectral image group, the spectral information mapping processing unit 43 maps spectral information to the RGB image pixel by pixel based on the corresponding RGB image and the spectral image, thereby generating a spectral information mapping image. Thus, the spectral information mapping image group is output to the three-dimensional reconfiguration unit 44.

[0086] The three-dimensional reconfiguration unit 44 reconfigures the spectral three-dimensional model by configuring three-dimensional point cloud information composed of spectral information based on the spectral information mapping image group.

[0087] By configuring in this way, it is possible to reconstruct the spectral three-dimensional model, and to generate a spectral three-dimensional image from a desired viewpoint position based on the spectral three-dimensional model.

[0088] <<2. First embodiment>>

[0089] <2-1. First Configuration Example of First Embodiment>

[0090] Next, we will refer to Figure 2 A first configuration example of the first embodiment of the image processing device that generates a spectral three-dimensional model is described.

[0091] Figure 2 The image processing device 101 obtains an RGB image group and a spectral image group taken from multiple angles, maps the RGB image and the spectral image taken at corresponding angles to the RGB image to generate a spectral information mapping image, and generates a spectral three-dimensional model based on the spectral information mapping image group.

[0092] More specifically, Figure 2 The image processing device 101 in the embodiment includes an RGB camera 111, a spectral camera 112, a feature point matching processing unit 113, a pixel-by-pixel registration processing unit 114, a mapping processing unit 115 and a three-dimensional reconfiguration unit 116.

[0093] The RGB camera 111 corresponds to Figure 1 RGB camera 41 in the lower part. The RGB camera 111 configures an RGB image group by capturing an object from multiple angles, and outputs the RGB image group to the feature point matching processing unit 113.

[0094] The images constituting the RGB image group are not limited to RGB images, but may also include brightness images, such as Figure 3 as shown in the upper part of .

[0095] The format of the RGB image can be a three-dimensional array (3D array) with image height H, image width W and RGB, for example, for a plurality of captured images N, such as Figure 3In this case, the size of the array corresponding to the size of a single image is image height H×image width W×3 (H×W×3).

[0096] Likewise, the format of the luminance image may be a two-dimensional array (2D array) having an image height H and an image width W, for example, for the number N of captured images, such as Figure 3 In this case, the size of the array corresponding to the size of a single image is image height H×image width W (H×W).

[0097] The spectral camera 112 corresponds to Figure 1 The spectral camera 42 at the lower part is constituted by, for example, a multi-spectral sensor, etc. The spectral camera 112 constitutes a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the feature point matching processing unit 113 .

[0098] The format of the spectral image can be a two-dimensional array (2D array) with wavelength numbers Nw, for example, for a plurality of captured images N, such as Figure 3 In this case, the size of the array corresponding to the size of a single image is image height H × image width W (H×W).

[0099] The format of the spectral image can be a three-dimensional array (3D array) with image height H, image width W and number of wavelengths Nw, for example, for the number of captured images N, as Figure 3 In this case, the size of the array corresponding to the size of a single image is image height H×image width W×number of wavelengths Nw (H×W×Nw).

[0100] In addition, the format of the spectral image can be a two-dimensional array (2D array) with wavelength number Nw×image height H and image width W, for example, for a plurality of captured images N, such as Figure 3 In this case, the size of the array corresponding to the size of a single image is (wavelength number Nw×image height H)×image width W ((Nw×H)×W).

[0101] When the RGB image group and the spectral image group are obtained, the feature point matching processing unit 113 ( Figure 2 ) extracts feature points from the corresponding RGB image and spectral image, matches the extracted feature points, and outputs information about the matched feature points together with the RGB image group and the spectral image group to the pixel-by-pixel registration processing unit 114.

[0102] The pixel-by-pixel registration processing unit 114 generates registration information by matching the pixels of the feature points in the RGB image with the feature point matching processing unit 113 and aligning the pixels of the feature points in the spectral image provided from the feature point matching processing unit 113 in association with each other, and outputs the registration information together with the RGB image group and the spectral image group to the mapping processing unit 115.

[0103] The mapping processing unit 115 maps the spectral information of the spectral image consistent with the feature points registered corresponding to the RGB image to each pixel of the RGB image based on the registration information provided from the pixel-by-pixel registration processing unit 114, thereby generating a spectral information mapping image, which is output as a spectral information mapping image group to the three-dimensional reconfiguration unit 116, and the spectral information mapping image is mapped to each pixel of the RGB image.

[0104] In other words, the feature point matching processing unit 113, the pixel-by-pixel registration processing unit 114, and the mapping processing unit 115 correspond to Figure 1 Configuration of the spectrum information mapping processing unit 43 of the image processing device 31 at the lower part.

[0105] The three-dimensional reconfiguration unit 116 reconfigures the spectral three-dimensional model by mapping the image group based on the spectral information, configuring the three-dimensional point cloud information composed of the spectral information by structure from motion (SfM), or the like.

[0106] Here, the spectral three-dimensional model may be spectral three-dimensional point cloud (spectral 3D point cloud) information having three-dimensional coordinates at each vertex of the three-dimensional mesh constituting the object and data including information about each spectral reflectance, such as in Figure 3 The top of the lower part is shown.

[0107] The spectral three-dimensional model may also be information about a spectral three-dimensional grid having data including three-dimensional coordinates at each vertex of the three-dimensional grid constituting the object, and, for each, information about spectral reflectance constituted by array data in a format similar to that of the spectral image, such as Figure 3 The bottom of the lower part is shown.

[0108] According to such a configuration, a spectral information map image group can be generated from the RGB image group and the spectral image group, and a spectral three-dimensional model can be reconstructed based on the generated spectral information map image group.

[0109] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0110] < Figure 2 Spectral three-dimensional model generation processing of image processing device in>

[0111] Next, we will refer to Figure 4 The flowchart in Figure 2 Spectral three-dimensional model generation processing of the image processing device 101 in.

[0112] In step S11 , the RGB camera 111 configures an RGB image group by capturing an object from multiple angles, and outputs the RGB image group to the feature point matching processing unit 113 .

[0113] In step S12 , the spectral camera 112 forms a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the feature point matching processing unit 113 .

[0114] In step S13 , the feature point matching processing unit 113 resets the counter f for counting the number of frames of the RGB image group and the spectral image group to 1.

[0115] In step S14, the feature point matching processing unit 113 extracts feature points from the RGB image and the spectral image of frame f, matches the extracted feature points, and outputs the information of the matched feature points together with the RGB image group and the spectral image group of frame f to the pixel-by-pixel registration processing unit 114.

[0116] In step S15, the pixel-by-pixel registration processing unit 114 generates registration information by associating the pixels of the feature points in the RGB image and the pixels of the feature points in the spectral image matched and provided from the feature point matching processing unit 113 with each other through registration processing, and outputs the registration information together with the RGB image group and the spectral image group to the mapping processing unit 115.

[0117] In step S16, the mapping processing unit 115 maps the spectral information of the spectral image registered correspondingly to each pixel of the RGB image based on the registration information provided from the pixel-by-pixel alignment processing unit 114, thereby generating a spectral information mapping image, and outputs it to the three-dimensional reconstruction unit 116 as the spectral information mapping image of frame f, which maps each pixel of the RGB image of frame f.

[0118] At this time, the three-dimensional reconfiguration unit 116 stores the spectrum information map image of the frame f in association with the frame f.

[0119] In step S17 , the feature point matching processing unit 113 determines whether the counter f is below the frame number N, and if the counter f is below the frame number N, the sequence moves to step S18 .

[0120] In step S18 , the feature point matching processing unit 113 increases the counter f by 1, and the sequence returns to step S14 .

[0121] That is, the processing of steps S14 to S18 is repeated until the spectral information map images are generated for all N frames, which is the number of images included in the RGB image group and the spectral image group.

[0122] If it is determined in step S17 that the counter f is not lower than N and it is therefore determined that all spectral information map images have been generated, the sequence moves to step S19 .

[0123] In step S19 , the three-dimensional reconfiguration unit 116 reconfigures the spectral three-dimensional model by configuring three-dimensional point cloud information composed of spectral information based on the spectral information mapping image group.

[0124] According to the above processing, a spectral information mapping image group can be generated from the RGB image group and the spectral image group, and a spectral three-dimensional model can be reconstructed based on the generated spectral information mapping image group.

[0125] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0126] The foregoing assumes that the image capture of the RGB camera 111 and the spectral camera 112 is performed at the same timing and at a similar angle relative to the object. Thus, an example of mapping the spectral information to the RGB image is described based on the registration information generated by the feature point matching and registration process for the RGB image and the spectral image of the same frame number.

[0127] However, the RGB image and the spectral image for feature point matching do not necessarily have to be captured at the same timing as long as the images are captured at similar angles relative to the object.

[0128] However, in this case, for example, when taking RGB images and spectral images, it is necessary to use the RGB images and spectral images at the same angle to generate a spectral information mapping image so that the posture information of the RGB camera 111 and the spectral camera 112 can be obtained and the angle relative to the subject can be identified.

[0129] The spectral camera 112 is also not limited to a multi-spectral sensor, and may have another sensor, such as a temperature sensor or the like.

[0130] Furthermore, the RGB image group may be replaced by the luminance image group as described above, or may be any other image group, such as a grayscale image group, as long as the images can be captured by the RGB camera 111 .

[0131] The RGB camera 111 may also be replaced by a monochrome camera.

[0132] Additionally, an illumination estimation process may be added to the spectral images captured by the spectral camera 112 .

[0133] In this way, by adding illumination estimation processing to the spectral image captured by the spectral camera 112, the spectrum of ambient light can be separated from the spectral image to obtain a pure spectrum of the subject. This can obtain a more accurate spectral image, and as a result, a more accurate spectral three-dimensional model can be generated.

[0134] The resolutions of the RGB image group captured by the RGB camera 111 and the spectral image group captured by the spectral camera 112 may be the same, or may be different.

[0135] However, when the resolutions of the two are different, for example, upsampling, downsampling, etc. may be performed at a stage before mapping of spectral information of the spectral image to the RGB image so that the resolutions of the images match.

[0136] <2-2. Second Configuration Example of First Embodiment>

[0137] In the above, an example of generating a spectral information mapping image group from an RGB image group and a spectral image group to generate a spectral 3D model is described. However, a 3D point cloud of both the RGB image group and the spectral image group may be generated, and the spectral information aligned with the position of the spectral 3D point cloud may be mapped to the position of the RGB 3D point cloud to generate a spectral 3D model.

[0138] Figure 5 A second configuration example of the first embodiment of the image processing device is shown, wherein a spectral three-dimensional model is generated by generating a three-dimensional point cloud for an RGB image group and a spectral image group, and then mapping spectral information aligned with the position of the spectral three-dimensional point cloud with the position of the RGB three-dimensional point cloud.

[0139] Figure 5 The image processing device 101 includes an RGB camera 131, a spectral camera 132, three-dimensional reconfiguration units 133 and 134, a three-dimensional point cloud alignment processing unit 135, a corresponding point cloud extraction processing unit 136 and a spectral information mapping processing unit 137.

[0140] The RGB camera 131 and the spectral camera 132 correspond to the RGB camera 111 and the spectral camera 112 , respectively. The RGB camera 131 and the spectral camera 132 capture an RGB image group and a spectral image group, respectively, and output the RGB image group and the spectral image group to the three-dimensional reconfiguration units 133 and 134 .

[0141] The three-dimensional reconfiguration unit 133 reconfigures the RGB three-dimensional point cloud information based on the RGB image group and outputs the result as RGB three-dimensional point cloud information to the three-dimensional point cloud alignment processing unit 135 .

[0142] The three-dimensional reconfiguration unit 134 reconfigures the three-dimensional point cloud information of the spectral information based on the spectral image group, and outputs the result as spectral three-dimensional point cloud information to the three-dimensional point cloud alignment processing unit 135 .

[0143] The three-dimensional point cloud alignment processing unit 135 generates three-dimensional point cloud information including both RGB and spectral information by aligning the positions of the RGB three-dimensional point cloud and the spectral three-dimensional point cloud, and outputs the information to the corresponding point cloud extraction processing unit 136 .

[0144] The corresponding point cloud extraction processing unit 136 extracts information of a three-dimensional point cloud in, for example, nearby spectral information existing within a predetermined distance in association with each point in the RGB three-dimensional point cloud, and outputs the information to the spectral information mapping processing unit 137 .

[0145] The spectral information mapping processing unit 137 generates a spectral three-dimensional model by mapping spectral information at each point in the RGB three-dimensional point cloud using the nearby spectral information extracted by the corresponding point cloud extraction processing unit 136 in association with each RGB three-dimensional point cloud.

[0146] More specifically, the spectral information mapping processing unit 137 uses the spectral information within a predetermined distance extracted corresponding to each point of the RGB three-dimensional point cloud to generate the spectral information of each point of the RGB three-dimensional point cloud, maps the generated spectral information to each point of the RGB three-dimensional point cloud, and generates a spectral three-dimensional model.

[0147] The spectral information mapping processing unit 137 generates spectral information of each point in the RGB three-dimensional point cloud by calculating an average value of spectral information existing within a predetermined distance or a weighted average value according to distance extracted in association with each point in the RGB three-dimensional point cloud.

[0148] According to such a structure, an RGB three-dimensional point cloud and a spectral three-dimensional point cloud are generated based on the RGB image group and the spectral image group, these point clouds are aligned, and mapped based on the spectral information extracted corresponding to each point in the RGB three-dimensional point cloud, thereby generating a spectral three-dimensional model.

[0149] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0150] < Figure 5 Spectral three-dimensional model generation processing of image processing device in>

[0151] Next, refer to Figure 6 Flowchart Description Figure 5 The image processing device 101 performs a spectral three-dimensional model generation process.

[0152] In step S31 , the RGB camera 131 configures an RGB image group by capturing an object from multiple angles, and outputs the RGB image group to the three-dimensional reconfiguration unit 133 .

[0153] In step S32 , the spectral camera 132 forms a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the three-dimensional reconfiguration unit 134 .

[0154] In step S33 , the three-dimensional reconfiguration unit 133 generates an RGB three-dimensional point cloud based on the RGB image group, and outputs the result to the three-dimensional point cloud alignment processing unit 135 .

[0155] In step S34 , the three-dimensional reconfiguration unit 134 generates a spectral three-dimensional point cloud based on the spectral image group, and outputs the result to the three-dimensional point cloud alignment processing unit 135 .

[0156] In step S35 , the 3D point cloud alignment processing unit 135 generates 3D point cloud information mixed with RGB and spectral information by aligning the positions of the RGB 3D point cloud and the spectral 3D point cloud, and outputs the information to the corresponding point cloud extraction processing unit 136 .

[0157] In step S36 , the corresponding point cloud extraction processing unit 136 extracts spectral information existing within a predetermined distance associated with each point in the RGB three-dimensional point cloud based on the three-dimensional point cloud information of mixed RGB and spectral information, and outputs the result to the spectral information mapping processing unit 137 .

[0158] In step S37, the spectral information mapping processing unit 137 generates a spectral three-dimensional model by mapping the average value of spectral information within a predetermined distance, the weighted average value according to the distance, etc. as spectral information to each point in the RGB three-dimensional point cloud, wherein the spectral information is provided from the corresponding point cloud extraction processing unit 136 that has been extracted in association with each point in the RGB three-dimensional point cloud.

[0159] Through such processing, an RGB three-dimensional point cloud and a spectral three-dimensional point cloud are generated based on the RGB image group and the spectral image group, the point clouds are made consistent, and mapping is performed based on the spectral information extracted corresponding to each point in the RGB three-dimensional point cloud, thereby generating a spectral three-dimensional model.

[0160] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0161] <2-3. Third Configuration Example of First Embodiment>

[0162] The above describes an example of generating a spectral three-dimensional model by obtaining a three-dimensional point cloud from each of an RGB image group and a spectral image group, aligning their positions, synthesizing the RGB three-dimensional point cloud and the spectral three-dimensional point cloud, and then mapping spectral information onto the RGB three-dimensional point cloud.

[0163] However, the 3D point cloud information of the mixture of RGB 3D point cloud and spectral 3D point cloud can be obtained directly from the RGB image group and the spectral image group, and the spectral 3D model can be generated by mapping the spectral information onto the RGB 3D point cloud with RGB+spectral 3D point cloud information. In the following, the 3D point cloud information of the mixture of RGB 3D point cloud and spectral 3D point cloud obtained directly from the RGB image group and the spectral image group will also be referred to as "RGB+spectral 3D point cloud information".

[0164] Figure 7 A third configuration example of the first embodiment of the image processing device is shown, wherein RGB+spectral three-dimensional point cloud information which is a mixture of RGB three-dimensional point cloud and spectral three-dimensional point cloud is obtained directly from the RGB image group and the spectral image group, and a spectral three-dimensional model is generated by mapping the spectral information in the RGB+spectral three-dimensional point cloud information onto the RGB three-dimensional point cloud.

[0165] Figure 7 The image processing device 101 in the embodiment includes an RGB camera 151 , a spectral camera 152 , a three-dimensional reconfiguration unit 153 , a corresponding point cloud extraction processing unit 154 , and a spectral information mapping processing unit 155 .

[0166] The RGB camera 151 and the spectral camera 152 correspond to the RGB camera 111 and the spectral camera 112 , respectively. The RGB camera 151 and the spectral camera 152 capture an RGB image group and a spectral image group, respectively, and output the RGB image group and the spectral image group to the three-dimensional reconfiguration units 133 and 134 .

[0167] The three-dimensional reconfiguration unit 153 reconfigures RGB+spectral three-dimensional point cloud information, which is three-dimensional point cloud information mixing RGB and spectral information, based on the RGB image group and the spectral image group and outputs the result to the corresponding point cloud extraction processing unit 154 .

[0168] The corresponding point cloud extraction processing unit 154 corresponds to the corresponding point cloud extraction processing unit 136. For example, based on the RGB+spectral three-dimensional point cloud information, the corresponding point cloud extraction processing unit 154 extracts information of a three-dimensional point cloud in nearby spectral information existing within a predetermined distance in association with each point in the RGB three-dimensional point cloud, and outputs the information to the spectral information mapping processing unit 155.

[0169] The spectral information mapping processing unit 155 corresponds to the spectral information mapping processing unit 137, and generates a spectral three-dimensional model by mapping the spectral information at each point in the RGB three-dimensional point cloud using the spectral information of the nearby spectral three-dimensional point cloud extracted in association with each RGB three-dimensional point cloud by the corresponding point cloud extraction processing unit 154.

[0170] According to such a structure, a three-dimensional point cloud mixed with RGB and spectral information, i.e., RGB+spectral three-dimensional point cloud information, is generated based on the RGB image group and the spectral image group. Based on the RGB+spectral three-dimensional point cloud information, mapping is performed based on the spectral information extracted corresponding to each point in the RGB three-dimensional point cloud, thereby generating a spectral three-dimensional model.

[0171] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0172] < Figure 7 Spectral three-dimensional model generation processing of image processing device in>

[0173] Next, refer to Figure 8 Flowchart Description Figure 7 The image processing device 101 performs a spectral three-dimensional model generation process.

[0174] In step S51 , the RGB camera 151 configures an RGB image group by capturing an object from multiple angles, and outputs the RGB image group to the three-dimensional reconfiguration unit 153 .

[0175] In step S52 , the spectral camera 152 forms a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the three-dimensional reconfiguration unit 153 .

[0176] In step S53 , the three-dimensional reconfiguration unit 153 generates RGB+spectral three-dimensional point cloud information based on the RGB image group and the spectral image group and outputs the result to the corresponding point cloud extraction processing unit 154 . The RGB+spectral three-dimensional point cloud information is a three-dimensional point cloud that mixes RGB and spectral information.

[0177] In step S54, the corresponding point cloud extraction processing unit 154 extracts the spectral information existing within a predetermined distance associated with each point in the RGB three-dimensional point cloud based on the RGB+spectral three-dimensional point cloud information (three-dimensional point cloud information mixed with RGB and spectral information), and outputs the result to the spectral information mapping processing unit 155.

[0178] In step S55, the spectral information mapping processing unit 155 generates a spectral three-dimensional model by mapping the average value of spectral information within a predetermined distance, the weighted average value according to the distance, etc. as spectral information to each point in the RGB three-dimensional point cloud, wherein the spectral information is provided by the corresponding point cloud extraction processing unit 154 that has been extracted in association with each point in the RGB three-dimensional point cloud.

[0179] According to the above processing, a three-dimensional point cloud mixed with RGB and spectral information, i.e., RGB+spectral three-dimensional point cloud information, is generated based on the RGB image group and the spectral image group. Based on the RGB+spectral three-dimensional point cloud information, mapping is performed based on the spectral information extracted corresponding to each point in the RGB three-dimensional point cloud, thereby generating a spectral three-dimensional model.

[0180] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0181] <2-4. Fourth Configuration Example of First Embodiment>

[0182] In the above, an example is described in which a spectral three-dimensional model is generated by mapping based on spectral information extracted corresponding to each point of the RGB three-dimensional point cloud based on RGB+spectral three-dimensional point cloud information generated from an RGB image group and a spectral image group.

[0183] However, a 3D image group corresponding to each RGB image may be reconstructed based on the RGB image group, and spectral information of the spectral image group may be mapped to generate a spectral 3D model.

[0184] Fig. 9 It is a fourth configuration example of the first embodiment of the image processing device, in which a spectral three-dimensional model is generated by reconfiguring a three-dimensional image group corresponding to each RGB image based on an RGB image group and mapping spectral information of the spectral image group onto the three-dimensional image group.

[0185] Fig. 9 The image processing device 101 in the embodiment includes an RGB camera 171 , a spectral camera 172 , a three-dimensional reconfiguration unit 173 and a spectral information mapping processing unit 174 .

[0186] The RGB camera 171 and the spectral camera 172 correspond to the RGB camera 111 and the spectral camera 112 , respectively. The RGB camera 171 and the spectral camera 172 take RGB image groups and spectral image groups, respectively, and output the RGB image groups and spectral image groups to the three-dimensional reconfiguration unit 173 and the spectral information mapping processing unit 174 .

[0187] Here, it is assumed that the positional relationship between the RGB camera 171 and the spectral camera 172 is fixed, and information on the positional relationship between the RGB camera 171 and the spectral camera 172 is obtained in advance by stereo calibration or the like.

[0188] The three-dimensional reconfiguration unit 173 reconfigures an RGB three-dimensional image group corresponding to each RGB image group based on the RGB image group, and outputs the result to the spectrum information mapping processing unit 174 .

[0189] In addition, when generating an RGB three-dimensional image, the three-dimensional reconfiguration unit 173 can generate posture information of the RGB camera 171 based on the information of each of the generated RGB three-dimensional images, and therefore the posture information of the RGB camera 171 is also output to the spectral information mapping processing unit 174 in association with the RGB three-dimensional image.

[0190] The spectral information mapping processing unit 174 maps spectral information in the spectral image group to the 3D image group based on the RGB 3D image group and the corresponding camera pose information of the RGB camera 171 provided from the 3D reconfiguration unit 173 , thereby generating a spectral 3D model.

[0191] In other words, it is assumed here that the positional relationship between the RGB camera 171 and the spectral camera 172 is obtained in advance through stereo calibration. Therefore, the spectral information mapping processing unit 174 obtains the posture information of the spectral camera 172 based on the posture information of the RGB camera 171 for each RGB three-dimensional image, and maps the spectral information to the position of the spectral image and the position of the corresponding RGB three-dimensional image.

[0192] However, since an RGB three-dimensional image does not have a scale, it is necessary to form a three-dimensional model that matches the actual size of the captured object in order to design a method that can obtain scale information from an RGB image by including information that can be used as a scale in the RGB image captured by the RGB camera 171.

[0193] In other words, for example, an RGB image or a spectral image may be captured such that a ruler or an object of known size falls within the field of view of the RGB camera 171 and the spectral camera 172 .

[0194] According to such a configuration, an RGB three-dimensional image group is generated from an RGB image group, and spectral information is mapped onto the RGB three-dimensional image based on the spectral image group, thereby generating a spectral three-dimensional model.

[0195] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0196] < Fig. 9 Spectral three-dimensional model generation processing of image processing device in>

[0197] Next, refer to Fig.10 Flowchart Description Fig. 9 The image processing device 101 performs a spectral three-dimensional model generation process.

[0198] In step S71 , the RGB camera 171 configures an RGB image group by capturing an object from multiple angles, and outputs the RGB image group to the three-dimensional reconfiguration unit 173 .

[0199] In step S72 , the spectral camera 172 forms a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the spectral information mapping processing unit 174 .

[0200] In step S73 , the three-dimensional reconfiguration unit 173 reconfigures the RGB three-dimensional point cloud information based on the RGB image groups, generates an RGB three-dimensional image group corresponding to each RGB image group, and outputs the result to the spectral information mapping processing unit 174 .

[0201] Furthermore, when generating an RGB three-dimensional image, the three-dimensional reconfiguration unit 173 may generate posture information of the RGB camera 171 from information of each of the generated RGB three-dimensional images, and thus the posture information of the RGB camera 171 is also output to the spectral information mapping processing unit 174 .

[0202] In step S74 , the spectral information mapping processing unit 174 resets the counter f for counting the number of frames of the RGB three-dimensional image group and the spectral image group to 1.

[0203] In step S75 , the spectral information mapping processing unit 174 maps the spectral information of the spectral image of the corresponding frame f to each position where the RGB information is located in the 3D image of frame f according to the RGB 3D image of frame f and the posture information of the RGB camera 171 .

[0204] In other words, it is assumed here that the positional relationship between the RGB camera 171 and the spectral camera 172 is obtained in advance through stereo calibration, etc. Therefore, the spectral information mapping processing unit 174 obtains the posture information of the spectral camera 172 based on the posture information of the RGB camera 171 in the RGB three-dimensional image of the frame f, and maps the spectral information in the spectral image to the corresponding position in the RGB three-dimensional image.

[0205] At this time, for example, by designing the RGB camera 171 to capture an RGB image in which a ruler, an object of known size, or the like appears in the image, the spectral information mapping processing unit 174 generates a three-dimensional image in which the spectral information is mapped, including information indicating the positional relationship of each pixel in the RGB three-dimensional image in three-dimensional space, such as a scale.

[0206] In step S76 , the spectrum information mapping processing unit 174 determines whether the counter f is lower than the frame number N, and if the counter f is lower than the frame number N, the sequence proceeds to step S77 .

[0207] In step S77 , the spectrum information mapping processing unit 174 increases the counter f by 1, and the sequence returns to step S75 .

[0208] That is, the processing of steps S75 to S77 is repeated until the spectral information is mapped for all N frames, which is the number of images included in the RGB three-dimensional image group and the spectral image group.

[0209] If it is determined in step S76 that the counter f is not lower than N and it is thus determined that the spectral information has been mapped to all three-dimensional image groups, the sequence moves to step S78.

[0210] In step S78 , the spectral information mapping processing unit 174 generates and outputs a spectral three-dimensional model based on the three-dimensional image group to which the spectral information has been mapped.

[0211] According to such processing, an RGB three-dimensional image group is generated from the RGB image group, and spectral information is mapped on the RGB three-dimensional image based on the spectral image group, thereby generating a spectral three-dimensional model.

[0212] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0213] <2-5. Fifth Configuration Example of First Embodiment>

[0214] In the above, an example has been described in which a three-dimensional image group corresponding to each RGB image is reconstructed from an RGB image group, and spectral information of the spectral image group is mapped to generate a spectral three-dimensional model.

[0215] However, it is also possible to obtain a three-dimensional image group consisting of range images from a range camera instead of an RGB camera, and map the spectral information of the spectral image group to generate a spectral three-dimensional model.

[0216] Fig.11 It is a fifth configuration example of the first embodiment of the image processing device provided with a range camera instead of an RGB camera, and generates a spectral three-dimensional model by supplying a three-dimensional image group corresponding to a range image and mapping spectral information of a spectral image group onto the three-dimensional image group.

[0217] Fig.11 The image processing device 101 in the embodiment includes a range camera 191 , a spectral camera 192 , a camera posture estimation unit 193 , a spectral information mapping processing unit 194 and a three-dimensional reconfiguration unit 195 .

[0218] The range camera 191 is a so-called "depth camera" and is provided in place of the RGB camera 111. The range camera 191 captures a plurality of range images including a subject while changing the angle, and outputs the range images to the camera attitude estimation unit 193 and the spectral information mapping processing unit 194 as a three-dimensional image group.

[0219] The spectral camera 192 is equivalent to the spectral camera 112 that captures a spectral image group and outputs the spectral image group to the spectral information mapping processing unit 194 .

[0220] Here, it is assumed that the positional relationship between the range camera 191 and the spectrum camera 192 is fixed, and information indicating the positional relationship between the range camera 191 and the spectrum camera 192 is obtained in advance by stereo calibration or the like.

[0221] Furthermore, if the light source used for image capture by the range camera 191 affects image capture by the spectrum camera 192 , for example, if the light source is an infrared light source, etc., it is necessary to offset the image capture timings of the range camera 191 and the spectrum camera 192 from each other.

[0222] Camera pose estimation section 193 estimates the pose of distance camera 191 for each 3D image in the 3D image group based on information about the distance to the subject of the 3D image, associates and adds the estimation result with the 3D image, and outputs it to 3D reconfiguration section 195 .

[0223] The spectral information mapping processing unit 194 maps the spectral information of the spectral image group to the corresponding three-dimensional image group supplied from the range camera 191 , and generates a spectral information mapping image group.

[0224] In other words, it is assumed here that the positional relationship between the range camera 191 and the spectral camera 192 is obtained in advance through calibration. Therefore, the spectral information mapping processing unit 194 maps the spectral information to the position in the spectral image and the corresponding position in the three-dimensional image based on the positional relationship between the range camera 191 and the spectral camera 192 for each three-dimensional image, and generates a spectral information mapping image group (spectral information mapping three-dimensional image group).

[0225] The three-dimensional reconfiguration unit 195 reconfigures and outputs the spectral three-dimensional model based on: the spectral information mapping image group (spectral information mapping three-dimensional image group) provided from the spectral information mapping processing unit 194; and the posture estimation result of the distance camera 191, which is added to each three-dimensional image group provided from the camera posture estimation unit 193.

[0226] According to such a configuration, a spectral three-dimensional model can be generated by acquiring a three-dimensional image group composed of range images from a range camera and mapping spectral information of the spectral image group.

[0227] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0228] < Fig.11 Spectral three-dimensional model generation processing of image processing device in>

[0229] Next, refer to Fig.12 Flowchart Description Fig. 9 The image processing device 101 performs a spectral three-dimensional model generation process.

[0230] In step S91 , the range camera 191 forms a three-dimensional image group consisting of multiple range images by capturing range images of the subject from multiple angles, and outputs the three-dimensional image group to the camera posture estimation unit 193 and the spectral information mapping processing unit 194 .

[0231] In step S92 , the spectral camera 192 forms a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the spectral information mapping processing unit 194 .

[0232] In step S93 , the camera pose estimation unit 193 estimates the pose of the range camera 191 based on the three-dimensional image group, and outputs pose information of the range camera 191 as a result of the estimation to the three-dimensional reconfiguration unit 195 in association with the three-dimensional image.

[0233] In step S94 , the spectral information mapping processing unit 194 resets the counter f for counting the frame numbers of the three-dimensional image group and the spectral image group, respectively, to 1.

[0234] In step S95, the spectral information mapping processing unit 194 maps the spectral information of the corresponding spectral image to the positions where the distance information of the three-dimensional image of frame f is located based on the information of the positional relationship between the distance camera 191 and the spectral camera 192, and outputs the spectral information mapping image group (spectral information mapping three-dimensional image group) to the three-dimensional reconfiguration unit 195.

[0235] In step S96 , the three-dimensional reconfiguration unit 195 determines whether the counter f is below the frame number N, and if the counter f is below the frame number N, the sequence moves to step S97 .

[0236] In step S97 , the three-dimensional reconfiguration unit 195 increases the counter f by 1, and the sequence returns to step S95 .

[0237] In other words, the processes of steps S95 to S97 are repeated until the spectral information is mapped for the number of images included in each of the three-dimensional image group and the spectral image group, that is, all N frames.

[0238] If it is determined in step S96 that the counter f is not lower than N and it is thus determined that the spectral information has been mapped to all three-dimensional image groups, the sequence moves to step S98.

[0239] In step S98 , the three-dimensional reconfiguration unit 195 generates and outputs a spectral three-dimensional model based on the three-dimensional image group after the spectrum information is mapped and the pose information of the range camera 191 provided from the camera pose estimation unit 193 .

[0240] Through the above processing, a three-dimensional image group consisting of a range image group taken by a range camera and the posture information of the range camera can be obtained, and the spectral information of the spectral image group is mapped to the obtained three-dimensional image group to generate a spectral information mapping image, and a spectral three-dimensional model is generated based on the spectral information mapping image and the posture information of the range camera.

[0241] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0242] <2-6. Sixth Configuration Example of First Embodiment>

[0243] In the above, an example has been described in which a spectral three-dimensional model is generated by acquiring a three-dimensional image group consisting of range images from a range camera and mapping spectral information of the spectral image group to the acquired three-dimensional image group.

[0244] However, a spectral three-dimensional model can also be generated by generating a spectral information mapping image group from an RGB image group and a spectral image group; generating an RGB three-dimensional model and posture information of the RGB camera from the RGB image group, the RGB three-dimensional model being a three-dimensional model in RGB; and mapping the spectral information onto the RGB three-dimensional model.

[0245] Fig.13 It is the sixth configuration example of the first embodiment of the image processing device, wherein a spectral three-dimensional model is generated by generating a spectral information mapping image group from an RGB image group and a spectral image group; an RGB three-dimensional model and posture information of the RGB camera are generated from the RGB image group, the RGB three-dimensional model being a three-dimensional model in RGB; and the spectral information is mapped onto the RGB three-dimensional model.

[0246] Fig.13The image processing device 101 includes an RGB camera 211, a spectral camera 212, a three-dimensional reconfiguration unit 213, a spectral information mapping processing unit 214, a feature point matching processing unit 215, a pixel-by-pixel registration processing unit 216 and a mapping processing unit 217.

[0247] The RGB camera 211 corresponds to the RGB camera 111 . The RGB camera 111 configures an RGB image group by capturing an object from multiple angles, and outputs the RGB image group to the three-dimensional reconfiguration unit 213 and the feature point matching processing unit 215 .

[0248] The spectral camera 212 corresponds to the spectral camera 112 . The spectral camera 212 forms a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the feature point matching processing unit 215 .

[0249] The feature point matching processing unit 215, the pixel-by-pixel registration processing unit 216, and the mapping processing unit 217 correspond to the feature point matching processing unit 113, the pixel-by-pixel registration processing unit 114, and the mapping processing unit 115, respectively. These units generate a spectral information mapping image group based on the RGB image group and the spectral image group, and output their results to the spectral information mapping processing unit 214.

[0250] The three-dimensional reconfiguration unit 213 generates RGB three-dimensional point cloud information from the RGB image group, generates an RGB three-dimensional model as a three-dimensional model in RGB based on the three-dimensional point cloud information, generates posture information of the RGB camera for each RGB image, and outputs the result to the spectral information mapping processing unit 214.

[0251] The spectral information mapping processing unit 214 generates and outputs a spectral three-dimensional model based on the RGB three-dimensional model, the posture information of the RGB camera 211 , and the spectral information mapping image group.

[0252] Specifically, the spectral information mapping processing unit 214 maps the spectral information in the spectral information mapping image corresponding to the posture information of the RGB camera 211 to the RGB three-dimensional model, thereby generating a spectral three-dimensional model.

[0253] According to this structure, a spectral information mapping image group can be generated based on an RGB image group and a spectral image group, thereby generating a spectral three-dimensional model; an RGB three-dimensional model and posture information of the RGB camera are generated from the RGB image group, and the RGB three-dimensional model is a three-dimensional model in RGB; and the spectral information is mapped onto the RGB three-dimensional model.

[0254] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0255] < Fig.13 Spectral three-dimensional model generation processing of image processing device in>

[0256] Next, refer to Fig.14 Flowchart Description Fig.13 The image processing device 101 performs a spectral three-dimensional model generation process.

[0257] In step S111 , the RGB camera 211 configures an RGB image group by photographing a subject from a plurality of angles, and outputs the RGB image group to the three-dimensional reconfiguration unit 213 and the feature point matching processing unit 215 .

[0258] In step S112 , the spectral camera 212 configures a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the feature point matching processing unit 215 .

[0259] In step S113 , the feature point matching processing unit 215 resets the counter f for counting the number of frames of the RGB image group and the spectral image group to 1.

[0260] In step S114, the feature point matching processing unit 215 extracts feature points from each of the RGB image and the spectral image of frame number f, matches the extracted feature points, and outputs information about the matched feature points together with the RGB image group and the spectral image group of frame number f to the pixel-by-pixel registration processing unit 216.

[0261] In step S115, the pixel-by-pixel registration processing unit 216 generates registration information by associating the pixels of the feature points in the RGB image with the pixels of the feature points in the spectral image matched and provided from the feature point matching processing unit 215 through registration processing, and outputs the registration information together with the RGB image group and the spectral image group to the mapping processing unit 217.

[0262] In step S116, the mapping processing unit 217 maps the spectral information of the spectral image registered correspondingly to each pixel of the RGB image based on the registration information provided from the pixel-by-pixel alignment processing unit 216, thereby generating a spectral information mapping image that maps the spectral information to each pixel of the RGB image of frame f, and outputs the spectral information mapping image as the spectral information mapping image of frame f to the spectral information mapping processing unit 214.

[0263] At this time, the spectrum information mapping processing unit 214 stores the spectrum information mapping image of the frame f in correspondence with the frame f.

[0264] In step S117 , the feature point matching processing unit 215 determines whether the counter f is lower than the frame number N, and if the counter f is lower than the frame number N, the sequence moves to step S118 .

[0265] In step S118 , the feature point matching processing unit 215 increases the counter f by 1, and the sequence returns to step S114 .

[0266] That is, the processing of steps S114 to S118 is repeated until spectral information mapping images are generated for all N frames, which is the number of images included in each of the RGB image group and the spectral image group.

[0267] If it is determined in step S117 that the counter f is not lower than N and it is therefore determined that all spectral information map images have been generated, the sequence moves to step S119 .

[0268] In step S119 , the three-dimensional reconfiguration unit 213 reconfigures the RGB three-dimensional model by configuring the RGB three-dimensional point cloud information based on the RGB image group, and outputs the result to the spectral information mapping processing unit 214 .

[0269] In addition, when generating the RGB three-dimensional model, the three-dimensional reconfiguration unit 213 also generates posture information of the RGB camera 211 in the RGB image group, associates the posture information with each of the RGB images, and outputs the result to the spectral information mapping processing unit 214.

[0270] In step S120, the spectral information mapping processing unit 214 maps the spectral information in the spectral information mapping image corresponding to the posture information of the RGB camera 211 to the RGB 3D model according to the RGB 3D model, the posture information of the RGB camera 211 and the spectral information mapping image group, thereby generating a spectral 3D model.

[0271] Through the above processing, a spectral information mapping image group can be generated according to the RGB image group and the spectral image group to generate a spectral three-dimensional model; an RGB three-dimensional model and the posture information of the RGB camera are generated from the RGB image group, and the RGB three-dimensional model is a three-dimensional model in RGB; and the spectral information of the spectral information mapping image group is mapped to the RGB three-dimensional model based on the posture information of the RGB camera.

[0272] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0273] <2-7. Seventh Configuration Example of First Embodiment>

[0274] The above describes an example of generating a spectral three-dimensional model by generating a spectral information mapping image group based on an RGB image group and a spectral image group; generating an RGB three-dimensional model and posture information of an RGB camera from the RGB image group, the RGB three-dimensional model being a three-dimensional model in RGB; and mapping the spectral information onto the RGB three-dimensional model.

[0275] However, the range image group and posture information of all cameras can be reconfigured according to the RGB image group and the spectral image group, a three-dimensional point cloud of RGB+spectral information can be generated according to the range image group of all cameras, and a three-dimensional model of RGB+spectral information can be generated.

[0276] Then, according to the above-mentioned distance image group and the above-mentioned posture information, the spectral information in the above-mentioned spectral image group is mapped to the above-mentioned RGB image group, thereby generating a spectral information mapping image group, and according to the information in the above-mentioned spectral information mapping image group, the above-mentioned spectral information is mapped to the above-mentioned RGB+spectral information three-dimensional model, thereby generating a spectral three-dimensional model.

[0277] Fig.15 The seventh configuration example of the first embodiment of the image processing device is shown, wherein a spectral three-dimensional model is generated by reconfiguring the distance image group and posture information of all cameras from the RGB image group and the spectral image group, the three-dimensional model of the RGB+spectral information is generated from the distance image group of all cameras, a spectral information mapping image group is generated by mapping the spectral information in the spectral image group to each RGB image group from the distance image group and the posture information, and, based on the information of the spectral information mapping image group, the spectral information is mapped to the three-dimensional model of the RGB+spectral information.

[0278] Fig.15 The image processing device 101 in the embodiment includes an RGB camera 231 , a spectral camera 232 , a three-dimensional reconfiguration unit 233 , a mapping processing unit 234 , and a spectral information mapping processing unit 235 .

[0279] The RGB camera 231 and the spectral camera 232 correspond to the RGB camera 111 and the spectral camera 112 , respectively. The RGB camera 231 and the spectral camera 232 capture an RGB image group and a spectral image group, respectively, and output the image groups to the three-dimensional reconfiguration unit 233 .

[0280] The three-dimensional reconfiguration unit 233 generates a three-dimensional model of RGB+spectral information by reconstructing the distance image groups and posture information of all cameras based on the RGB image group and the spectral image group and generating three-dimensional point cloud information of RGB+spectral information from the distance image groups of all cameras, and outputs the three-dimensional model to the spectral information mapping processing unit 235.

[0281] The three-dimensional reconfiguration unit 233 also outputs the range image group and the posture information of all cameras generated when generating the three-dimensional model of RGB+spectral information to the mapping processing unit 234 together with the RGB image group and the spectral image group.

[0282] The mapping processing unit 234 maps the spectral information in the spectral image to the RGB image to generate a spectral information mapping image according to the range image group and the posture information of all cameras, and outputs the result as a spectral information mapping image group to the spectral information mapping processing unit 235 .

[0283] The spectral information mapping processing unit 235 maps the spectral information onto the three-dimensional model composed of RGB+spectral information supplied from the three-dimensional reconfiguration unit 233 based on the spectral information mapping image group supplied from the mapping processing unit 234 , thereby generating a spectral three-dimensional model.

[0284] According to such a structure, the distance image groups of all cameras can be reconfigured according to the RGB image group and the spectral image group to generate a spectral three-dimensional model, the three-dimensional model of RGB+spectral information is generated from the distance image groups of all cameras, a spectral information mapping image group is generated by mapping the spectral information in the spectral image group to each RGB image group according to the distance image group and the posture information, and, according to the information of the spectral information mapping image group, the spectral information is mapped to the three-dimensional model of RGB+spectral information.

[0285] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0286] < Fig.15 Spectral three-dimensional model generation processing of image processing device in>

[0287] Next, refer to Fig.16 Flowchart Description Fig.15 The image processing device 101 performs a spectral three-dimensional model generation process.

[0288] In step S131 , the RGB camera 231 configures an RGB image group by photographing a subject from a plurality of angles, and outputs the RGB image group to the three-dimensional reconfiguration unit 233 .

[0289] In step S132 , the spectral camera 232 configures a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the three-dimensional reconfiguration unit 233 .

[0290] In step S133, the three-dimensional reconfiguration unit 233 reconstructs the distance image group of all cameras based on the RGB image group and the spectral image group, generates three-dimensional point cloud information of RGB+spectral information from the distance image group of all cameras, and outputs the information as a three-dimensional model of RGB+spectral information to the spectral information mapping processing unit 235.

[0291] The three-dimensional reconfiguration unit 233 also outputs the posture information and the distance image group of all cameras obtained when generating the three-dimensional model of RGB+spectral information based on the RGB image group and the spectral image group to the mapping processing unit 234 together with the RGB image group and the spectral image group.

[0292] In step S134 , the mapping processing unit 234 resets the counter f for counting the frame number of the distance image group to 1.

[0293] In step S135 , the mapping processing unit 234 maps the spectral information of the corresponding spectral image to each position of the RGB image of frame f according to the range image of frame f and the posture information of the RGB camera 231 and the spectral camera 232 , and outputs the result as a spectral information mapping image.

[0294] In step S136 , the mapping processing unit 234 determines whether the counter f is below the frame number N, and if the counter f is below the frame number N, the sequence moves to step S137 .

[0295] In step S137 , the spectrum information mapping processing unit 234 increases the counter f by 1, and the sequence returns to step S135 .

[0296] In other words, the processes of steps S135 to S137 are repeated until the spectrum information is mapped for the number of images included in each of the RGB image group and the spectral image group, that is, all N frames.

[0297] If it is determined in step S136 that the counter f is not lower than N and it is thus determined that the spectral information has been mapped to all RGB image groups, the sequence moves to step S138 .

[0298] In step S138 , the spectral information mapping processing unit 235 maps the spectral information onto a three-dimensional model composed of RGB+spectral information based on the spectral information mapping image group, thereby generating a spectral three-dimensional model and outputting the spectral three-dimensional model.

[0299] Through the above processing, the distance image groups of all cameras can be reconstructed according to the RGB image group and the spectral image group to generate a spectral three-dimensional model, the three-dimensional model of RGB+spectral information can be generated from the distance image groups of all cameras, the posture information of all cameras is generated, and a spectral information mapping image group is generated by mapping the spectral information in the spectral image group to each RGB image group from the distance image group and the posture information, and, according to the information of the spectral information mapping image group, the spectral information is mapped to the three-dimensional model of RGB+spectral information.

[0300] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0301] <2-8. Eighth Configuration Example of First Embodiment>

[0302] The above illustrates an example of first generating a three-dimensional model of RGB+spectral information from an RGB image group and a spectral image group, generating a distance image group and posture information of all cameras, mapping the spectral information in the spectral image group to each RGB image group from the distance image group and the posture information to generate a spectral information mapping image group; and mapping the spectral information to the three-dimensional model of RGB+spectral information based on the information of the spectral information mapping image group.

[0303] However, a true scaled spectral three-dimensional model can be generated by performing rescaling, wherein a range image group obtained based on three-dimensional reconfiguration of the RGB image group is compared with a true scaled range image group obtained by stereo matching between an RGB camera and a spectral camera; an unscaled three-dimensional model obtained from the RGB image group is rescaled into a true three-dimensional model; and then spectral information is mapped into the true three-dimensional model.

[0304] Fig.17 An eighth configuration example of the first embodiment of the image processing device is shown, wherein a real scaled spectral three-dimensional model is generated by performing rescaling, in which a range image group obtained from a three-dimensional reconfiguration based on an RGB image group is compared with a real scaled range image group obtained by stereo matching between an RGB camera and a spectral camera; an unscaled three-dimensional model based on the RGB image group is rescaled into a real scaled three-dimensional model; and spectral information is then mapped into the real three-dimensional model.

[0305] Fig.17 The image processing device 101 includes an RGB camera 251, a spectral camera 252, a three-dimensional reconfiguration unit 253, a rescaling processing unit 254, a spectral information mapping processing unit 255, a stereo calibration processing unit 256, a stereo matching processing unit 257 and a mapping processing unit 258.

[0306] RGB camera 251 and spectral camera 252 correspond to RGB camera 111 and spectral camera 112, respectively. RGB camera 251 and spectral camera 252 take RGB image groups and spectral image groups, respectively, and output these image groups to 3D reconfiguration unit 253, stereo calibration processing unit 256, and mapping processing unit 258.

[0307] The three-dimensional reconfiguration unit 253 generates an unscaled RGB three-dimensional model by reconfiguring the RGB three-dimensional point cloud information based on the RGB image group, and outputs the result to the rescaling processing unit 254 .

[0308] The three-dimensional reconfiguration unit 253 also outputs the unscaled range image group and the posture information of the RGB camera 231 generated when the RGB three-dimensional model is generated to the rescaling processing unit 254 .

[0309] Note that the range image group generated by the three-dimensional reconfiguration unit 253 is composed of range images without scale, and therefore can be said to be essentially a three-dimensional image group composed of RGB three-dimensional images without scale information.

[0310] The rescaling processing unit 254 obtains the real scaled distance image group provided by the stereo matching processing unit 257 and the relative posture information of the RGB camera 251 and the spectral camera 252, and based on the comparison between the unscaled distance image group and the real scaled distance image group, rescales the unscaled three-dimensional model provided by the three-dimensional reconfiguration unit 253 into a real scaled three-dimensional model, and outputs the real scaled three-dimensional model to the spectral information mapping processing unit 255.

[0311] The stereo calibration processing unit 256 obtains relative posture information, which is information about the relative posture of the RGB camera device 251 and the spectral camera device 252, through stereo calibration based on the RGB image group and the spectral image group, and outputs the information together with the RGB image group and the spectral image group to the stereo matching processing unit 257.

[0312] The stereo matching processing unit 257 uses the relative posture information of the RGB camera 251 and the spectral camera 252 to generate a real scaled distance image group based on the RGB image group and the spectral image group through stereo matching, and outputs the real scaled distance image group together with the relative posture information of the RGB camera 251 and the spectral camera 252 to the rescaling processing unit 254 and the mapping processing unit 258.

[0313] The mapping processing unit 258 maps the spectral information of the spectral image group to the RGB image group according to the distance image group and the relative posture information of the RGB camera 251 and the spectral camera 252 , and outputs the result as a spectral information mapping image group to the spectral information mapping processing unit 255 .

[0314] The spectral information mapping processing unit 255 maps the spectral information onto the actual three-dimensional model supplied from the rescaling processing unit 254 based on the spectral information mapping image group supplied from the mapping processing unit 258 , thereby generating an actual spectral three-dimensional model.

[0315] According to such a structure, a real three-dimensional model is generated by rescaling the distance image group obtained by three-dimensional reconstruction based on the RGB image group and comparing it with the real scaled distance image group obtained by stereo matching of the RGB camera and the spectral camera, and the spectral information is mapped to the generated real three-dimensional model, thereby generating a real spectral three-dimensional model.

[0316] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0317] < Fig.17 Spectral three-dimensional model generation processing of image processing device in>

[0318] Next, we will refer to Fig.18 The flowchart in the description is Fig.17 The spectral three-dimensional model generation process is performed by the image processing device 101 in FIG.

[0319] In step S151 , the RGB camera 251 configures an RGB image group by photographing a subject from a plurality of angles, and outputs the RGB image group to the three-dimensional reconfiguration unit 253 , the stereo calibration processing unit 256 , and the mapping processing unit 258 .

[0320] In step S152 , the spectral camera 252 configures a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the stereo calibration processing unit 256 and the mapping processing unit 258 .

[0321] In step S153 , the three-dimensional reconfiguration unit 253 generates an unscaled RGB three-dimensional model by reconfiguring the RGB three-dimensional point cloud information based on the RGB image group, and outputs the result to the rescaling processing unit 254 .

[0322] The three-dimensional reconfiguration unit 253 also outputs the unscaled range image group and the posture information of the RGB camera 231 generated when the RGB three-dimensional model is generated to the rescaling processing unit 254 .

[0323] In step S154 , the stereo calibration processing unit 256 obtains relative posture information of the RGB camera 251 and the spectral camera 252 through stereo calibration based on the RGB image group and the spectral image group, and outputs the information together with the RGB image group and the spectral image group to the stereo matching processing unit 257 .

[0324] In step S155, the stereo matching processing unit 257 generates a real scaled distance image group based on the RGB image group and the spectral image group by performing stereo matching using the relative posture information of the RGB camera 251 and the spectral camera 252 as a result of stereo calibration, and outputs the real scaled distance image group together with the relative posture information of the RGB camera 251 and the spectral camera 252 to the rescaling processing unit 254 and the mapping processing unit 258.

[0325] In step S156, when obtaining the real scaled distance image group provided by the stereo matching processing unit 257, the relative posture information of the RGB camera 251 and the spectral camera 252, and the unscaled three-dimensional model, the distance image group and the posture information of the RGB camera provided by the three-dimensional reconfiguration unit 253, the rescaling processing unit 254 rescales the unscaled three-dimensional model into an actual-scale three-dimensional model by comparing the distance image group of the three-dimensional model without scaling with the actual-scale distance image group, and outputs the real-time three-dimensional model to the spectral information mapping processing unit 255.

[0326] In step S157 , the mapping processing unit 258 resets the counter f for counting the number of frames of the RGB image group and the spectral image group to 1.

[0327] In step S158 , the mapping processing unit 258 maps the spectral information of the spectral image to each position of the RGB image of the frame f based on the real-time range image and the relative posture information, and outputs the result as a spectral information mapping image.

[0328] In step S159 , the mapping processing unit 258 determines whether the counter f is below the frame number N, and if the counter f is below the frame number N, the sequence moves to step S160 .

[0329] In step S160 , the mapping processing unit 258 increases the counter f by 1, and the sequence returns to step S158 .

[0330] That is, the processing of steps S158 to S160 is repeatedly performed until the spectral information is mapped for all N frames, which is the number of images included in the RGB image group and the spectral image group.

[0331] If, in step S159 , it is determined that the counter f is not lower than N and it is thus determined that the spectral information has been mapped to all RGB image groups, the sequence moves to step S161 .

[0332] In step S161 , the spectral information mapping processing unit 255 maps the spectral information onto the actual three-dimensional model supplied from the rescaling processing unit 254 based on the spectral information mapping image group supplied from the mapping processing unit 258 , thereby generating an actual spectral three-dimensional model.

[0333] Through the above processing, a real scaled three-dimensional model is generated by rescaling the distance image group obtained by three-dimensional reconstruction based on the RGB image group and comparing it with the real scaled distance image group obtained by stereo matching between the RGB camera and the spectral camera, and then the spectral information is mapped to the generated real scaled three-dimensional model, so that a real distance spectral three-dimensional model can be generated.

[0334] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0335] <2-9. Ninth Configuration Example of First Embodiment>

[0336] In addition, in the above example, the distance image group obtained by three-dimensionally reconstructing the RGB image group is compared with the actual distance image group obtained by stereo matching the RGB camera and the spectral camera, and rescaled to generate an actual distance three-dimensional model, and then the spectral information is mapped on the generated actual distance three-dimensional model.

[0337] However, the RGB camera and the spectral camera may be coaxially configured using, for example, a beam splitter or the like, and a spectral three-dimensional model may be generated by mapping spectral information of the spectral image to a three-dimensional model obtained by three-dimensional reconstruction from the RGB image group.

[0338] Fig.19 A ninth configuration example of the first embodiment of the image processing device is shown, wherein, for example, an RGB camera and a spectral camera are coaxially configured using a beam splitter, and a spectral three-dimensional model is generated by mapping spectral information of the spectral image to a three-dimensional model obtained by three-dimensional reconfiguration of the RGB image group.

[0339] Fig.19 The image processing device 101 in the embodiment includes an RGB camera 271 , a spectral camera 272 , a three-dimensional reconfiguration unit 273 and a spectral information mapping processing unit 274 .

[0340] The RGB camera 271 and the spectral camera 272 correspond to the RGB camera 111 and the spectral camera 112, respectively, but are also coaxially configured by using, for example, a beam splitter, etc. The RGB camera 271 and the spectral camera 272 capture an RGB image group and a spectral image group, respectively, and output these image groups to a three-dimensional reconfiguration unit 273 and a spectral information mapping processing unit 274.

[0341] The three-dimensional reconfiguration unit 273 generates an RGB three-dimensional model by reconfiguring RGB three-dimensional point cloud information as three-dimensional point cloud information in RGB based on the RGB image group, and outputs the RGB three-dimensional model together with the posture information of the RGB camera 271 to the spectral information mapping processing unit 274.

[0342] The spectral information mapping processing unit 274 maps the spectral information of the spectral image group on each point of the RGB three-dimensional point cloud using the RGB three-dimensional model as the RGB three-dimensional model and the posture information of the RGB camera of each RGB image constituting the RGB image group, thereby generating a spectral three-dimensional model.

[0343] In other words, in this example, the RGB image group and the spectral image group from the RGB camera 271 and the spectral camera 272 have the same optical axis, so that there is no need to align the image groups with each other. Therefore, the posture information of the RGB camera 271 can be regarded as the posture information of the spectral camera 272, making the mapping of the spectral information to the RGB three-dimensional model relatively easy and highly accurate.

[0344] According to this configuration, for example, an RGB camera and a spectral camera can be coaxially configured using a beam splitter or the like, and a spectral three-dimensional model can be generated by mapping spectral information of a spectral image to a three-dimensional model obtained by three-dimensionally reconstructing a group of RGB images.

[0345] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0346] < Fig.19 Spectral three-dimensional model generation processing of image processing device in>

[0347] Next, refer to Fig. 20 Flowchart Description Fig.19 The image processing device 101 performs a spectral three-dimensional model generation process.

[0348] In step S171 , the RGB camera 271 configures an RGB image group by photographing a subject from a plurality of angles, and outputs the RGB image group to the three-dimensional reconfiguration unit 273 .

[0349] In step S172 , the spectral camera 272 forms a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the spectral information mapping processing unit 274 .

[0350] In step S173, the three-dimensional reconfiguration unit 273 generates RGB three-dimensional point cloud information as an RGB three-dimensional model based on the RGB image group, where the RGB three-dimensional point cloud information is a three-dimensional point cloud in RGB, and outputs the RGB three-dimensional model together with the posture information of the RGB camera in each RGB image group to the spectral information mapping processing unit 274.

[0351] In step S174 , the spectral information mapping processing unit 274 reads out the spectral information of the corresponding spectral image based on the posture information of the RGB camera 271 , and maps it to the corresponding points in the three-dimensional point cloud constituting the RGB three-dimensional model, thereby generating a spectral three-dimensional model.

[0352] Such processing may, for example, coaxially configure an RGB camera and a spectral camera using a beam splitter or the like, and generate a spectral three-dimensional model by mapping spectral information of the spectral image to a three-dimensional model obtained by three-dimensional reconstruction from the RGB image group.

[0353] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0354] <2-10. Tenth Configuration Example of First Embodiment>

[0355] Furthermore, in the above example, the RGB camera and the spectral camera are coaxially configured using a beam splitter, for example, and the spectral 3D model is generated by mapping the spectral information of the spectral image to a 3D model obtained by 3D reconstruction of the RGB image group.

[0356] However, the RGB camera and the spectral camera can be coaxially configured using a beam splitter, for example, and a spectral information mapping image can be generated by first mapping the spectral information of the spectral image to the RGB image, and then mapping the spectral information to a three-dimensional model obtained by three-dimensional reconstruction of the RGB image group based on the spectral image mapping image group to generate a spectral three-dimensional model.

[0357] Fig.21 It is the tenth configuration example of the first embodiment of the image processing device, wherein the RGB camera and the spectral camera are coaxially configured using, for example, a beam splitter, and a spectral information mapping image is generated by first mapping the spectral information of the spectral image onto the RGB image, and then mapping the spectral information onto a three-dimensional model obtained by three-dimensionally reconfiguring the RGB image group based on the spectral image mapping image group to generate a spectral three-dimensional model.

[0358] Fig.21The image processing device 101 in the embodiment includes an RGB camera 291 , a spectral camera 292 , a three-dimensional reconfiguration unit 293 , a spectral information mapping processing unit 294 and a mapping processing unit 295 .

[0359] The RGB camera 291 and the spectral camera 292 correspond to the RGB camera 271 and the spectral camera 272 , respectively, and are coaxially configured, for example, by using a beam splitter or the like.

[0360] The RGB camera 291 captures an RGB image group and outputs the RGB image group to the three-dimensional reconfiguration unit 293 and the mapping processing unit 295 .

[0361] The spectral camera 292 captures a spectral image group and outputs the spectral image group to the mapping processing unit 295 .

[0362] The three-dimensional reconfiguration unit 293 generates an RGB three-dimensional model by reconfiguring RGB three-dimensional point cloud information as three-dimensional point cloud information in RGB based on the RGB image group, and outputs the RGB three-dimensional model together with the posture information of the RGB camera 291 to the spectral information mapping processing unit 294.

[0363] The mapping processing unit 295 generates a spectral information mapping image group in which spectral information is mapped to each RGB image based on the RGB image group and the spectral image group, and outputs the generated spectral information mapping image group to the spectral information mapping processing unit 294 .

[0364] The spectral information mapping processing unit 294 uses the RGB three-dimensional model as RGB three-dimensional point cloud information and the posture information of the RGB camera of each RGB image constituting the RGB image group to map the spectral information of the spectral information mapping image group on each point of the RGB three-dimensional point cloud, thereby generating a spectral three-dimensional model.

[0365] That is, in this example, the RGB image group and the spectral image group from the RGB camera 291 and the spectral camera 292 have the same optical axis, so there is no need to align the image groups. In addition, by mapping the spectral information into the RGB image group, a spectral information mapping image group is generated and mapped into the three-dimensional model, so that the mapping of the spectral information to the RGB three-dimensional model is relatively simple and accurate.

[0366] According to this structure, for example, a beam splitter or the like can be used to coaxially structure the RGB camera and the spectral camera. First, the spectral information of the spectral image is mapped on the RGB image to generate a spectral information mapping image. Then, based on the spectral image mapping image group, the spectral information is mapped on the three-dimensional model obtained in the three-dimensional reconstruction of the RGB image group to generate a spectral three-dimensional model.

[0367] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0368] < Fig.21 Spectral three-dimensional model generation processing of image processing device in>

[0369] Next, refer to Fig. 22 The flowchart in Fig.21 The spectral three-dimensional model generation process is performed by the image processing device 101 in FIG.

[0370] In step S191 , the RGB camera 291 configures an RGB image group by capturing an object from multiple angles, and outputs the RGB image group to the three-dimensional reconfiguration unit 293 and the mapping processing unit 295 .

[0371] In step S192 , the spectral camera 292 forms a spectral image group by photographing spectral images from a plurality of angles, and outputs the spectral image group to the mapping processing unit 295 .

[0372] In step S193, the three-dimensional reconfiguration unit 293 generates RGB three-dimensional point cloud information as an RGB three-dimensional model based on the RGB image group, where the RGB three-dimensional point cloud information is a three-dimensional point cloud in RGB, and outputs the RGB three-dimensional model together with the posture information of the RGB camera in each RGB image group to the spectral information mapping processing unit 294.

[0373] In step S194 , the mapping processing unit 295 resets the counter f for counting the number of frames of the RGB image group and the spectral image group to 1.

[0374] In step S195 , the mapping processing unit 295 maps the spectral information of the spectral image to each position of the RGB image of the frame f, and outputs the result as a spectral information mapped image.

[0375] In step S196, the mapping processing unit 295 determines whether the counter f is below the frame number N, and if the counter f is below the frame number N, the sequence moves to step S197.

[0376] In step S197 , the mapping processing unit 295 increases the counter f by 1, and the sequence returns to step S195 .

[0377] In other words, the processes of steps S195 to S197 are repeated until the spectrum information is mapped for the number of images included in each of the RGB image group and the spectral image group, that is, all N frames.

[0378] If it is determined in step S196 that the counter f is not lower than N and it is therefore determined that the spectral information has been mapped to all RGB image groups, the sequence moves to step S198.

[0379] In step S198 , the spectral information mapping processing unit 294 generates a spectral three-dimensional model by reading out spectral information of the corresponding spectral information mapping image based on the posture information of the RGB camera 291 and mapping the spectral information to corresponding points in the three-dimensional point cloud constituting the RGB three-dimensional model.

[0380] Through the above processing, for example, the RGB camera and the spectral camera can be coaxially structured using a beam splitter, etc., first, a spectral information mapping image is generated by mapping the spectral information of the spectral image to the RGB image, and then, the spectral information is mapped by a three-dimensional model obtained by three-dimensionally reconstructing the RGB image group based on the spectral image mapping image group to generate a spectral three-dimensional model.

[0381] As a result, a spectral three-dimensional image from a desired viewpoint position can be generated based on the spectral three-dimensional model.

[0382] <<3. Second Embodiment>>

[0383] In the above, an example of generating a spectral three-dimensional model from an RGB image group and a spectral image group has been described. In the examples given so far, a spectral three-dimensional model including the entire object of interest and its surroundings is generated.

[0384] However, usually, since the information really needed is the spectral three-dimensional model of the object of interest, a reconstruction operation of extracting only the spectral three-dimensional model of the object of interest from the spectral three-dimensional model extracted in the above example is also required.

[0385] That is, consider the following situation: in a situation such as Fig.23 In the image PI shown on the left side of , a spectral three-dimensional model of the object of interest SB corresponding to the statue of a person surrounded by a circle is required.

[0386] In the method described in the first embodiment, when extracting a spectral three-dimensional model from the image PI, a spectral three-dimensional model including both the target object SB and the surrounding environment AB is extracted.

[0387] Therefore, in the method described according to the first embodiment, the following reshaping operation is required: Fig.23 In the spectral three-dimensional model including the object of interest SB and the surrounding environment AB represented by the image PI on the left side of Fig.23 The image PSB on the right side shows a spectral three-dimensional model of the object of interest SB.

[0388] This reshaping often requires intense work from the perspective of expert working hours.

[0389] Therefore, the present disclosure enables extraction of only a spectral three-dimensional model of an object of interest by segmenting a spectral image using a spectrum, performing a masking process, and extracting only the object of interest, and then generating a spectral three-dimensional model through three-dimensional reconstruction.

[0390] <3-1. First Configuration Example of Second Embodiment>

[0391] Next, we will refer to Fig.24 A first configuration example of the second embodiment of the image processing device that generates a spectral three-dimensional model of an object of interest is described.

[0392] Fig.24 The image processing device in the method obtains an RGB image group and a spectral image group taken from multiple angles respectively, generates a spectral information mapping image by mapping spectral information from the RGB image and the spectral image taken at corresponding angles onto the RGB image, and then generates a spectral three-dimensional model of the object of interest based on the spectral information mapping image group.

[0393] More specifically, Fig.24 The image processing device 301 includes an RGB camera 311, a spectral camera 312, a feature point matching processing unit 313, a pixel-by-pixel registration processing unit 314, a mapping processing unit 315, an object of interest point cloud extraction processing unit 316 and a three-dimensional reconfiguration unit 317.

[0394] Fig.24 The RGB camera 311, the spectral camera 312, the feature point matching processing unit 313, the pixel-by-pixel registration processing unit 314, the mapping processing unit 315 and the three-dimensional reconfiguration unit 317 in the image processing device 301 have the same Figure 2 The RGB camera 111, the spectral camera 112, the feature point matching processing unit 113, the pixel-by-pixel registration processing unit 114, the mapping processing unit 115 and the three-dimensional reconfiguration unit 116 have substantially the same configuration, so their description will be appropriately omitted.

[0395] In other words, Fig.24 The image processing device 301 in Figure 2 The image processing device 101 in FIG. 1 is different in that an attention object point cloud extraction processing unit 316 is provided between the mapping processing unit 315 and the three-dimensional reconfiguration unit 317 .

[0396] The object of interest point cloud extraction processing unit 316 performs masking processing on each of the spectral information mapping image groups by performing image segmentation using the spectrum, extracts only the area to which the spectral information of the object of interest has been mapped, and outputs the result to the three-dimensional reconfiguration unit 317 as the object of interest spectral information mapping image group.

[0397] For example, if in Fig.23 In the image PI on the left, it is known that the object of interest SB, which is a human statue, is created from plaster. The object of interest point cloud extraction processing unit 316 generates a spectral information mapping image group of only the object of interest SB by performing masking processing on the area except the plaster using spectral image segmentation and extracting only the area of ​​the spectral information of the object of interest.

[0398] Through this, the three-dimensional reconfiguration unit 317 reconfigures the attention object spectral three-dimensional model by configuring three-dimensional point cloud information composed of spectral information of the attention object based on the attention object spectral information mapping image group.

[0399] According to this structure, it is possible to generate a spectral information mapping image group from the RGB image group and the spectral image group, and reconstruct a spectral three-dimensional model of the object of interest from the spectral information mapping image group of the object of interest that maps only the spectral information of the object of interest in the generated spectral information mapping image group.

[0400] As a result, based on the spectral three-dimensional model of the object of interest, a spectral three-dimensional image of only the object of interest from a desired viewpoint position can be generated.

[0401] < Fig.24 Spectral three-dimensional model generation process of an object of interest in an image processing device>

[0402] Next, we will refer to Fig.25 The flowchart in the description is Fig.24 The image processing device 301 in the present invention is used to generate a three-dimensional model of a spectrum of interest.

[0403] It should be noted that in Fig.25 In the flowchart of FIG. , the processing except step S217 is the same as Figure 4 The processing corresponds to the processing in step S11 to step S19 in , and therefore, its description is appropriately omitted.

[0404] In other words, once the spectral information map image of the frame f is generated from the RGB image group and the spectral image group through the processing of steps S211 to S216 , the sequence moves to step S217 .

[0405] In step S217 , the object of interest point cloud extraction processing unit 316 performs masking processing on the spectral information map image by performing image segmentation using the spectrum, extracts only the region of the object of interest, generates an object of interest spectral information map image, and outputs the image to the three-dimensional reconfiguration unit 317 .

[0406] At this time, the three-dimensional reconfiguration unit 317 stores the object of interest spectrum information map image of the frame f in association with the frame f.

[0407] Then, the processing of steps S214 to S219 is repeated until the attention-object spectral information map image is generated for each of the number of images included in the RGB image group and the spectral image group, that is, for all N frames.

[0408] In step S218 , when it is determined that the counter f is not less than N, and when it is determined that all the target spectrum information map images have been generated, the process proceeds to step S220 .

[0409] In step S220 , the three-dimensional reconfiguration unit 317 reconfigures the object of interest spectral three-dimensional model by configuring three-dimensional point cloud information composed of spectral information of the object of interest based on the object of interest spectral information mapping image group.

[0410] According to the above processing, a spectral information mapping image group can be generated from the RGB image group and the spectral image group, a spectral information mapping image group of interest can be generated from the generated spectral information mapping image group, and a spectral three-dimensional model of the object of interest can be reconstructed from the spectral information mapping image group of the object of interest.

[0411] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0412] In addition, in the above description, an example is described in which the spectral information mapping image group is segmented using the spectrum at a timing after the spectral information mapping image group is generated. However, the spectral image group may be segmented at a time after the spectral image group is generated by the spectral camera 312, and the spectral three-dimensional model (spectral three-dimensional point cloud) may be segmented at a time after the spectral three-dimensional model is generated.

[0413] Furthermore, image segmentation using a spectrum can be performed at a previous stage to limit processing to a region belonging to an object of interest, which makes it possible to reduce the overall processing load.

[0414] <3-2. Second Configuration Example of Second Embodiment>

[0415] In the above, examples are described of generating a spectral information mapping image group from an RGB image group and a spectral image group, generating an object spectral information mapping image group from the spectral information mapping image group, and generating an object spectral 3D model from the object spectral information mapping image group.

[0416] However, three-dimensional point clouds can also be generated from the RGB image group and the spectral image group respectively, and the spectral information can be mapped by aligning the position of the spectral three-dimensional point cloud with the position of the RGB three-dimensional point cloud, and only the point cloud of the object of interest can be extracted using the spectrum to generate a spectral three-dimensional model of the object of interest.

[0417] Fig.26A second construction example of a second embodiment of an image processing device is shown, wherein a spectral three-dimensional model of an object of interest is generated by generating three-dimensional point clouds for both an RGB image group and a spectral image group, mapping spectral information that has aligned the position of the spectral three-dimensional point cloud with the position of the RGB three-dimensional point cloud, and further extracting the point cloud of the object of interest.

[0418] Fig.26 The image processing device 301 includes an RGB camera 331, a spectral camera 332, three-dimensional reconfiguration units 333 and 334, a three-dimensional point cloud alignment processing unit 335, a corresponding point cloud extraction processing unit 336, a spectral information mapping processing unit 337, and an object of interest point cloud extraction processing unit 338.

[0419] Please note that Fig.26 The RGB camera 331 to the spectral information mapping processing unit 337 in the image processing device 301 have Figure 5 The RGB camera 131 to the spectral information mapping processing unit 137 in the image processing device 101 have the same configuration, so their description will be omitted.

[0420] In other words, Fig.26 The image processing device 301 in Figure 5 The image processing device 101 in FIG. 1 is different in that a focus object point cloud extraction processing unit 338 is added.

[0421] The object of interest point cloud extraction processing unit 338 performs point cloud segmentation using the spectrum of the point cloud to which the spectrum information is mapped, thereby extracting only the point cloud of the spectrum information of the object of interest, thereby generating an object of interest spectrum three-dimensional model.

[0422] For example, if it is known that Fig.23 In the image PI on the left, the object of interest SB as a human statue is created from plaster, and the object of interest point cloud extraction processing unit 338 generates a spectral three-dimensional model of the object of interest, which is constructed by extracting only the point cloud of the object of interest SB using spectral point cloud segmentation, and only extracting the point cloud of the spectral information of the object of interest as the plaster area.

[0423] According to this structure, an RGB three-dimensional point cloud and a spectral three-dimensional point cloud are generated based on the RGB image group and the spectral image group, and these point clouds are aligned and mapped based on the spectral information extracted corresponding to each point in the RGB three-dimensional point cloud. Then, the point cloud is segmented using the spectrum, thereby generating a spectral three-dimensional model of the object of interest consisting only of the point cloud of the object of interest.

[0424] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0425] < Fig.26 Spectral three-dimensional model generation process of an object of interest in an image processing device>

[0426] Next, we will refer to Fig. 27 Flowchart description in Fig.26 The image processing device 301 in the present invention is used to generate a three-dimensional model of a spectrum of interest.

[0427] It should be noted that Fig. 27 The processing of steps S231 to S237 in the flowchart of Figure 6 The processing of steps S31 to S37 in the flowchart of FIG. 1 is similar, so the description thereof will be omitted.

[0428] That is, after the spectral three-dimensional model is generated based on the RGB image group and the spectral image group through the processing of steps S231 to S237, the process proceeds to step S238.

[0429] In step S238 , the object of interest point cloud extraction processing unit 338 extracts only the point cloud of the spectral information of the object of interest by segmenting the point cloud using the spectrum of the spectral three-dimensional model composed of the point cloud to which the spectral information is mapped, thereby generating the object of interest spectral three-dimensional model.

[0430] Through such processing, RGB three-dimensional point clouds and spectral three-dimensional point clouds are generated from the RGB image group and the spectral image group, the point clouds are made consistent, and mapping is performed based on the spectral information extracted corresponding to each point in the RGB three-dimensional point cloud to generate a spectral three-dimensional model. Furthermore, based on the spectral three-dimensional model, point cloud information of the spectral information of the object of interest is extracted, thereby generating a spectral three-dimensional model of the object of interest.

[0431] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0432] The above description describes an example in which the spectral three-dimensional model (spectral three-dimensional point cloud) is segmented using the spectrum at a timing after the spectral three-dimensional model (spectral three-dimensional point cloud) is generated. However, the segmentation of the spectral three-dimensional point cloud may be performed at a timing immediately after the spectral three-dimensional point cloud is generated by the three-dimensional reconfiguration unit 334, or the segmentation of the spectral image group may be performed at a timing immediately after the spectral image group is generated by the spectral camera 332.

[0433] Furthermore, point cloud segmentation using spectrum may be performed at a previous stage to limit processing to the region belonging to the object of interest, which makes it possible to reduce the overall processing load.

[0434] <3-3. Third Configuration Example of Second Embodiment>

[0435] The above-described example of generating a spectral three-dimensional model of an object of interest is achieved by first obtaining a three-dimensional point cloud from each of the RGB image group and the spectral image group, aligning their positions, synthesizing the RGB three-dimensional point cloud and the spectral three-dimensional point cloud, and then mapping the spectral information onto the RGB three-dimensional point cloud to generate a spectral three-dimensional model, and further extracting the point cloud information of the spectral information of the object of interest.

[0436] However, RGB+spectral three-dimensional point cloud information that is a mixture of RGB three-dimensional point cloud and spectral three-dimensional point cloud can be obtained directly from the RGB image group and the spectral image group. The spectral three-dimensional model can be generated by mapping the spectral information to the RGB three-dimensional point cloud in the RGB+spectral three-dimensional point cloud information, and further extracting the point cloud information of the spectral information of the object of interest to generate the spectral three-dimensional model of the object of interest.

[0437] Fig.28 A third configuration example of the second embodiment of the image processing device is shown, wherein RGB+spectral three-dimensional point cloud information (wherein the RGB three-dimensional point cloud and the spectral three-dimensional point cloud are mixed) is directly acquired from an RGB image group and a spectral image group, a spectral three-dimensional model is generated by mapping the spectral information onto the RGB three-dimensional point cloud in the RGB+spectral three-dimensional point cloud information, a spectral three-dimensional model of an object of interest is generated, and point cloud information of the spectral information of the object of interest is further extracted.

[0438] Fig.28 The image processing device 301 includes an RGB camera 351, a spectral camera 352, a three-dimensional reconfiguration unit 353, a corresponding point cloud extraction processing unit 354, a spectral information mapping processing unit 355, and a focus object point cloud extraction processing unit 356.

[0439] Please note that Fig.28 The RGB camera 351 to the spectral information mapping processing unit 355 in the image processing device 301 have Figure 7 The RGB camera 151 to the spectral information mapping processing unit 155 in the image processing device 101 have the same configuration, so their description will be omitted.

[0440] In other words, Fig.28 The image processing device 301 in Figure 7 The image processing device 101 in FIG. 1 is different in that a focus object point cloud extraction processing unit 356 is added.

[0441] The object point cloud extraction processing unit 356 has Fig.26 The same structure as the object of interest point cloud extraction processing unit 338 is used to extract only the point cloud of the spectral information of the object of interest through point cloud segmentation using the spectrum on the point cloud to which the spectral information is mapped, thereby generating a spectral three-dimensional model of the object of interest.

[0442] Such a configuration makes it possible to generate a spectral three-dimensional model of the object of interest by generating RGB+spectral three-dimensional point cloud information, wherein the RGB+spectral three-dimensional point cloud information is a three-dimensional point cloud mixed with RGB and spectral information, and from the RGB image group and the spectral image group, based on the RGB+spectral three-dimensional point cloud information, mapping is performed based on the spectral information extracted from each point in the associated RGB three-dimensional point cloud to generate a spectral three-dimensional model, and the point cloud of the subject of interest is also extracted from the spectral three-dimensional model.

[0443] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0444] < Fig.28 Spectral three-dimensional model generation process of an object of interest in an image processing device>

[0445] Next, we will refer to Fig.29 The flowchart in the description is Fig.28 The image processing device 301 in the present invention is used to generate a three-dimensional model of a spectrum of interest.

[0446] It should be noted that Fig.29 The processing of steps S251 to S255 in the flowchart of Figure 8 The processing of steps S51 to S55 in the flowchart of FIG. 1 is similar, so their description will be omitted.

[0447] In other words, once the spectrum three-dimensional model is generated through the processing of steps S251 to S255 , the sequence moves to step S256 .

[0448] In step S256 , the object of interest point cloud extraction processing unit 356 uses the spectrum on the point cloud mapped with the spectral information constituting the spectral three-dimensional model to extract only the point cloud of the spectral information of the object of interest by point cloud segmentation, thereby generating the object of interest spectral three-dimensional model.

[0449] The above processing enables generation of a spectral three-dimensional model of the object of interest by generating RGB+spectral three-dimensional point cloud information, wherein the RGB+spectral three-dimensional point cloud information is a three-dimensional point cloud mixed with RGB and spectral information, and mapping is performed based on the RGB+spectral three-dimensional point cloud information and the spectral information extracted from each point in the associated RGB three-dimensional point cloud from the RGB image group and the spectral image group to generate a spectral three-dimensional model, and also point cloud information of only the spectral information of the object of interest is extracted from the spectral three-dimensional model.

[0450] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0451] <3-4. Fourth Configuration Example of Second Embodiment>

[0452] The above illustrates the following example: based on the RGB+spectral three-dimensional point cloud information generated by the RGB image group and the spectral image group, a spectral three-dimensional model is generated by mapping the spectral information extracted corresponding to each point of the RGB three-dimensional point cloud, and then only the point cloud information of the spectral information of the object of interest is extracted from the spectral three-dimensional model, thereby generating a spectral three-dimensional model of the object of interest.

[0453] However, the spectral three-dimensional model of the object of interest can be generated by reconfiguring the three-dimensional image group corresponding to each RGB image based on the RGB image group, and mapping the spectral information of the spectral image group to generate a spectral three-dimensional model, and further extracting only the point cloud information of the spectral information of the object of interest from the spectral three-dimensional model.

[0454] Fig.30 It is a fourth configuration example of the second embodiment of the image processing device, which generates a spectral three-dimensional model of an object of interest by reconfiguring a three-dimensional image group corresponding to each RGB image based on an RGB image group and mapping the spectral information of the spectral image group to generate a spectral three-dimensional model, and further extracting only point cloud information of the spectral information of the object of interest from the spectral three-dimensional model.

[0455] Fig.30 The image processing device 301 in the embodiment includes an RGB camera 371 , a spectral camera 372 , a three-dimensional reconfiguration unit 373 , a spectral information mapping processing unit 374 , and an object of interest point cloud extraction processing unit 375 .

[0456] Please note that Fig.30 The RGB camera 371 to the spectral information mapping processing unit 374 in the image processing device 301 have Fig. 9 The RGB camera 171 to the spectral information mapping processing unit 174 in the image processing device 101 have the same configuration, so their description will be omitted.

[0457] In other words, Fig.30 The image processing device 301 in Fig. 9 The image processing device 101 in FIG. 1 is different in that a focus object point cloud extraction processing unit 375 is added.

[0458] The object point cloud extraction processing unit 375 has Fig.26 The same structure as the object of interest point cloud extraction processing unit 338 is used to extract only the point cloud of the spectral information of the object of interest through point cloud segmentation using the spectrum on the point cloud mapped with the spectral information, so as to generate a spectral three-dimensional model of the object of interest.

[0459] According to this structure, an RGB three-dimensional image group is generated based on the RGB image group, spectral information is mapped on the RGB three-dimensional image based on the spectral image group to generate a spectral three-dimensional model, and then a point cloud of the object of interest is extracted from the spectral three-dimensional model, thereby generating a spectral three-dimensional model of the object of interest.

[0460] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0461] < Fig.30 Spectral three-dimensional model generation process of an object of interest in an image processing device>

[0462] Next, we will refer to Fig.31 The flowchart in the description is Fig.30 The image processing device 301 in the present invention is used to generate a three-dimensional model of a spectrum of interest.

[0463] It should be noted that Fig.31 The processing of steps S271 to S278 in the flowchart of Fig.10 The processing of steps S71 to S78 in the flowchart of FIG. 1 is similar, so their description will be omitted.

[0464] In other words, once the spectrum three-dimensional model is generated through the processing of steps S271 to S278 , the sequence moves to step S279 .

[0465] In step S279 , the attention object point cloud extraction processing unit 375 generates an attention object spectral three-dimensional model by extracting only the point cloud of the spectral information of the attention object using point cloud segmentation of the spectrum on the point cloud on which the spectral information constituting the spectral three-dimensional model is mapped.

[0466] Through such processing, an RGB three-dimensional image group is generated from the RGB image group, and spectral information is mapped to the RGB three-dimensional image based on the spectral image group to generate a spectral three-dimensional model. Furthermore, a point cloud of the object of interest is extracted from the spectral three-dimensional model, thereby generating a spectral three-dimensional model of the object of interest.

[0467] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0468] <3-5. Fifth Configuration Example of Second Embodiment>

[0469] In addition, in the above description, the following example is described: a three-dimensional image group corresponding to each RGB image is reconstructed based on the RGB image group, and the spectral information of the spectral image group is mapped to generate a spectral three-dimensional model, and then, a point cloud of the object of interest is extracted from the spectral three-dimensional model, thereby generating a spectral three-dimensional model of the object of interest.

[0470] However, it is also possible to obtain a three-dimensional image group consisting of range images from a range camera instead of an RGB camera, map the spectral information of the spectral image group to generate a spectral three-dimensional model, and then extract a point cloud of the object of interest from the spectral three-dimensional model to generate a spectral three-dimensional model of the object of interest.

[0471] Fig.32 It is a fifth configuration example of the second embodiment of the image processing device in which a range camera is set instead of an RGB camera, and a spectral three-dimensional model of an object of interest is generated by obtaining a three-dimensional image group from a range image group captured by the range camera and mapping the spectral information of the spectral image group to generate a spectral three-dimensional model, and further extracting a point cloud of the object of interest from the spectral three-dimensional model.

[0472] Fig.32 The distance camera 391 to the three-dimensional reconfiguration unit 395 in the image processing device 301 has Fig.11 The image processing device 101 has the same configuration from the distance camera 191 to the three-dimensional reconfiguration unit 195, so its description will be omitted.

[0473] In other words, Fig.32 The image processing device 301 in Fig.11 The difference between the image processing device 101 in FIG. 1 and FIG. 2 is that a focus object point cloud extraction processing unit 396 is added.

[0474] The object cloud extraction processing unit 396 has Fig.26 The object of interest cloud extraction processing unit 338 has the same structure as that of the object of interest, and uses the spectrum on the point cloud mapped with the spectral information constituting the spectral three-dimensional model to extract only the point cloud of the spectral information of the object of interest through point cloud segmentation to generate the object of interest spectral three-dimensional model.

[0475] According to this structure, a spectral three-dimensional model is generated by obtaining a three-dimensional image group consisting of range images from a range camera, mapping the spectral information of the spectral image group, and then extracting a point cloud of the object of interest from the spectral three-dimensional model, thereby generating a spectral three-dimensional model of the object of interest.

[0476] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0477] < Fig.32 Spectral three-dimensional model generation process of an object of interest in an image processing device>

[0478] Next, we will refer to Fig.33 The flowchart in the description is Fig.32The image processing device 301 in the present invention is used to generate a three-dimensional model of a spectrum of interest.

[0479] It should be noted that Fig.33 The processing of steps S291 to S298 in the flowchart of Fig.12 The processing of steps S91 to S98 in the flowchart of FIG. 1 is similar, so their description will be omitted.

[0480] In other words, once the spectrum three-dimensional model is generated through the processing of steps S291 to S298 , the sequence moves to step S299 .

[0481] In step S299 , the object of interest point cloud extraction processing unit 396 extracts only the point cloud of the spectral information of the object of interest by segmenting the point cloud using the spectrum on which the spectral information constituting the spectral three-dimensional model is mapped, thereby generating the object of interest spectral three-dimensional model.

[0482] Through the above processing, a three-dimensional image group composed of range images from a range camera can be obtained, and the spectral information of the spectral image group is mapped to generate a spectral three-dimensional model. Furthermore, the point cloud of the object of interest is extracted from the spectral three-dimensional model, thereby generating a spectral three-dimensional model of the object of interest.

[0483] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0484] <3-6. Sixth Configuration Example of Second Implementation>

[0485] In addition, in the above description, the following example is described: a three-dimensional image group consisting of range images from a range camera is obtained, the spectral information of the spectral image group is mapped to the obtained three-dimensional image group to generate a spectral three-dimensional model, and then, a point cloud of an object of interest is extracted from the spectral three-dimensional model, thereby generating a spectral three-dimensional model of the object of interest.

[0486] However, a spectral three-dimensional model of the object of interest can also be generated by generating a spectral information mapping image group from an RGB image group and a spectral image group; generating an RGB three-dimensional model and posture information of the RGB camera from the RGB image group, the RGB three-dimensional model being a three-dimensional model in RGB; mapping the spectral information from the spectral information mapping image group onto the RGB three-dimensional model to generate a spectral three-dimensional model; and also extracting a point cloud of the subject of interest from the spectral three-dimensional model.

[0487] Fig.34It is the sixth configuration example of the second embodiment of the image processing device for generating a spectral three-dimensional model of an object of interest by generating a spectral information mapping image group from an RGB image group and a spectral image group; generating an RGB three-dimensional model and posture information of the RGB camera from the RGB image group, the RGB three-dimensional model being a three-dimensional model in RGB; mapping spectral information from the spectral information mapping image group onto the RGB three-dimensional model to generate a spectral three-dimensional model; and also extracting a point cloud of the subject of interest from the spectral three-dimensional model.

[0488] Fig.34 The image processing device 301 includes an RGB camera 411, a spectral camera 412, a three-dimensional reconfiguration unit 413, a spectral information mapping processing unit 414, a feature point matching processing unit 415, a pixel-by-pixel registration processing unit 416, a mapping processing unit 417 and a focus object point cloud extraction processing unit 418.

[0489] Please note that Fig.34 The RGB camera 411 in the image processing device 301 to the mapping processing unit 417 has Fig.13 The RGB camera 211 to the mapping processing unit 217 in the image processing device 101 have the same configuration, so the description thereof will be omitted.

[0490] In other words, Fig.34 The image processing device 301 in Fig.13 The image processing device 101 in FIG. 1 is different in that an attention object point cloud extraction processing unit 418 is added.

[0491] The object point cloud extraction processing unit 418 has Fig.26 The same structure as the object of interest point cloud extraction processing unit 338 is used to map the spectrum on the point cloud of the spectral information constituting the spectral three-dimensional model, and only the point cloud of the spectral information of the object of interest is extracted through point cloud segmentation to generate the object of interest spectral three-dimensional model.

[0492] According to this structure, a spectral information mapping image group can be generated based on an RGB image group and a spectral image group, thereby generating a spectral three-dimensional model of the object of interest; an RGB three-dimensional model and posture information of the RGB camera are generated from the RGB image group, and the RGB three-dimensional model is a three-dimensional model in RGB; the spectral information from the spectral information mapping image group is mapped onto the RGB three-dimensional model to generate a spectral three-dimensional model; and a point cloud of the subject of interest is also extracted from the spectral three-dimensional model.

[0493] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0494] < Fig.34 Spectral three-dimensional model generation process of an object of interest in an image processing device>

[0495] Next, we will refer to Fig.35 The flowchart in the description is Fig.34 The image processing device 301 in the present invention is used to generate a three-dimensional model of a spectrum of interest.

[0496] It should be noted that Fig.35 The processing of steps S311 to S320 in the flowchart of Fig.14 The processing of steps S111 to S120 in the flowchart of FIG. 1 is similar, so the description thereof will be omitted.

[0497] In other words, once the spectrum three-dimensional model is generated through the processing of steps S311 to S320, the sequence proceeds to step S321.

[0498] In step S321 , the object of interest point cloud extraction processing unit 418 extracts only the point cloud of the spectral information of the object of interest by segmenting the point cloud using the spectrum on which the spectral information constituting the spectral three-dimensional model is mapped, thereby generating the object of interest spectral three-dimensional model.

[0499] Through the above processing, a spectral information mapping image group can be generated according to the RGB image group and the spectral image group to generate a spectral three-dimensional model of the object of interest; an RGB three-dimensional model and the posture information of the RGB camera are generated from the RGB image group, and the RGB three-dimensional model is a three-dimensional model in RGB; the spectral information from the spectral information mapping image group is mapped onto the RGB three-dimensional model to generate a spectral three-dimensional model; and a point cloud of the subject of interest is also extracted from the spectral three-dimensional model.

[0500] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0501] <3-7. Seventh Configuration Example of Second Embodiment>

[0502] The above describes an example of generating a spectral information mapping image group based on an RGB image group and a spectral image group to generate a spectral three-dimensional model of an object of interest; generating an RGB three-dimensional model and posture information of the RGB camera from the RGB image group, the RGB three-dimensional model being a three-dimensional model in RGB; mapping the spectral information from the spectral information mapping image onto the RGB three-dimensional model to generate a spectral three-dimensional model; and also extracting a point cloud of an object of interest from the spectral three-dimensional model.

[0503] However, the distance image group of all cameras can be reconfigured according to the RGB image group and the spectral image group, a three-dimensional point cloud of RGB+spectral information can be generated according to the distance image group of all cameras, a three-dimensional model of RGB+spectral information can be generated, and the posture information of all cameras can be generated.

[0504] Then, according to the distance image group and the posture information, the spectral information of the spectral image group is mapped to the RGB image group to generate a spectral information mapping image group, thereby generating a spectral three-dimensional model of the object of interest; according to the information of the spectral information mapping image group, the spectral information is mapped to the three-dimensional model of the RGB+spectral information to generate a spectral three-dimensional model; and a point cloud of the subject of interest is also extracted from the spectral three-dimensional model.

[0505] Fig.36 The seventh configuration example of the second embodiment of the image processing device is shown, wherein a spectral three-dimensional model of the object of interest is generated by reconfiguring the distance image group of all cameras from the RGB image group and the spectral image group; a three-dimensional model of RGB+spectral information is generated from the distance image group of all cameras, and posture information of all cameras is generated; based on the distance image group and the posture information, the spectral information of the spectral image group is mapped to each RGB image group to generate a spectral information mapping image group; based on the information of the spectral information mapping image group, the spectral information is mapped to the three-dimensional model of RGB+spectral information to generate a spectral three-dimensional model; and a point cloud of the subject of interest is also extracted from the spectral three-dimensional model.

[0506] Fig.36 The image processing device 301 in the embodiment includes an RGB camera 431 , a spectral camera 432 , a three-dimensional reconfiguration unit 433 , a mapping processing unit 434 , a spectral information mapping processing unit 435 , and an object of interest point cloud extraction processing unit 436 .

[0507] Please note that Fig.36 The RGB camera 431 in the image processing device 301 to the spectral information mapping processing unit 435 and Fig.15 The RGB camera 231 to the spectral information mapping processing unit 235 in the image processing device 101 are the same, so their description will be omitted.

[0508] In other words, Fig.36 The image processing device 301 in Fig.15 The difference between the image processing device 101 in FIG. 1 and FIG. 2 is that a focus object point cloud extraction processing unit 436 is added.

[0509] The object point cloud extraction processing unit 436 has Fig.26The same structure as the object of interest point cloud extraction processing unit 338 is used to map the spectrum on the point cloud of the spectral information constituting the spectral three-dimensional model, and only the point cloud of the spectral information of the object of interest is extracted through point cloud segmentation to generate the object of interest spectral three-dimensional model.

[0510] According to this structure, a three-dimensional model of RGB+spectral information is generated based on the RGB image group and the spectral image group, and the distance image group and posture information of all cameras are generated, thereby generating a spectral three-dimensional model of the object of interest; the spectral information of the spectral image group is mapped from the distance image group and the posture information to the RGB image group to generate a spectral information mapping image group; based on the information of the spectral information mapping image group, the spectral information is mapped to the three-dimensional model of RGB+spectral information to generate a spectral three-dimensional model; and a point cloud of the subject of interest is also extracted from the spectral three-dimensional model.

[0511] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0512] < Fig.36 Spectral three-dimensional model generation process of an object of interest in an image processing device>

[0513] Next, we will refer to Fig.37 Flowchart description in Fig.36 The image processing device 301 in the present invention is used to generate a three-dimensional model of a spectrum of interest.

[0514] It should be noted that Fig.37 The processing of steps S331 to S338 in the flowchart of Fig.16 The processing of steps S131 to S138 in the flowchart of FIG. 1 is similar, so their description will be omitted.

[0515] In other words, once the spectrum three-dimensional model is generated through the processing of steps S331 to S338 , the sequence moves to step S339 .

[0516] In step S339 , the object of interest point cloud extraction processing unit 436 extracts only the point cloud of the spectral information of the object of interest by segmenting the point cloud, using the spectrum on which the spectral information constituting the spectral three-dimensional model is mapped, thereby generating a spectral three-dimensional model of the object of interest.

[0517] Through the above processing, a three-dimensional model of RGB+spectral information can be generated according to the RGB image group and the spectral image group, and the distance image group and posture information of all cameras can be generated to generate a spectral three-dimensional model of the object of interest; the spectral information of the spectral image group is mapped from the distance image group and the posture information to the RGB image group to generate a spectral information mapping image group; according to the information of the spectral information mapping image group, the spectral information is mapped to the three-dimensional model of RGB+spectral information to generate a spectral three-dimensional model; and a point cloud of the subject of interest is also extracted from the spectral three-dimensional model.

[0518] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0519] <3-8. Eighth Configuration Example of Second Embodiment>

[0520] The above illustrates an example of generating a three-dimensional model of RGB+spectral information based on an RGB image group and a spectral image group, generating a range image group and posture information of all cameras to generate a spectral three-dimensional model of an object of interest; mapping the spectral information of the spectral image group from the range image group and the posture information to the RGB image group to generate a spectral information mapping image group; mapping the spectral information to the three-dimensional model of RGB+spectral information based on the information of the spectral information mapping image group to generate a spectral three-dimensional model; and also extracting a point cloud of the subject of interest from the spectral three-dimensional model.

[0521] However, a spectral three-dimensional model of the object of interest can be generated by generating an unscaled three-dimensional model and a range image group through three-dimensional reconstruction based on an RGB image group; rescaling the unscaled three-dimensional model into a real scaled three-dimensional model by comparing the unscaled range image group with a real scaled range image group obtained by stereo matching the RGB camera and the spectral camera; mapping the spectral information onto the rescaled real scaled three-dimensional model to generate a real scaled spectral three-dimensional model; and further extracting a point cloud of the object of interest from the real scaled spectral three-dimensional model.

[0522] Fig.38An eighth configuration example of the second embodiment of the image processing device is shown, which generates a spectral three-dimensional model of an object of interest by generating an unscaled three-dimensional model and a range image group based on three-dimensional reconfiguration of an RGB image group; rescaling the unscaled three-dimensional model into a real scaled three-dimensional model by comparing the unscaled range image group with a real scaled range image group obtained by stereo matching the RGB camera and the spectral camera; mapping the spectral information onto the rescaled real scaled three-dimensional model to generate a real scaled spectral three-dimensional model; and further extracting a point cloud of the object of interest from the real scaled spectral three-dimensional model.

[0523] Fig.38 The image processing device 301 includes an RGB camera 451, a spectral camera 452, a three-dimensional reconfiguration unit 453, a rescaling processing unit 454, a spectral information mapping processing unit 455, a stereo calibration processing unit 456, a stereo matching processing unit 457, a mapping processing unit 458 and a point cloud extraction processing unit 459 for an object of interest.

[0524] Please note that Fig.38 The RGB camera 451 in the image processing device 301 to the mapping processing unit 458 and Fig.17 The RGB camera 251 to the mapping processing unit 258 in the image processing device 101 are the same, so their description will be omitted.

[0525] In other words, Fig.38 The image processing device 301 in Fig.17 The image processing device 101 in FIG. 1 is different in that a focus object point cloud extraction processing unit 459 is added.

[0526] The object point cloud extraction processing unit 459 has Fig.26 The same structure as the object of interest point cloud extraction processing unit 338 is used to map the spectrum on the point cloud of the spectral information constituting the spectral three-dimensional model, and only the point cloud of the spectral information of the object of interest is extracted through point cloud segmentation to generate the object of interest spectral three-dimensional model.

[0527] This configuration makes it possible to generate a truly scaled spectral three-dimensional model of an object of interest by generating an unscaled three-dimensional model and a range image group based on three-dimensional reconfiguration of an RGB image group; rescaling the unscaled three-dimensional model into a truly scaled three-dimensional model by comparing the unscaled range image group with the truly scaled range image group obtained by stereo matching the RGB camera and the spectral camera; mapping the spectral information onto the rescaled truly scaled three-dimensional model to produce a truly scaled spectral three-dimensional model; and further extracting a point cloud of the object of interest from the truly scaled spectral three-dimensional model.

[0528] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0529] < Fig.38 Spectral three-dimensional model generation process of an object of interest in an image processing device>

[0530] Next, we will refer to Fig.39 The flowchart in the description is Fig.38 The image processing device 301 in the present invention is used to generate a three-dimensional model of a spectrum of interest.

[0531] It should be noted that Fig.39 The processing of steps S351 to S361 in the flowchart of Fig.19 The processing of steps S151 to S161 in the flowchart is similar, so their description will be omitted.

[0532] That is, when the spectrum three-dimensional model is generated through the processing of steps S351 to S361, the process proceeds to step S362.

[0533] In step S362 , the attention object point cloud extraction processing unit 459 generates an attention object spectral three-dimensional model by extracting only the point cloud of spectral information of the attention object through point cloud segmentation using the spectrum on the point cloud onto which the spectral information constituting the spectral three-dimensional model is mapped.

[0534] The above processing enables generating a truly scaled spectral three-dimensional model of the object of interest by generating an unscaled three-dimensional model and a range image group based on three-dimensional reconstruction of the RGB image group; rescaling the unscaled three-dimensional model into a truly scaled three-dimensional model by comparing the unscaled range image group with the truly scaled range image group obtained by stereo matching the RGB camera and the spectral camera; mapping the spectral information onto the rescaled truly scaled three-dimensional model to generate a truly scaled spectral three-dimensional model; and further extracting a point cloud of the object of interest from the truly scaled spectral three-dimensional model.

[0535] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0536] <3-9. Ninth Configuration Example of Second Embodiment>

[0537] The foregoing describes an example in which a spectral three-dimensional model of an object of interest is generated by generating an unscaled three-dimensional model and a range image group through three-dimensional reconstruction based on an RGB image group; the unscaled three-dimensional model is rescaled into a real scaled three-dimensional model by comparing the unscaled range image group with a real scaled range image group obtained by stereo matching the RGB camera and the spectral camera; the spectral information is mapped onto the rescaled real scaled three-dimensional model to produce a real scaled spectral three-dimensional model; and a point cloud of the object of interest is further extracted from the real scaled spectral three-dimensional model.

[0538] However, a spectral three-dimensional model of the object of interest can be generated by coaxially configuring an RGB camera and a spectral camera using, for example, a beam splitter; mapping the spectral information of the spectral image onto a three-dimensional model obtained by three-dimensional reconstruction of the RGB image group to generate the spectral three-dimensional model; and also extracting a point cloud of the subject of interest from the spectral three-dimensional model.

[0539] Fig.40 A ninth configuration example of the second embodiment of the image processing device is shown, wherein, for example, a spectral three-dimensional model of an object of interest is generated by coaxially configuring an RGB camera and a spectral camera using a beam splitter, etc.; spectral information of the spectral image is mapped onto a three-dimensional model obtained by three-dimensionally reconstructing the RGB image group to generate the spectral three-dimensional model; and a point cloud of the subject of interest is also extracted from the spectral three-dimensional model.

[0540] Fig.40 The image processing device 301 in the embodiment includes an RGB camera 471 , a spectral camera 472 , a three-dimensional reconfiguration unit 473 , a spectral information mapping processing unit 474 , and an object of interest point cloud extraction processing unit 475 .

[0541] Please note that Fig.40 The RGB camera 471 to the spectral information mapping processing unit 474 in the image processing device 301 have Fig.19 The RGB camera 271 to the spectral information mapping processing unit 274 in the image processing device 101 have the same configuration, so their description will be omitted.

[0542] In other words, Fig.40 The image processing device 301 in Fig.19 The image processing device 101 in FIG. 1 is different in that a focus object point cloud extraction processing unit 475 is added.

[0543] The object point cloud extraction processing unit 475 has Fig.26The same structure as the object of interest point cloud extraction processing unit 338 is used to map the spectrum on the point cloud of the spectral information constituting the spectral three-dimensional model, and only the point cloud of the spectral information of the object of interest is extracted through point cloud segmentation to generate the object of interest spectral three-dimensional model.

[0544] Such a configuration makes it possible to generate a spectral three-dimensional model of the object of interest by, for example, coaxially configuring an RGB camera and a spectral camera using a beam splitter, etc.; mapping the spectral information of the spectral image onto the three-dimensional model obtained by three-dimensionally reconstructing the RGB image group to generate the spectral three-dimensional model; and also extracting a point cloud of the subject of interest from the spectral three-dimensional model.

[0545] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0546] < Fig.40 Spectral three-dimensional model generation process of an object of interest in an image processing device>

[0547] Next, we will refer to Fig.41 The flowchart in the description is Fig.40 The image processing device 301 in the present invention is used to generate a three-dimensional model of a spectrum of interest.

[0548] It should be noted that in Fig.41 The processing of steps S371 to S374 in the flowchart of Fig. 20 The processing of steps S171 to S174 in the flowchart is similar, and therefore, will not be described.

[0549] In other words, once the spectrum three-dimensional model is generated through the processing of steps S371 to S374, the sequence moves to step S375.

[0550] In step S375 , the object of interest point cloud extraction processing unit 475 extracts only the point cloud of the spectral information of the object of interest by segmenting the point cloud using the spectrum on which the spectral information constituting the spectral three-dimensional model is mapped, thereby generating the object of interest spectral three-dimensional model.

[0551] This processing makes it possible to generate a spectral three-dimensional model of the object of interest by, for example, coaxially configuring an RGB camera and a spectral camera using a beam splitter; mapping the spectral information of the spectral image onto a three-dimensional model obtained by three-dimensionally reconstructing the RGB image group to generate the spectral three-dimensional model; and also extracting a point cloud of the subject of interest from the spectral three-dimensional model.

[0552] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0553] <3-10. Tenth Configuration Example of Second Implementation>

[0554] The above describes an example in which a spectral three-dimensional model of an object of interest is generated by coaxially configuring an RGB camera and a spectral camera using, for example, a beam splitter; the spectral information of the spectral image is mapped onto a three-dimensional model obtained by three-dimensionally reconstructing the RGB image group to generate the spectral three-dimensional model; and a point cloud of the subject of interest is also extracted from the spectral three-dimensional model.

[0555] However, the RGB camera and the spectral camera can be coaxially configured using a beam splitter, for example, and a spectral information mapping image can be generated by first mapping the spectral information of the spectral image to the RGB image; mapping the image group according to the spectral image, mapping the spectral information to a three-dimensional model obtained by three-dimensional reconstruction of the RGB image group, to generate a spectral three-dimensional model; and also extracting a point cloud of the object of interest from the spectral three-dimensional model.

[0556] Fig.42 It is the tenth configuration example of the second embodiment of the image processing device, wherein, for example, an RGB camera and a spectral camera are coaxially configured using a beam splitter, and a spectral three-dimensional model of an object of interest can be generated by first mapping the spectral information of a spectral image onto an RGB image to generate a spectral information mapping image; based on the spectral image mapping image group, the spectral information is mapped onto a three-dimensional model obtained by three-dimensional reconstruction of the RGB image group to generate a spectral three-dimensional model; and a point cloud of the object of interest is also extracted from the spectral three-dimensional model.

[0557] Fig.42 The image processing device 301 in the embodiment includes an RGB camera 491 , a spectral camera 492 , a three-dimensional reconfiguration unit 493 , a mapping processing unit 494 , a spectral information mapping processing unit 495 , and an object of interest point cloud extraction processing unit 496 .

[0558] Please note that Fig.42 The RGB camera 491 in the image processing device 301 to the spectral information mapping processing unit 495 and Fig.21 The RGB camera 291 to the mapping processing unit 295 in the image processing device 101 have the same configuration, and a description thereof will be omitted.

[0559] In other words, Fig.42 The image processing device 301 in Fig.21 The difference between the image processing device 101 in FIG. 1 and FIG. 2 is that a focus object point cloud extraction processing unit 496 is added.

[0560] The object point cloud extraction processing unit 496 has Fig.26The same structure as the object of interest point cloud extraction processing unit 338 is used, and the spectrum on the point cloud mapped with the spectral information constituting the spectral three-dimensional model is used to extract only the point cloud of the spectral information of the object of interest through point cloud segmentation, thereby generating the object of interest spectral three-dimensional model.

[0561] According to this structure, for example, a beam splitter or the like can be used to coaxially structure the RGB camera and the spectral camera, and a spectral information mapping image is generated by first mapping the spectral information of the spectral image to the RGB image, thereby generating a spectral three-dimensional model of the object of interest; according to the spectral image mapping image group, the spectral information is mapped to the three-dimensional model obtained by three-dimensional reconstruction of the RGB image group to generate a spectral three-dimensional model; and a point cloud of the object of interest is also extracted from the spectral three-dimensional model.

[0562] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0563] < Fig.42 Spectral three-dimensional model generation process of an object of interest in an image processing device>

[0564] Next, refer to Fig.43 The flowchart in the description is Fig.42 The image processing device 301 in FIG. 3 performs a process of generating a three-dimensional model of a spectrum of an object of interest.

[0565] It should be noted that Fig.43 The processing of steps S391 to S398 in the flowchart of Fig. 22 The processing of steps S191 to S198 in the flowchart is similar, and therefore the description thereof will be omitted.

[0566] In other words, once the spectrum three-dimensional model is generated through the processing of steps S391 to S398, the sequence moves to step S399.

[0567] In step S399 , the object of interest point cloud extraction processing unit 496 extracts only the point cloud of the spectral information of the object of interest by segmenting the point cloud using the spectrum on which the spectral information constituting the spectral three-dimensional model is mapped, thereby generating the object of interest spectral three-dimensional model.

[0568] The above-mentioned processing enables the RGB camera and the spectral camera to be coaxially configured using, for example, a beam splitter, and to generate a spectral three-dimensional model of the object of interest by first mapping the spectral information of the spectral image onto the RGB image to generate a spectral information mapping image; based on the spectral image mapping image group, the spectral information is mapped onto the three-dimensional model obtained by three-dimensionally reconstructing the RGB image group to generate a spectral three-dimensional model; and a point cloud of the object of interest is also extracted from the spectral three-dimensional model.

[0569] As a result, it is possible to generate a spectral three-dimensional image of the object of interest from a desired viewpoint position based on the spectral three-dimensional model of the object of interest.

[0570] <<4. Examples of execution through software>>

[0571] The above series of processing can also be performed by hardware or software. For example, when the series of processing is performed by software, the program constituting the software is installed from a recording medium in a computer incorporated in dedicated hardware, or installed in a general-purpose computer capable of performing various functions by installing various programs.

[0572] Fig.44 An example of a configuration of a general-purpose computer is shown. The computer includes a central processing unit (CPU) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A read-only memory (ROM) 1002 and a random access memory (RAM) 1003 are connected to the bus 1004.

[0573] An input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009 are connected to the input / output interface 1005. The input unit 1006 is composed of an input device such as a keyboard, a mouse, etc., through which a user inputs an operation command. The output unit 1007 outputs an image of a processing operation screen or a processing result to a display device. The storage unit 1008 is composed of a hard disk drive that stores programs and various types of data. The communication unit 1009 is such as a local area network (LAN) adapter, etc., and performs communication processing through a network such as the Internet. A drive 1010 is also connected for reading data from and writing data to a removable recording medium 1011, such as a magnetic disk (including a floppy disk), an optical disk (including a compact disk read-only memory (CD-ROM), a digital versatile disk (DVD)), a magneto-optical disk (including a mini disk (MD)), or a semiconductor memory.

[0574] The CPU 1001 executes various types of processing according to a program stored in the ROM 1002 or a program read out from a removable storage medium 1011 (such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc.) installed in the storage unit 1008 and then loaded from the storage unit 1008 into the RAM 1003. Data and the like necessary for the CPU 1001 to execute various processes are also appropriately stored in the RAM 1003.

[0575] In the computer configured as described above, for example, the CPU 1001 performs the above-described series of processing by loading a program stored in the storage unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and, for example, executing the program.

[0576] The program executed by the computer (CPU 1001) can be recorded on, for example, the removable storage medium 1011 as a package medium and provided in this state. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0577] In the computer, by inserting the removable storage medium 1011 into the drive 1010, the program can be installed in the storage unit 1008 via the input / output interface 1005. The program can be received by the communication unit 1009 via a wired or wireless transmission medium to be installed in the storage unit 1008. In addition, the program can be installed in the ROM 1002 or the storage unit 1008 in advance.

[0578] It should be noted that the program executed by the computer may be a program that executes processing chronologically in the order described in this specification, or may be a program that executes processing in parallel or at necessary timing such as call time.

[0579] It should be noted that in Fig.44 The CPU 1001 in the embodiment implements the functions of the configuration excluding the RGB camera, the range camera, and the spectral camera in all the image processing devices 101 and 301 described in this specification.

[0580] In this specification, "system" means a group of multiple constituent elements (devices, modules (components), etc.), and it does not matter whether all the constituent elements are arranged in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a device in which multiple modules are arranged in a single housing both constitute a system.

[0581] It should be noted that the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications may be made without departing from the basic spirit of the present disclosure.

[0582] For example, the present disclosure may be configured through cloud computing in which a plurality of devices share and collaboratively process one function through a network.

[0583] Furthermore, each step described in the above flowchart may be performed by one device or by a plurality of devices in a shared manner.

[0584] Furthermore, when a single step includes a plurality of processes, the plurality of processes included in the single step may be performed by a single device, or may be performed by a plurality of devices in a distributed manner.

[0585] It should be noted that the present disclosure can also adopt configurations such as those described below.

[0586] <1> An image processing device, comprising:

[0587] A spectral three-dimensional model generating unit generates a spectral three-dimensional model based on a spectral image group consisting of a plurality of spectral images having spectral information of a predetermined space and a spatial image group consisting of a plurality of spatial images for generating a three-dimensional model of the predetermined space, in which the spectral information is mapped onto the three-dimensional model.

[0588] <2> according to <1> an image processing device,

[0589] The spatial image is an RGB image, a brightness image, a monochrome image, a distance image or a three-dimensional image, and the spatial image group is an RGB image group, a brightness image group, a monochrome image group, a distance image group or a three-dimensional image group.

[0590] <3> according to <2> an image processing device,

[0591] Wherein, the spectral three-dimensional model generating unit includes:

[0592] a mapping processing unit that generates a spectral information mapping image group by mapping spectral information of the spectral image group to each pixel constituting the RGB image group; and

[0593] The three-dimensional reconfiguration unit generates a spectral three-dimensional model in which spectral information is mapped onto the three-dimensional model by reconfiguring the three-dimensional model based on the spectral information mapping image group.

[0594] <4> according to <3> an image processing device,

[0595] The mapping processing unit generates a spectral information mapping image by matching the feature points of the RGB image with the feature points of the spectral image in the same frame and mapping the spectral information in the spectral image to the positions where the feature points on the RGB image match.

[0596] <5> according to <2> an image processing device,

[0597] Wherein, the spectral three-dimensional model generating unit includes:

[0598] An RGB three-dimensional reconfiguration unit generates an RGB three-dimensional point cloud based on the RGB image group;

[0599] A spectral three-dimensional reconfiguration unit, which generates a spectral three-dimensional point cloud based on the spectral image group;

[0600] an alignment unit, aligning the RGB three-dimensional point cloud and the spectral three-dimensional point cloud; and

[0601] The mapping processing unit generates a spectral three-dimensional model by mapping the spectral information of the spectral three-dimensional point cloud near the aligned RGB three-dimensional point cloud.

[0602] <6> according to <5> an image processing device,

[0603] The mapping processing unit generates a spectral three-dimensional model by mapping the average value of spectral information of spectral three-dimensional point clouds near the aligned RGB three-dimensional point cloud, the weighted average value based on distance, or the spectral information of the spectral three-dimensional point cloud closest to the RGB three-dimensional point cloud.

[0604] <7> according to <2> an image processing device,

[0605] Wherein, the spectral three-dimensional model generating unit includes:

[0606] A three-dimensional reconfiguration unit generates a hybrid three-dimensional point cloud based on the RGB image group and the spectral image group, wherein the hybrid three-dimensional point cloud is a three-dimensional point cloud in which RGB information and spectral information are mixed; and

[0607] The mapping processing unit generates a spectral three-dimensional model by mapping the spectral information of the three-dimensional point cloud composed of spectral information near the three-dimensional point cloud composed of RGB information in the mixed three-dimensional point cloud.

[0608] <8> according to <2> an image processing device,

[0609] Wherein, a camera position relationship is specified, the camera position relationship is the position relationship between an RGB camera for capturing an RGB image and a spectral camera for capturing a spectral image, and

[0610] The spectral three-dimensional model generation unit includes:

[0611] a three-dimensional reconfiguration unit that generates a three-dimensional image group corresponding to each RGB image group and pose information of the RGB camera based on the RGB image group; and

[0612] The mapping processing unit maps the spectral information of the spectral image group to a three-dimensional model specified from the three-dimensional image group based on the posture information of the RGB camera and the camera position relationship, thereby generating a spectral three-dimensional model.

[0613] <9> according to <2> an image processing device,

[0614] Wherein, a camera position relationship is specified, the camera position relationship is the position relationship between a range camera for capturing a range image and a spectral camera for capturing a spectral image, and

[0615] The spectral three-dimensional model generation unit includes:

[0616] a posture estimation unit, which estimates the posture information of the range camera that captures the range image based on the range image group;

[0617] a mapping processing unit, which generates a spectral information mapping image group by mapping the spectral information of the spectral image group onto the range image group; and

[0618] The three-dimensional reconfiguration unit generates a spectral three-dimensional model from the spectral information mapping image group based on the attitude information of the range camera and the camera position relationship.

[0619] <10> according to <2> an image processing device,

[0620] Wherein, the spectral three-dimensional model generating unit includes:

[0621] a 3D reconfiguration unit that estimates the pose of the 3D model and the RGB camera that captured the RGB images based on the RGB image set;

[0622] a first mapping processing unit that generates a spectral information mapping image group by mapping spectral information of the spectral image group to each pixel constituting the RGB image group; and

[0623] The second mapping processing unit maps the spectral information of the spectral information mapping image group to the three-dimensional model according to the posture of the RGB camera, thereby generating a spectral three-dimensional model.

[0624] <11> according to <2> an image processing device,

[0625] Wherein, the spectral three-dimensional model generating unit includes:

[0626] a three-dimensional reconfiguration unit that generates, based on the RGB image group and the spectral image group, a hybrid three-dimensional model composed of a three-dimensional point cloud mixed with RGB information and spectral information, attitude information of an RGB camera that captures the RGB image, attitude information of a spectral camera that captures the spectral image, and a range image group corresponding to each of the RGB image group and the spectral image group;

[0627] a first mapping processing unit, which maps spectral information of the spectral image group to each RGB image group based on the posture information of the RGB camera, the posture information of the spectral camera, and the range image group, thereby generating a spectral information mapping image group; and

[0628] The second mapping processing unit maps the spectral information to the hybrid three-dimensional model based on the spectral information mapping image group to generate a spectral three-dimensional model.

[0629] <12> according to <2> an image processing device,

[0630] Wherein, the spectral three-dimensional model generating unit includes:

[0631] A distance image generating unit generates a real scaled distance image through stereo matching based on the RGB image group and the spectral image group;

[0632] a three-dimensional reconfiguration unit for generating an unscaled three-dimensional model and an unscaled range image based on the RGB image group;

[0633] a rescaling processing unit, which rescales the unscaled three-dimensional model into a true scaled three-dimensional model by comparing the true scaled range image with the unscaled range image;

[0634] a first mapping processing unit, mapping spectral information of the spectral image group onto each RGB image group based on the real scaled distance image, thereby generating a spectral information mapping image group; and

[0635] The second mapping processing unit generates a spectral three-dimensional model by mapping the spectral information of the spectral information mapping image group to the real scaled three-dimensional model.

[0636] <13> according to <12> an image processing device,

[0637] In addition to the real scaled distance image through stereo matching, the distance image generation unit also generates relative posture information between the RGB camera capturing the RGB image and the spectral camera capturing the spectral image based on the RGB image group and the spectral image group.

[0638] In addition to the unscaled 3D model and the unscaled range image, the 3D reconfiguration unit also generates pose information of the RGB camera based on the RGB image set, and

[0639] The rescaling processing unit rescales the unscaled three-dimensional model into the real scaled three-dimensional model by comparing the real scaled range image with the unscaled range image based on the posture information and the relative posture information of the RGB camera.

[0640] <14> according to <2> an image processing device,

[0641] wherein the optical axis of the RGB camera for capturing the RGB image and the optical axis of the spectral camera for capturing the spectral image are adjusted to be coaxial, and

[0642] The spectral three-dimensional model generation unit includes:

[0643] a 3D reconfiguration unit for generating a 3D model and pose information of an RGB camera based on the RGB image group; and

[0644] The first mapping processing unit generates a spectral three-dimensional model by mapping spectral information of the spectral image group onto the three-dimensional model.

[0645] <15> according to <14> an image processing device,

[0646] Wherein, the spectral three-dimensional model generating unit further comprises:

[0647] a second mapping processing unit, generating a spectral information mapping image group by mapping the spectral information of the spectral image group to the RGB image group;

[0648] The first mapping processing unit generates a spectral three-dimensional model by mapping spectral information of the spectral information mapping image group to the three-dimensional model.

[0649] <16> according to <1> to <15> The image processing device of any one of the above, further comprising:

[0650] The attention object spectrum information extraction unit extracts spectrum information of the attention object from the spectrum information.

[0651] <17> according to <16> an image processing device,

[0652] The object spectral information extraction unit extracts the spectral information of the object from any of the following items: a spectral image group; a spectral information mapping image group, in which the spectral information of the spectral image group is mapped onto the spatial image group; and a spectral three-dimensional model.

[0653] <18> An image processing method comprises the following steps:

[0654] A spectral three-dimensional model is generated, which maps spectral information onto a three-dimensional model based on a spectral image group consisting of a plurality of spectral images having spectral information of a predetermined space and a spatial image group consisting of a plurality of spatial images used to generate a three-dimensional model of the predetermined space.

[0655] <19> A program that causes a computer to:

[0656] A spectral three-dimensional model generating unit generates a spectral three-dimensional model that maps spectral information to a three-dimensional model based on a spectral image group consisting of a plurality of spectral images having spectral information of a predetermined space and a spatial image group consisting of a plurality of spatial images for generating a three-dimensional model of the predetermined space.

[0657] <20> An image processing device, comprising:

[0658] a spectral three-dimensional model generating unit for generating a spectral three-dimensional model that maps the spectral information to the three-dimensional model based on a spectral image group consisting of a plurality of spectral images having spectral information of a predetermined space and a spatial image group consisting of a plurality of spatial images for generating a three-dimensional model of the predetermined space; and

[0659] The attention object spectrum information extraction unit extracts spectrum information of the attention object from the spectrum information.

[0660] [Reference Numbers List]

[0661] 101 Image processing device

[0662] 111 RGB Camera

[0663] 112 Spectral Camera

[0664] 113 Feature point matching processing unit

[0665] 114 Pixel-by-Pixel Registration Processing Unit

[0666] 115 Mapping Processing Unit

[0667] 116 Three-dimensional reconfiguration unit

[0668] 131 RGB Camera

[0669] 132 Spectral Camera

[0670] 133, 134 Three-dimensional reconfiguration unit

[0671] 135 3D point cloud alignment processing unit

[0672] 136 Corresponding point cloud extraction processing unit

[0673] 137 Spectral information mapping processing unit

[0674] 151 RGB Camera

[0675] 152 Spectral Camera

[0676] 153 Three-dimensional reconfiguration unit

[0677] 154 Corresponding point cloud extraction processing unit

[0678] 155 Spectral information mapping processing unit

[0679] 171 RGB Camera

[0680] 172 Spectral Camera

[0681] 173 Three-dimensional reconfiguration unit

[0682] 174 Spectral information mapping processing unit

[0683] 191 Distance Camera

[0684] 192 Spectral Camera

[0685] 193 Camera Pose Estimation Unit

[0686] 194 Spectral information mapping processing unit

[0687] 195 Three-dimensional reconfiguration unit

[0688] 211 RGB Camera

[0689] 212 Spectral Camera

[0690] 213 Three-dimensional reconfiguration unit

[0691] 214 Spectral information mapping processing unit

[0692] 215 Feature point matching processing unit

[0693] 216 Pixel-by-Pixel Registration Processing Unit

[0694] 217 Matching Processing Unit

[0695] 231 RGB Camera

[0696] 232 Spectral Camera

[0697] 233 Three-dimensional reconfiguration unit

[0698] 234 Mapping Processing Unit

[0699] 235 Spectral information mapping processing unit

[0700] 251 RGB Camera

[0701] 252 Spectral Camera

[0702] 253 Three-dimensional reconfiguration unit

[0703] 254 Rescaling Processing Units

[0704] 255 Spectral information mapping processing unit

[0705] 256 Stereo Calibration Processing Units

[0706] 257 stereo matching processing units

[0707] 258 Spectral information mapping processing unit

[0708] 271 RGB Camera

[0709] 272 Spectral Camera

[0710] 273 Three-dimensional reconfiguration unit

[0711] 274 Spectral Information Mapping Processing Unit

[0712] 291 RGB Camera

[0713] 292 Spectral Camera

[0714] 293 Three-dimensional reconfiguration unit

[0715] 294 Spectral Information Mapping Processing Unit

[0716] 295 Mapping Processing Unit

[0717] 316, 338, 356, 375, 396, 418, 436, 459, 475, 496 focus on the object point cloud extraction processing unit.

Claims

1. An image processing device, comprising: A spectral three-dimensional model generating unit generates a spectral three-dimensional model based on a spectral image group consisting of a plurality of spectral images having spectral information of a predetermined space and a spatial image group consisting of a plurality of spatial images for generating a three-dimensional model of the predetermined space, wherein the spectral information is mapped onto the three-dimensional model.

2. The image processing device according to claim 1, in, The spatial image is an RGB image, a brightness image, a monochrome image, a distance image or a three-dimensional image, and the spatial image group is an RGB image group, a brightness image group, a monochrome image group, a distance image group or a three-dimensional image group.

3. The image processing device according to claim 2, in, The spectrum three-dimensional model generating unit comprises: a mapping processing unit that generates a spectral information mapping image group by mapping the spectral information of the spectral image group to each pixel constituting the RGB image group; and The three-dimensional reconfiguration unit generates a spectral three-dimensional model in which the spectral information is mapped onto the three-dimensional model by reconfiguring the three-dimensional model based on the spectral information mapping image group.

4. The image processing device according to claim 3, in, The mapping processing unit generates a spectral information mapping image by matching feature points of the RGB image with feature points of the spectral image in the same frame and mapping the spectral information in the spectral image to positions where the feature points on the RGB image match.

5. The image processing device according to claim 2, in, The spectrum three-dimensional model generating unit comprises: an RGB three-dimensional reconfiguration unit, which generates an RGB three-dimensional point cloud based on the RGB image group; a spectral three-dimensional reconfiguration unit, which generates a spectral three-dimensional point cloud based on the spectral image group; an alignment unit, aligning the RGB three-dimensional point cloud and the spectral three-dimensional point cloud; and A mapping processing unit generates the spectral three-dimensional model by mapping the spectral information of the spectral three-dimensional point cloud near the aligned RGB three-dimensional point cloud.

6. The image processing device according to claim 5, in, The mapping processing unit generates the spectral three-dimensional model by mapping the average value of the spectral information of the spectral three-dimensional point cloud near the aligned RGB three-dimensional point cloud, the weighted average value according to the distance, or the spectral information of the spectral three-dimensional point cloud closest to the RGB three-dimensional point cloud.

7. The image processing device according to claim 2, in, The spectrum three-dimensional model generating unit comprises: a three-dimensional reconfiguration unit, which generates a mixed three-dimensional point cloud based on the RGB image group and the spectral image group, wherein the mixed three-dimensional point cloud is a three-dimensional point cloud in which RGB information and the spectral information are mixed; and A mapping processing unit generates the spectral three-dimensional model by mapping the spectral information of the three-dimensional point cloud composed of the spectral information near the three-dimensional point cloud composed of the RGB information in the mixed three-dimensional point cloud.

8. The image processing device according to claim 2, in, specifying a camera position relationship, the camera position relationship being a position relationship between an RGB camera that captures the RGB image and a spectral camera that captures the spectral image, and The spectrum three-dimensional model generating unit comprises: a three-dimensional reconfiguration unit, which generates a three-dimensional image group corresponding to each of the RGB image groups and posture information of the RGB camera based on the RGB image groups; and A mapping processing unit maps the spectral information of the spectral image group to the three-dimensional model specified from the three-dimensional image group based on the posture information of the RGB camera and the camera position relationship, thereby generating the spectral three-dimensional model.

9. The image processing device according to claim 2, in, specifying a camera position relationship, the camera position relationship being a position relationship between a range camera that captures the range image and a spectral camera that captures the spectral image, and The spectrum three-dimensional model generating unit comprises: a posture estimation unit, which estimates the posture information of the range camera that captures the range image based on the range image group; a mapping processing unit, which generates a spectral information mapping image group by mapping the spectral information of the spectral image group onto the range image group; and A three-dimensional reconfiguration unit generates a spectral three-dimensional model from the spectral information mapping image group based on the posture information of the range camera and the camera position relationship.

10. The image processing device according to claim 2, in, The spectrum three-dimensional model generating unit comprises: a 3D reconfiguration unit that estimates the pose of the 3D model and an RGB camera that captures the RGB images based on the RGB image set; a first mapping processing unit, generating a spectral information mapping image group by mapping spectral information of the spectral image group to each pixel constituting the RGB image group; and The second mapping processing unit maps the spectral information of the spectral information mapping image group to the three-dimensional model according to the posture of the RGB camera, thereby generating the spectral three-dimensional model.

11. The image processing device according to claim 2, in, The spectrum three-dimensional model generating unit comprises: a three-dimensional reconfiguration unit that generates, based on the RGB image group and the spectral image group, a hybrid three-dimensional model consisting of a three-dimensional point cloud mixed with RGB information and the spectral information, attitude information of an RGB camera that captures the RGB image, attitude information of a spectral camera that captures the spectral image, and a range image group corresponding to each of the RGB image group and the spectral image group; a first mapping processing unit, which maps the spectral information of the spectral image group to each of the RGB image groups based on the posture information of the RGB camera, the posture information of the spectral camera and the range image group, thereby generating a spectral information mapping image group; and The second mapping processing unit maps the spectral information to the hybrid three-dimensional model based on the spectral information mapping image group to generate the spectral three-dimensional model.

12. The image processing device according to claim 2, in, The spectrum three-dimensional model generating unit comprises: A distance image generating unit, which generates a real scaled distance image through stereo matching based on the RGB image group and the spectral image group; a three-dimensional reconfiguration unit, which generates an unscaled three-dimensional model and an unscaled range image based on the RGB image group; a rescaling processing unit, for rescaling the unscaled three-dimensional model into a true scaled three-dimensional model by comparing the true scaled range image with the unscaled range image; a first mapping processing unit, mapping the spectral information of the spectral image group to each of the RGB image groups based on the real scaled distance image, thereby generating a spectral information mapping image group; and The second mapping processing unit generates the spectral three-dimensional model by mapping the spectral information of the spectral information mapping image group to the real scaled three-dimensional model.

13. The image processing device according to claim 12, in, In addition to the real scaled distance image through the stereo matching, the distance image generating unit further generates relative posture information between an RGB camera capturing the RGB image and a spectral camera capturing the spectral image based on the RGB image group and the spectral image group. In addition to the unscaled 3D model and the unscaled range image, the 3D reconfiguration unit also generates pose information of the RGB camera based on the RGB image group, and The rescaling processing unit rescales the unscaled three-dimensional model into the real scaled three-dimensional model by comparing the real scaled range image with the unscaled range image based on the posture information of the RGB camera and the relative posture information.

14. The image processing device according to claim 2, in, The optical axis of the RGB camera capturing the RGB image and the optical axis of the spectral camera capturing the spectral image are adjusted to be coaxial, and The spectrum three-dimensional model generating unit comprises: a three-dimensional reconfiguration unit, which generates the three-dimensional model and the pose information of the RGB camera based on the RGB image group; and The first mapping processing unit generates the spectral three-dimensional model by mapping the spectral information of the spectral image group onto the three-dimensional model.

15. The image processing device according to claim 14, in, The spectral three-dimensional model generating unit further comprises: a second mapping processing unit, generating a spectral information mapping image group by mapping the spectral information of the spectral image group to the RGB image group; The first mapping processing unit generates the spectral three-dimensional model by mapping the spectral information of the spectral information mapping image group to the three-dimensional model.

16. The image processing apparatus according to claim 1, further comprising: The attention object spectrum information extraction unit extracts the spectrum information of the attention object from the spectrum information.

17. The image processing device according to claim 16, in, The object of interest spectral information extraction unit extracts the spectral information of the object of interest from any of the following items: the spectral image group; a spectral information mapping image group, in which the spectral information of the spectral image group is mapped onto the spatial image group; And the spectral three-dimensional model.

18. An image processing method comprising the following steps: A spectral three-dimensional model is generated in which the spectral information is mapped onto the three-dimensional model based on a spectral image group consisting of a plurality of spectral images having spectral information of a predetermined space and a spatial image group consisting of a plurality of spatial images used to generate the three-dimensional model of the predetermined space.

19. A program for causing a computer to: A spectral three-dimensional model generating unit generates a spectral three-dimensional model based on a spectral image group consisting of a plurality of spectral images having spectral information of a predetermined space and a spatial image group consisting of a plurality of spatial images for generating a three-dimensional model of the predetermined space, wherein the spectral information is mapped onto the three-dimensional model.

20. An image processing device, comprising: a spectral three-dimensional model generating unit, which generates a spectral three-dimensional model based on a spectral image group consisting of a plurality of spectral images having spectral information of a predetermined space and a spatial image group consisting of a plurality of spatial images for generating a three-dimensional model of the predetermined space, wherein the spectral information is mapped onto the three-dimensional model; as well as The attention object spectrum information extraction unit extracts the spectrum information of the attention object from the spectrum information.

Images