Reliability calculation device, three-dimensional information creation device and reliability calculation method

Abstract

To calculate the reliability of a ranging value.SOLUTION: A three-dimensional information processing device includes: a first acquisition part for acquiring first point group data being point group data of a subject acquired with a first zoom magnification by a first point group data acquisition device; a second acquisition part for acquiring second point group data being point group data of the subject acquired with second zoom magnification by a second point group data acquisition device, wherein the second point group data acquisition device acquires the point group data with an angle of view which at least partially and mutually overlaps an angle of view at which the first point group data acquisition device acquires the first point group data; an identification part for identifying overlapping parts in a point group included in the first point group data and a point group included in the second point group data; a calculation part for calculating a contraction / expansion magnification of the second point group data with respect to the first point group data in the case that the identified overlapping parts are abutted; and a determination part for determining whether or not the calculated contraction / expansion magnification is smaller than a prescribed threshold.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The present invention relates to a reliability calculation device, a three-dimensional information creation device, and a reliability calculation method. [Background technology]

[0002] Conventionally, there has been a method of calculating the distance to an object by irradiating a distance measuring light to the object, receiving the reflected light reflected from the object, and measuring the time from irradiating the light to receiving the light. These distance measuring methods are widely known as the ToF (Time of Flight) method. There is a technology that uses such a ToF distance measuring technology to observe the distance to the object from multiple viewpoints and generate three-dimensional information of the object by synthesizing the distance information to the object observed from the multiple viewpoints (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 7-174538 Summary of the Invention [Problem to be solved by the invention]

[0004] A limitation of the ToF distance measurement technology is the random error caused by shot noise. In addition, when the object is made of a material that easily absorbs near-infrared rays or transmits infrared rays, the ToF distance measurement technology has a problem that the distance measurement light irradiated on the object is not reflected accurately, resulting in an inaccurate distance measurement value. Furthermore, the ToF distance measurement technology has a problem that the distance measurement value is inaccurate due to the optical path difference of the multipath.

[0005] When generating three-dimensional information of a subject based on a distance image including inaccurate distance measurements, it is possible to manually correct abnormal values. Since manual correction is time-consuming, if the reliability of the distance measurement value for each pixel can be known, it is possible to generate three-dimensional information after excluding the pixel value. However, it is not easy to determine the reliability of the distance measurement value. Therefore, there is a demand for calculating the reliability of the distance measurement value for each pixel in order to generate accurate three-dimensional information.

[0006] The present invention has been made in view of the above circumstances, and has an object to provide a reliability calculation device, a three-dimensional information creation device, and a reliability calculation method that are capable of calculating the reliability of a distance measurement value. [Means for solving the problem]

[0007] [1] One aspect of the present invention is a reliability calculation device that includes a distance image acquisition unit that acquires a distance image having distance information at each coordinate in a two-dimensional coordinate system; an IR image acquisition unit that acquires an IR image having information corresponding to the distance image, the IR image having information about the amount of infrared light received at each coordinate of the distance image; an ideal light intensity calculation unit that calculates an ideal light intensity based on the distance information at each coordinate of the acquired distance image by referring to correspondence data, which is data that measures in advance the correspondence relationship between the distance to the evaluation chart and the amount of received light using an evaluation chart having a specific reflectance; and a calculation unit that calculates a reliability at each coordinate based on a result of comparing the ideal light intensity obtained by calculation at each coordinate with the amount of infrared light received included in the IR image.

[0008] [2] Also, in one aspect of the present invention, in the reliability calculation device described in [1] above, the evaluation chart is a gray chart having a plurality of different gray levels, the correspondence data is data in which distance and received light amount are associated with each other for each gray level, and the device further includes a selection unit that selects an appropriate gray level from the plurality of gray levels based on distance information at each coordinate of the acquired distance image, and the ideal light amount calculation unit calculates the ideal light amount based on the gray level data selected by the selection unit.

[0009] [3] Also, one aspect of the present invention is a three-dimensional information creation device that creates three-dimensional information using the reliability calculation device described in [1] or [2] above, wherein the distance image acquisition unit acquires a first distance image having distance information from a first point to a subject and a second distance image having distance information from a second point different from the first point to the subject, the IR image acquisition unit acquires a first IR image corresponding to the first distance image and a second IR image corresponding to the second distance image, compares the reliability based on the first distance image and the first IR image with the reliability based on the second distance image and the second IR image, and creates three-dimensional information based on the distance image with a higher reliability.

[0010] [4] Moreover, one aspect of the present invention is a three-dimensional information creation device as described in [3] above, further comprising a polarized image acquisition unit that acquires a polarized image having polarization information of the subject at each coordinate of the distance image, the information corresponding to the distance image, a mirror flag assignment unit that assigns a mirror flag for each coordinate based on the acquired polarized image, a noise removal unit that removes distance information at the coordinates to which the mirror flag is assigned from the distance information that the distance image has at each coordinate, and a surface identification unit that identifies a surface based on the acquired polarized image, and creates three-dimensional information by smoothing out unevenness of the surface by complementing the removed distance information within the range identified as the surface.

[0011] [5] Also, one aspect of the present invention is a reliability calculation method including a distance image acquisition step of acquiring a distance image having distance information at each coordinate of a two-dimensional coordinate system; an IR image acquisition step of acquiring an IR image having information corresponding to the distance image, the IR image having information about the amount of infrared light received at each coordinate of the distance image; an ideal light intensity calculation step of calculating an ideal light intensity based on the distance information at each coordinate of the acquired distance image by referring to correspondence data, which is data obtained by previously measuring the correspondence relationship between the distance to the subject and the amount of received light using a subject having a specific reflectance; and a calculation step of calculating reliability at each coordinate based on a result of comparing the ideal light intensity obtained by calculation with the amount of infrared light received included in the IR image at each coordinate. Effect of the Invention

[0012] According to the present invention, the reliability of the distance measurement can be calculated. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a first diagram for explaining an overview of a three-dimensional information generation system according to an embodiment. [Diagram 2] FIG. 2 is a second diagram for explaining the outline of the three-dimensional information generation system according to the embodiment. [Diagram 3] 1 is a diagram for explaining a method for generating three-dimensional information by a three-dimensional information generation system according to an embodiment. FIG. [Figure 4] 4 is a first flowchart for explaining the operation of the three-dimensional information generation system according to the embodiment. [Diagram 5] 11 is a second flowchart illustrating the operation of the three-dimensional information generation system according to the embodiment. [Figure 6] 11 is a third flowchart for explaining the operation of the three-dimensional information generation system according to the embodiment. [Figure 7] FIG. 1 is a first diagram for explaining a method for determining an ideal IR value according to an embodiment. [Figure 8]FIG. 2 is a second diagram for explaining a method for determining an ideal IR value according to the embodiment. [Figure 9] 13 is a fourth flowchart for explaining the operation of the three-dimensional information generation system according to the embodiment. [Figure 10] 13 is a fifth flowchart illustrating the operation of the three-dimensional information generation system according to the embodiment. [Figure 11] FIG. 11 is a diagram for explaining a feature point according to the embodiment. [Figure 12] 10 is a sixth flowchart illustrating the operation of the three-dimensional information generation system according to the embodiment. [Figure 13] FIG. 2 is a functional configuration diagram showing an example of a functional configuration of a three-dimensional information generation system according to an embodiment. [Figure 14] FIG. 2 is a functional configuration diagram showing an example of a functional configuration of a reliability calculation device according to an embodiment. [Figure 15] FIG. 2 is a functional configuration diagram illustrating an example of a functional configuration of a reliability calculation unit according to the embodiment. [Figure 16] 1 is a functional configuration diagram showing an example of a functional configuration of a three-dimensional information creation device according to an embodiment; [Figure 17] 1 is a block diagram showing an example of an internal configuration of a reliability calculation device and a three-dimensional information creation device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] [Embodiment] A reliability calculation device, a three-dimensional information creation device, and a reliability calculation method according to the aspects of the present invention will be described in detail below with reference to the accompanying drawings, with preferred embodiments presented. Note that the embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment. In addition, "based on XX" in this application means "based on at least XX", and includes cases based on other elements in addition to XX. In addition, "based on XX" is not limited to cases where XX is directly used, but also includes cases where XX is based on something that has been calculated or processed. "XX" is any element (for example, any information). In addition, in the following drawings, the scale and number of each structure may be different from the scale and number of the actual structure in order to make each configuration easier to understand.

[0015] [Outline of the 3D information generation system] First, an overview of a three-dimensional information generation system 1 according to an embodiment will be described with reference to FIGS. 1 to 3. FIG.

[0016] FIG. 1 is a first diagram for explaining an overview of a three-dimensional information generating system according to an embodiment. The three-dimensional information generating system 1 generates three-dimensional information of a subject S. The three-dimensional information generating system 1 includes one or more imaging devices 10. The imaging device 10 is a device capable of acquiring a distance image, an IR image, a polarized image, and an RGB image. The distance image, the IR image, the polarized image, and the RGB image may have the same optical axis. That is, the imaging device 10 has a ToF (Time of Flight) sensor for acquiring a distance image and an IR image, a polarization sensor for acquiring a polarized image, and an image sensor for acquiring an RGB image, and each sensor may be disposed at a position where the light incident on the lens is incident on each sensor by splitting the optical path of the light. Note that this embodiment is not limited to an example of the case where the light has the same optical axis, and the distance image, the IR image, the polarized image, and the RGB image may have two-dimensional coordinate systems corresponding to each other.

[0017] The imaging device 10 acquires distance information from the imaging device 10 to the subject S, for example, using a ToF method. The imaging device 10 acquires distance information to the subject S from a plurality of points (i.e., from multiple viewpoints) and generates three-dimensional information of the subject S by synthesizing the acquired distance information. Note that, according to the ToF method, infrared light is irradiated onto the subject S, and the distance to the subject S is measured based on the time it takes for the reflected light to be received. In the following description, infrared light is irradiated by a ToF sensor, and information on the amount of light received within a predetermined time may be referred to as an IR image.

[0018] Note that the imaging device 10 may continuously capture frame images while moving from one imaging point to another, for example, as indicated by an arrow in the figure, to continuously obtain distance information from a plurality of points to the subject S. The movement of the imaging device 10 may be determined based on, for example, acceleration, angular velocity, and the like measured by an inertial measurement unit (IMU) (not shown) included in the imaging device 10.

[0019] Also, the imaging device 10 may be arranged in advance at two points shown in the figure, and the subject S may be imaged from each point to acquire distance information from the multiple points to the subject S. In this case, it is preferable for the three-dimensional information generation system 1 to acquire the positional relationship of the multiple imaging devices 10 in advance. When the three-dimensional information generation system 1 includes multiple imaging devices 10, the number of imaging devices 10 is not limited, and using more imaging devices 10 makes it possible to acquire more information and generate more accurate three-dimensional information.

[0020] FIG. 2 is a second diagram for explaining an overview of the three-dimensional information generating system according to the embodiment. The diagram shows an example in which the three-dimensional information generating system 1 generates three-dimensional information using one imaging device 10. As shown in the diagram, a marker MK is placed in front of a subject S. The marker MK is used to specify a world coordinate system. Therefore, it is preferable that the marker MK is one whose position and orientation can be identified by being imaged by the imaging device 10. The marker MK may be, for example, an identification code such as a QR code (registered trademark). The marker MK may be, for example, an image showing the identification code printed on a medium such as paper or plastic.

[0021] The imaging device 10 captures the marker MK in addition to the subject S. The imaging device 10 may capture the subject S and the marker MK separately and independently (i.e., an image in which the subject S is captured and an image in which the marker MK is captured are separate frame images), or may capture an image in which both the subject S and the marker MK are captured. For example, the imaging device 10 captures the subject S from multiple viewpoints after capturing the marker MK. A world coordinate system may be obtained with the position in which the marker MK is captured as the origin. That is, the three-dimensional information generating system 1 specifies the amount of movement of the imaging device 10 from the point in which the marker MK is captured based on the acceleration, angular velocity, etc. measured by an inertial measurement device (not shown), and specifies the point (coordinates and orientation) where the imaging device 10 captured the image. Note that in this embodiment, it is not necessarily required to use the marker MK, and the world coordinate system may be created by other methods.

[0022] 3 is a diagram for explaining a method for generating three-dimensional information by the three-dimensional information generating system according to the embodiment. The three-dimensional information generating system 1 calculates the reliability for each pixel and each surface from the acquired distance information of the subject S. The method for calculating the reliability will be described later in detail.

[0023] 3A shows an example in which the subject S is a cylindrical subject S1. The three-dimensional information of the subject S1 includes a surface marked with a reference number 1 and a surface marked with a reference number 2. The three-dimensional information for each surface is acquired from an image having a high reliability for that surface among images captured from multiple viewpoints by the imaging device 10. The three-dimensional information for each surface may be acquired from different images. The three-dimensional information generation system 1 generates the three-dimensional information of the subject S1 by synthesizing the three-dimensional information for each surface acquired from different images.

[0024] 3B shows an example in which the subject S is a cubic subject S2. The three-dimensional information of the subject S2 includes a surface marked with a reference number 1, a surface marked with a reference number 2, and a surface marked with a reference number 3. Similarly, for the subject S2, three-dimensional information for each surface is acquired from different images, and the three-dimensional information for each surface is synthesized to generate the three-dimensional information of the subject S2.

[0025] [Operation of 3D information generation system] Next, an example of the operation of the three-dimensional information generation system 1 according to the embodiment will be described with reference to FIGS.

[0026] 4 is a first flowchart for explaining the operation of the three-dimensional information generation system according to the embodiment. The figure shows an example of a series of operations performed by the three-dimensional information generation system 1. In the following description, a person who generates three-dimensional information of a subject S by operating the three-dimensional information generation system 1 may be referred to as an operator.

[0027] (Step S10) First, if the subject S is made of a translucent material, a polarized light (not shown) is turned on. The polarized light is included in the three-dimensional information generation system 1, and is installed in a position where it can illuminate the subject S. The polarized light may be provided in the imaging device 10. When it is determined that the subject S is made of a translucent material, for example, the on / off control of the polarized light may be performed manually by an operator. Also, the on / off control of the polarized light may be performed mechanically according to the results of analysis based on the IR image and the polarized image acquired by the imaging device 10.

[0028] (Step S11) The three-dimensional information generation system 1 initializes the camera viewpoint and the number of frames. Specifically, the values ​​are reset to camera viewpoint N=0 and frame number FrameCnt=0.

[0029] (Step S12) The imaging device 10 captures an image of the subject S. This imaging includes obtaining a distance image, an IR image, a polarized image, and an RGB image. Hereinafter, when there is no need to distinguish between the distance image, the IR image, the polarized image, and the RGB image captured by the imaging device 10, they may be simply referred to as images.

[0030] (Step S13) The three-dimensional information generation system 1 generates a reliability map of the captured image. The reliability map is information in which a reliability is assigned to each coordinate of the image. The reliability map is generated for each of a plurality of images captured successively in time. A method for creating the reliability map will be described later with reference to FIG. 5 etc. The three-dimensional information generation system 1 generates three-dimensional information of the subject S by selectively synthesizing suitable distance information based on the created reliability map.

[0031] (Step S14) The three-dimensional information generating system 1 displays the generated three-dimensional information (3D model) of the subject S. The three-dimensional information of the subject S is displayed, for example, on a display (not shown). As another specific example, the three-dimensional information of the subject S may be projected by a predetermined method in a visually recognizable manner. The operator checks the displayed three-dimensional information of the subject S.

[0032] (Step S15) The operator checks the displayed three-dimensional information of the subject S and then judges whether or not to end the three-dimensional information generation process. For example, if the operator judges that noise is superimposed on the displayed three-dimensional information and that the three-dimensional shape of the subject S is not adequately represented (i.e., different from the real thing), the operator can continue the three-dimensional information generation process. If it is judged that the three-dimensional information generation process is to be ended (i.e., step S15; YES), the process proceeds to step S16. If it is judged that the three-dimensional information generation process is not to be ended (i.e., the three-dimensional information generation process is to be continued) (i.e., step S15; NO), the process proceeds to step S17.

[0033] (Step S16) The three-dimensional information generation system 1 exports the three-dimensional information into a target file format. The exported file is stored in a predetermined storage unit.

[0034] (Step S17) When the generation process of the three-dimensional information is continued, the imaging process is performed again by the imaging device 10. The imaging process may be performed by moving the position of the imaging device 10, or may be performed without moving the position of the imaging device 10. When the distance information cannot be acquired correctly due to the imaging position, it is preferable to move the position of the imaging device 10, and when the distance information cannot be acquired correctly regardless of the imaging position, it is not necessary to move the position of the imaging device 10. Whether or not the position of the imaging device 10 has been moved may be determined based on, for example, the acceleration, angular velocity, etc. measured by an inertial measurement device (not shown) provided in the imaging device 10. When it is determined that the position of the imaging device 10 has been moved (i.e., step S17; YES), the process proceeds to step S18. When it is determined that the position of the imaging device 10 has not been moved (i.e., step S17; NO), the process proceeds to step S19.

[0035] (Step S18) The three-dimensional information generation system 1 performs a process of incrementing the value of the camera viewpoint. Specifically, a process of setting the camera viewpoint N+=1 is performed.

[0036] (Step S19) The three-dimensional information generation system 1 performs a process of incrementing the value of the frame number. Specifically, a process of setting the frame number FrameCnt+=1 is performed.

[0037] 5 is a second flowchart for explaining the operation of the three-dimensional information generating system according to the embodiment. An example of the reliability map generating process will be explained with reference to the same figure. The reliability map generating process is the process described above as step S13.

[0038] (Step S20) The three-dimensional information generation system 1 acquires information about a reference point previously set as world coordinates. The reference point may be the position of the marker MK described with reference to Fig. 2. If the position of the marker MK has not yet been acquired, the three-dimensional information generation system 1 may instruct the operator to capture an image of the marker MK.

[0039] (Step S21) The three-dimensional information generation system 1 acquires information about acceleration and rotation from an inertial measurement device (not shown) provided in the imaging device 10.

[0040] (Step S22) Here, if the acquired acceleration is equal to or greater than the threshold, camera shake may have occurred and the image information may be incorrect (information about the subject S may not have been acquired properly). Therefore, if the acquired acceleration is equal to or greater than the threshold (i.e., step S22; YES), the three-dimensional information generation system 1 decides not to use the image in the three-dimensional information generation process and terminates the reliability map generation process. If the acquired acceleration is not equal to or greater than the threshold (i.e., step S22; NO), the three-dimensional information generation system 1 continues the reliability map generation process and proceeds to step S23.

[0041] (Step S23) The three-dimensional information generation system 1 determines the current camera position based on the displacement from the previous camera position and the information on the reference point acquired in step S20. The previous camera position may be the camera position at which the frame image obtained the instant before the frame image currently being processed among the consecutive frame images was captured. The current camera position may be the camera position at which the frame image currently being processed was captured.

[0042] (Step S24) The three-dimensional information generation system 1 creates a reliability when the subject S is imaged from the determined camera position. Specifically, the three-dimensional information generation system 1 creates a reliability map based on the distance image, the IR image, and the polarized image. A specific method for creating the reliability map will be described later in detail with reference to FIG. 6 etc.

[0043] (Step S25) The three-dimensional information generation system 1 divides the subject S into parts for each camera position based on the created reliability map. Details of the part division process for the subject S will be described later with reference to FIG.

[0044] 6 is a third flowchart for explaining the operation of the three-dimensional information generating system according to the embodiment. With reference to the same figure, an example of detailed processing of the reliability map generating process will be explained. The detailed processing of the reliability map generating process is, in other words, the process described above as step S24.

[0045] (Step S30) First, the three-dimensional information generating system 1 judges whether the distance value and IR value of each coordinate are valid based on the distance image and the IR image. The judgment of whether the value is valid may be performed based on whether the value is within a predetermined threshold. The distance value and IR value of each coordinate are not valid when they are abnormal values, for example, when the target pixel is a so-called dead pixel, or when noise is superimposed and the value becomes abnormal. When the distance value and IR value of each coordinate are valid (i.e., step S30; YES), the three-dimensional information generating system 1 advances the process to step S31. When the distance value and IR value of each coordinate are not valid (i.e., step S30; NO), the three-dimensional information generating system 1 advances the process to step S31.

[0046] (Step S31) Next, the three-dimensional information generating system 1 calculates an ideal IR value (amount of received light) based on the distance value for each pixel. Here, if the infrared reflectance of the subject S is constant, the IR value also differs as the distance to the subject S changes. Specifically, the amount of received light decreases as the distance to the subject S increases, and increases as the distance to the subject S decreases. Also, if the distance to the subject S is constant, the amount of received light (IR value) also differs depending on the infrared reflectance of the subject S. Specifically, the amount of received light decreases as the infrared reflectance of the subject S decreases, and increases as the infrared reflectance of the subject S increases.

[0047] According to this embodiment, an ideal IR value based on the distance value to the subject S is obtained by using the characteristic that the IR value varies depending on the distance and the infrared reflectance of the subject S, and the reliability is calculated based on the ideal IR value and the actual IR value. Specifically, the reliability may be the ratio of the actual IR value to the ideal IR value. Hereinafter, a specific example of a method for calculating the ideal IR value (amount of received light) will be described with reference to Figs. 7 and 8.

[0048] 7 is a first diagram for explaining how to obtain an ideal IR value according to an embodiment. In this figure, the horizontal axis indicates distance, and the vertical axis indicates IR value (amount of received light). In addition, this figure shows measurement results for three subjects with different infrared reflectances as lines W1, W2, and W3. Line W1 has a lower infrared reflectance, and line W3 has a higher infrared reflectance. In the figure, points expressed in black are points where the IR value is sufficient. On the other hand, points expressed in white are points where the IR value is insufficient.

[0049] The information shown in the figure is information obtained by actual measurement using an evaluation chart. The evaluation chart may be a gray chart having a plurality of different gray levels. When the evaluation chart is a gray chart, the line W1 is a chart having a gray level closer to black, and the line W3 is a chart having a gray level closer to white. Note that the figure shows an example of a case where an evaluation chart having three infrared reflectances is used, but when a gray chart with 16 gradations is used, for example, 16 lines are drawn. In addition, the information shown in the figure is actually measured, but this embodiment is not limited to an example of a case where information is obtained by actual measurement, and it is sufficient if the correspondence between the distance and the IR value can be obtained by some method.

[0050] In the illustrated example, distance values ​​1 to 11 indicate relative distances (i.e., without units) to simplify the explanation. Actual distance values ​​may be used as a result of actual measurements. The distance values ​​do not need to be spaced at equal intervals, and it is sufficient that information exists for multiple points. For example, in a range where the change in the amount of received light is small, the intervals do not need to be the same as in a range where the change in the amount of received light is large.

[0051] As a preliminary step, the three-dimensional information generating system 1 acquires correspondence information between distance and amount of received light as shown in the drawing in advance by conducting an experiment or the like, and stores the acquired information in a storage unit. Note that this embodiment is not limited to an example of acquiring the correspondence information between distance and amount of received light by an experiment or the like, and the correspondence information may be acquired by a simulation or the like.

[0052] In the reliability calculation stage, the three-dimensional information generating system 1 calculates an ideal IR value for each pixel based on the acquired distance image. First, the three-dimensional information generating system 1 selects two pieces of data that are closest to the distance value at the coordinates to be determined for reliability from among the data stored in advance. In the selection of data, data with a high IR value is selected. Also, data with a low infrared reflectance is selected. For example, in the illustrated example, a search is performed to see if there is suitable data on the line W1 with the lowest infrared reflectance, and if no suitable data is found, a search is performed to see if there is suitable data on the line W2 with the next lowest infrared reflectance after the line W1. For example, when the distance value of the pixel to be processed is 3.5, there is no suitable data on the line W1 (white dots), but there are two suitable pieces of data on the line W2 (black dots). More specifically, when the distance value of the pixel to be processed is 3.5, data with a distance value of 3 and data with a distance value of 4 on the line W2 are selected.

[0053] In addition, when selecting data, if the reflectance of the subject S from which three-dimensional information is to be generated is known in advance, data close to the reflectance may be selected. The reflectance of the subject S may be selected by an operator, or may be determined based on an IR image, an RGB image, or the like captured by the imaging device 10.

[0054] FIG. 8 is a second diagram for explaining how to obtain an ideal IR value according to an embodiment. In the figure, the horizontal axis indicates distance, and the vertical axis indicates IR value (amount of received light). In addition, in the figure, two pieces of data selected based on distance values ​​are shown as points P1 and P2. In association with the above description in FIG. 7, point P1 is, for example, a point on line W2 whose distance value is 3, and point P2 is, for example, a point on line W2 whose distance value is 4. In addition, line L1 is a line segment connecting points P1 and P2. The ideal IR value (point marked as CurrentRecommendIR in the figure) is obtained based on the distance value (point marked as CurrentDepth in the figure) at the coordinates to be calculated for the ideal IR value and line segment L1.

[0055] Returning to FIG. 6, the process after the ideal IR value is determined will be described.

[0056] (Step S33) The three-dimensional information generation system 1 calculates the reliability based on the calculated ideal IR value. The reliability may be a ratio between the ideal IR value at the coordinates for which the reliability is calculated and the distance value at the coordinates for which the reliability is calculated. If the actual IR value is greater than the ideal IR value, the upper limit of the reliability may be set to 100%, and any value exceeding 100% may be truncated.

[0057] (Step S32) If it is determined in step S30 that the pixel value is not a valid value, the reliability is set to 0.

[0058] (Step S34) The three-dimensional information generating system 1 stores the reliability calculated in steps S31 and S32 for each pixel. In the following description, information having reliability for each pixel on the two-dimensional coordinate system is also referred to as a reliability map.

[0059] (Step S35) The three-dimensional information generation system 1 removes from the distance image and the IR image pixels in the created reliability map whose reliability is lower than a predetermined threshold value.

[0060] (Step S36) The three-dimensional information generating system 1 assigns information based on the polarized image to the reliability map. The assignment of information based on the polarized image will be described later with reference to FIG. 9. Here, depending on the reflectance of the subject S and the amount of light irradiated onto the subject S, specular reflection may occur, and in such cases, the distance information may be inaccurate. By performing the processing in this step, a specular reflection flag is assigned to pixels that are estimated to be specular. Also, depending on the subject S, a transparent material may be used, and in such cases, the distance information may also be inaccurate. By performing the processing in this step, a transparent flag is assigned to pixels that are estimated to be using a transparent material.

[0061] (Step S37) The three-dimensional information generation system 1 removes pixels to which at least one of the specular reflection flag and the transparent flag has been assigned from the distance image and the IR image, regardless of the reliability.

[0062] (Step S38) The three-dimensional information generation system 1 performs a process of correcting unevenness for each surface of the subject S. Specifically, the three-dimensional information generation system 1 identifies the surface of the subject S, and corrects unevenness by uniformly rewriting distance values ​​along the surface to smooth values. Rewriting distance values ​​may mean complementing pixel values ​​removed from the distance image and IR image based on pixel values ​​in the vicinity of the pixel. Rewriting distance values ​​may also mean correcting pixel values ​​on the same surface based on pixel values ​​in the vicinity.

[0063] 9 is a fourth flowchart for explaining the operation of the three-dimensional information generating system according to the embodiment. With reference to the same figure, an example of the process of adding information based on a polarized image will be explained. Adding information based on a polarized image is the process described above as step S36.

[0064] (Step S40) First, a polarized image is captured by the imaging device 10. This polarized image corresponds to the distance image used to create the reliability map in the above description.

[0065] (Step S41) The three-dimensional information generation system 1 separates the specular and diffuse reflection components based on the acquired polarized image. Here, coordinates with a large degree of linear polarization (DoLP; Degree of Linear Polarization) are likely to be specular. Therefore, the three-dimensional information generation system 1 assigns a specular flag to coordinates with a large DoLP.

[0066] (Step S42) Next, the three-dimensional information generating system 1 obtains the polarization direction from the angle of linear polarization (AoLP; Angle of Linear Polarization). The three-dimensional information generating system 1 identifies the surfaces of the subject S and the normals to the surfaces based on the obtained polarization direction and DoLP. The identified surfaces are labeled (clustered). In multiple consecutive images, surfaces estimated to be the same are assigned the same label (class).

[0067] Due to the relationship between DoLP and reflection angle (incidence angle), two reflection angles are estimated except for Brewster angle (polarization angle). To resolve this ambiguity, the closest one may be selected by performing normal estimation based on tangential depth based on information from past frame images and information from the current frame image.

[0068] (Step S43) The three-dimensional information generation system 1 removes noise from the identified surface.

[0069] (Step S44) Next, the three-dimensional information generating system 1 uses polarized light to determine whether or not the subject S is made of a transparent material. However, some imaging devices 10 have polarized light and some do not. Therefore, the three-dimensional information generating system 1 first determines whether or not the imaging device 10 has polarized light. If the imaging device 10 has polarized light (i.e., step S44; YES), the process proceeds to step S45 to perform a process of determining whether or not the material is transparent. If the imaging device 10 does not have polarized light (i.e., step S44; NO), the process ends.

[0070] (Step S45) The three-dimensional information generating system 1 judges whether the subject S is a transparent material. Since many transparent materials have changing optical properties, it is possible to judge whether the material is a transparent material by irradiating it with polarized light. The judgment of whether the material is a transparent material may be performed for each pixel or for each surface. Specifically, a PassiveIR image is used to judge whether the material is a transparent material. Specifically, the PassiveIR image may be an image obtained by capturing ambient light without irradiating infrared light by a ToF sensor. In the case of an imaging device 10 having a polarized light, the influence of the reflected light from the polarized light appears in the PassiveIR image. Therefore, the three-dimensional information generating system 1 judges whether the material is a transparent material by determining the reflection intensity from the light source based on the PassiveIR image.

[0071] (Step S46) If the three-dimensional information generation system 1 determines that the material is a transparent material (i.e., step S46; YES), the process proceeds to step S47. If the three-dimensional information generation system 1 determines that the material is not a transparent material (i.e., step S46; NO), the process ends.

[0072] (Step S47) The three-dimensional information generation system 1 assigns a transparent flag to each pixel or surface determined to be a transparent material. Depending on the shape of the transparent material, the change in optical characteristics may only appear at the outline. That is, even if the entire surface is made of a transparent material, the periphery of the edge portion of the surface may be determined to be a transparent material, and the center portion of the surface may not be determined to be a transparent material. Therefore, a transparent flag may be assigned to all pixels on the same surface as a surface that has a pixel determined to be a transparent material.

[0073] 10 is a fifth flowchart for explaining the operation of the three-dimensional information generating system according to the embodiment. With reference to the same figure, an example of the parts processing for each camera viewpoint will be explained. The parts processing for each camera viewpoint is the processing described above as step S25.

[0074] (Step S50) First, the three-dimensional information generation system 1 obtains a reliability map of the current frame and a reliability map of a past frame.

[0075] (Step S51) Next, the three-dimensional information generation system 1 obtains surface elements for each of the current frame and the past frame. Whether or not an element is a surface is determined based on a distance image corresponding to the acquired reliability map.

[0076] (Step S52) The three-dimensional information generation system 1 identifies feature points based on the distance image. A feature point is a point that is characteristic among the points that make up a surface. For example, four feature points are identified for each surface. The three-dimensional information generation system 1 identifies the feature points of the current frame and the feature points of the past frame. The three-dimensional information generation system 1 finds similarities by matching the identified feature points of the current frame with the feature points of the past frame. Details of the feature points will be described with reference to FIG. 11.

[0077] FIG. 11 is a diagram for explaining feature points according to the embodiment. First, a specific example of feature points identified for each surface will be described with reference to FIG. 11(A). In the illustrated example, the subject S has at least a surface SF1, a surface SF2, and a surface SF3. The figure shows four feature points for the surface SF1. Specifically, the surface SF1 is rectangular, and vertices P11 to P14 of the rectangle are identified as feature points. Note that the number of feature points may be as many as necessary to identify the surface, and is not limited to the example of four points as illustrated. The number of feature points may be different for each surface. For example, a surface having a complex shape may have a large number of feature points.

[0078] FIG. 11(B) is a diagram for explaining matching between feature points of a current frame and feature points of a past frame. As shown in the figure, an imaging device 10 captures images of a subject S from a plurality of positions. In the illustrated example, the imaging device 10 captures images of the subject S from a position indicated by reference numeral 10-1 and a position indicated by reference numeral 10-2. Regardless of the position from which the image is captured, the feature points of the subject S are identified. The three-dimensional information generation system 1 finds similarities by matching the identified feature points of the current frame with the feature points of the past frames.

[0079] 10, the three-dimensional information generation system 1 calculates the displacement of the imaging device 10 based on the information acquired from the inertial measurement unit included in the imaging device 10 and the similarity calculated in step S52. The displacement of the imaging device 10 is the displacement from the point where the target past frame is captured to the point where the current frame is captured.

[0080] (Step S54) The three-dimensional information generating system 1 converts the three-dimensional information of the subject S in the current frame into coordinates of a world coordinate system with the reference point as the origin. The coordinates of the world coordinate system are those acquired in step S20.

[0081] (Step S55) The three-dimensional information generation system 1 searches for highly reliable frame images in the target frame images on a label-by-label basis (i.e., face-by-face basis), and uses the three-dimensional information of the relevant faces to synthesize three-dimensional information of the subject S. In other words, the three-dimensional information generation system 1 extracts highly reliable three-dimensional information for each face from different frame images, and generates three-dimensional information of the subject S by synthesizing the extracted three-dimensional information in the world coordinate system.

[0082] 12 is a sixth flowchart for explaining the operation of the three-dimensional information generating system according to the embodiment. A detailed example of the three-dimensional information synthesis process will be described with reference to the same figure. The three-dimensional information synthesis process is the process described above as step S55.

[0083] (Step S60) The three-dimensional information generation system 1 determines whether or not the label included in the target frame image is a new label. A new label is a label that has not appeared so far while the three-dimensional information generation system 1 is processing multiple frame images. If the label is new (i.e., step S60; YES), the three-dimensional information generation system 1 advances the process to step S61. If the label is not new (i.e., step S60; NO), the three-dimensional information generation system 1 advances the process to step S62.

[0084] (Step S61) When the label is new, the three-dimensional information generation system 1 places the three-dimensional information of the surface to which the label is assigned in the world coordinate system.

[0085] (Step S62) If the label is not new, the three-dimensional information generation system 1 compares the reliability of faces having the same label in the current frame and the past frame. The comparison of reliability may be a comparison of reliability of faces calculated based on the average reliability of each coordinate included in the face, etc.

[0086] (Step S63) If the reliability of faces having the same label in the current frame and the previous frame is the same (i.e., step S63; YES), the three-dimensional information generation system 1 advances the process to step S64. If the reliability of faces having the same label in the current frame and the previous frame is not the same (i.e., step S63; NO), the three-dimensional information generation system 1 advances the process to step S65.

[0087] (Step S64) If the reliability of faces with the same label is the same in the current frame and the previous frame, the label of the face with the larger area is placed. When comparing the areas of the same face, the face with the larger area is estimated to be an image of the face taken from the front, and is estimated to be more reliable information.

[0088] (Step S65) If the reliability of faces with the same label is not the same in the current frame and the previous frame, the label with the higher reliability is placed. This is because it is estimated that the higher reliability means that there is less noise superimposed and more accurate 3D information has been obtained.

[0089] [Functional configuration of the 3D information generation system] Next, an example of the functional configuration of a three-dimensional information generation system 1 according to an embodiment will be described with reference to Fig. 13 to Fig. 16. The three-dimensional information generation system 1 described below is an example of a specific configuration for realizing the three-dimensional information generation process as described above.

[0090] 13 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information generation system according to an embodiment. With reference to the diagram, an example of the functional configuration of the three-dimensional information generation system 1 will be described. The three-dimensional information generation system 1 includes an imaging device 10, an inertial measurement unit 20, a reliability calculation device 30, a reliability storage unit 40, and a three-dimensional information creation device 50.

[0091] The imaging device 10 captures an image of a subject S. The images captured by the imaging device 10 include a distance image, an IR image, a polarized image, and an RGB image. It is preferable that the two-dimensional coordinates of these images correspond to each other. The imaging device 10 captures images continuously. The continuous images captured by the imaging device 10 can also be called frame images. The imaging device 10 captures images at a predetermined frame rate. The predetermined frame rate may be, for example, 1 [FPS (frames per second)] or 24 [FPS].

[0092] The inertial measurement unit 20 measures the motion of the imaging device 10. The inertial measurement unit 20 may be configured as one of the components of the imaging device 10. Specifically, the inertial measurement unit 20 includes an acceleration sensor and a gyro sensor (not shown), and detects acceleration and angular velocity in three axes.

[0093] The reliability calculation device 30 calculates the reliability of each coordinate for each frame image based on the image captured by the imaging device 10 and the acceleration and angular velocity acquired by the inertial measurement device 20. Information on the reliability of each coordinate in each image is also referred to as a reliability map. A specific configuration of the reliability calculation device 30 will be described later with reference to FIG. 14.

[0094] The reliability storage unit 40 stores the reliability for each image calculated by the reliability calculation device 30. It can also be said that the reliability storage unit 40 stores a reliability map for a plurality of temporally consecutive images (frame images).

[0095] The three-dimensional information creation device 50 creates three-dimensional information of the subject S using a plurality of reliability maps calculated by the reliability calculation device 30 and stored in the reliability storage unit 40. The three-dimensional information creation device 50 identifies a portion having a high reliability for each configuration (e.g., a surface) of the subject S by cross-referencing a plurality of temporally consecutive reliability maps. The three-dimensional information creation device 50 creates three-dimensional information of the subject S by composing the identified portion having a high reliability into a world coordinate system.

[0096] FIG. 14 is a functional configuration diagram showing an example of the functional configuration of a reliability calculation device according to an embodiment. With reference to the same figure, an example of the functional configuration of the reliability calculation device 30 will be described. The reliability calculation device 30 includes a polarized image acquisition unit 311, an IR image acquisition unit 312, a distance image acquisition unit 313, a reliability calculation unit 314, a calibration information storage unit 315, a mirror surface flag assignment unit 316, a surface identification unit 317, a transparent material determination unit 318, a noise removal unit 319, a transparent flag assignment unit 320, an inertia information acquisition unit 331, an acceleration threshold information storage unit 332, a comparison unit 333, a displacement calculation unit 334, and a reliability output unit 340. Each of these functional units is realized, for example, by using an electronic circuit. Furthermore, each functional unit may include a storage unit such as a semiconductor memory or a magnetic hard disk device inside as necessary. Furthermore, each function may be realized by a computer having a CPU (Central Processing Unit) and software.

[0097] Some of the configurations described as the functional configuration of the reliability calculation device 30 may be included in other devices (eg, the three-dimensional information creation device 50) included in the three-dimensional information generation system 1.

[0098] The polarized image acquisition section 311 acquires a polarized image captured by the imaging device 10. The polarized image has polarization information of the subject S at each coordinate in a two-dimensional coordinate system.

[0099] The IR image acquisition unit 312 acquires an IR image captured by the imaging device 10. The IR image has information on the amount of infrared light received at each coordinate in a two-dimensional coordinate system. The IR image may be an IR image acquired by using a ToF sensor to acquire the amount of infrared light (i.e., ambient light) in a state where no irradiation light is irradiated.

[0100] Distance image acquisition unit 313 acquires a distance image (depth image) captured by imaging device 10. The distance image has distance information from imaging device 10 to subject S at each coordinate in a two-dimensional coordinate system. The distance information at each coordinate may be referred to as a distance value or a depth value.

[0101] The coordinates of the polarized image, IR image, and distance image correspond to each other. The number of pixels (resolution) of the polarized image, IR image, and distance image is preferably the same, but is not limited to the example where they are the same. In other words, the coordinates of the polarized image, IR image, and distance image do not need to correspond one-to-one, but may correspond to each other.

[0102] The imaging device 10 captures images of the subject S from a plurality of different points. The imaging device 10 may continue to capture images of the subject S from different imaging points. In the following description, for simplicity of description, a case will be described in which the imaging device 10 captures images of the subject S from each of a first point and a second point. A polarized image captured from the first point may be referred to as a first polarized image, an IR image captured from the first point may be referred to as a first IR image, and a distance image captured from the first point may be referred to as a first distance image. A polarized image captured from a second point different from the first point may be referred to as a second polarized image, an IR image captured from the second point may be referred to as a second IR image, and a distance image captured from the second point may be referred to as a second distance image.

[0103] The reliability calculation unit 314 acquires an IR image from the IR image acquisition unit 312, and acquires a distance image from the distance image acquisition unit 313. The reliability calculation unit 314 calculates the reliability for each pixel based on the acquired IR image and distance image. The reliability calculation is performed by referring to the corresponding data. The corresponding data is data obtained by measuring in advance the correspondence relationship between the distance to the evaluation chart and the amount of received light using an evaluation chart having a specific reflectance.

[0104] The calibration information storage unit 315 stores correspondence data measured in advance. The correspondence data can also be said to be a correspondence relationship between distance and IR value measured using an evaluation chart. The evaluation chart used for the measurement is preferably a gray chart having a plurality of different gray levels. When a gray chart is used for the measurement, the correspondence data is data in which the distance and the amount of received light are associated with each other for each gray level.

[0105] 15 is a functional configuration diagram showing an example of the functional configuration of the reliability calculation unit according to the embodiment. With reference to the same figure, an example of the functional configuration of the reliability calculation unit 314 will be described. The reliability calculation unit 314 includes a selection unit 3141, an ideal light amount calculation unit 3142, and a calculation unit 3143.

[0106] The selection unit 3141 acquires a distance image from the distance image acquisition unit 313. The selection unit 3141 selects a suitable gray level from among a plurality of gray levels based on distance information at each coordinate of the acquired distance image. The selection of a gray level also means the selection of data according to the infrared reflectance of the subject S. The selection of a gray level is performed by referring to the corresponding information stored in the calibration information storage unit 315. Note that when the infrared reflectance of the subject S is known, the selection unit 3141 may select a gray level according to the infrared reflectance of the subject S.

[0107] The ideal light amount calculation unit 3142 calculates the ideal light amount at each coordinate based on the data selected by the selection unit 3141 and the distance value of the distance image acquired by the distance image acquisition unit 313. The calculation of the ideal light amount is performed by referring to the corresponding data stored in the calibration information storage unit 315.

[0108] The calculation unit 3143 calculates the reliability (i.e., a reliability map) at each coordinate based on a result of comparing the ideal light amount obtained by calculation by the ideal light amount calculation unit 3142 at each coordinate with the amount of infrared light received included in the IR image acquired by the IR image acquisition unit 312. Returning to FIG. 14, the reliability map calculated by the reliability calculation unit 314 is output to the mirror surface flag assignment unit 316.

[0109] The mirror surface flag assigning unit 316 acquires a polarized image from the polarized image acquisition unit 311, and acquires a reliability map from the reliability calculation unit 314. The mirror surface flag assigning unit 316 assigns a mirror surface flag to the reliability map for each coordinate based on the acquired polarized image. Specifically, the mirror surface flag assigning unit 316 may assign a mirror surface flag to coordinates with a large DoLP.

[0110] The surface identifying section 317 acquires a polarized image from the polarized image acquiring section 311, and identifies a surface of the subject S based on the acquired polarized image. The surface identifying section 317 may obtain the polarization direction based on the AoLP, and identify the surface of the subject S based on the obtained polarization direction and the DoLP. Furthermore, the surface identifying section 317 performs labeling (clustering) for each identified surface.

[0111] The transparent material determination unit 318 acquires a polarized image from the polarized image acquisition unit 311, and determines whether or not the material is transparent based on the acquired polarized image. The determination of whether or not the material is transparent is preferably performed for each pixel. The result of the determination of whether or not the material is transparent performed for each pixel may be reflected on the surface identified by the surface identification unit 317.

[0112] The noise removal unit 319 obtains information on labeling performed for each surface from the surface identification unit 317, and obtains information on the reliability map after the mirror surface flag has been assigned from the mirror surface flag assignment unit 316. The noise removal unit 319 removes distance information at coordinates to which a polarization flag has been assigned, from among the distance information that the distance image has at each coordinate. Furthermore, the noise removal unit 319 complements the removed distance information within the range identified as a surface by the surface identification unit 317, thereby smoothing out unevenness of the surface.

[0113] The transparent flag assigning unit 320 obtains the reliability map after noise has been removed from the noise removal unit 319, and obtains information on the result of the determination as to whether or not the material is transparent from the transparent material determination unit 318. The transparent flag assigning unit 320 assigns a transparent flag to coordinates determined to be transparent materials among the coordinates of the reliability map after noise has been removed. The transparent flag assigning unit 320 outputs the reliability map after the transparent flag has been assigned to the reliability output unit 340.

[0114] The inertial information acquisition unit 331 acquires information on the motion of the imaging device 10 from the inertial measurement device 20. Specifically, the distance image acquisition unit 313 acquires information on the acceleration and angular velocity in three axes. The inertial information acquisition unit 331 may also acquire information on the timing at which an image is captured by the imaging device 10. The inertial information acquisition unit 331 acquires information on the motion of the imaging device 10 from when a first image is captured by the imaging device 10 to when a second image is captured by the imaging device 10 by acquiring information on the timing at which an image is captured by the imaging device 10.

[0115] The acceleration threshold information storage unit 332 stores information related to the acceleration threshold. The acceleration threshold is the limit of acceleration at which an image is recognized as normal. In other words, if the acceleration acquired by the inertial information acquisition unit 331 is equal to or greater than the acceleration threshold, it is estimated that the image is not normal (for example, that the image is blurred).

[0116] The comparison unit 333 compares the acceleration acquired by the inertial information acquisition unit 331 with the acceleration threshold value stored in the acceleration threshold value information storage unit 332. The comparison unit 333 outputs the comparison result to the displacement calculation unit 334. The comparison result indicates whether or not the image is normal.

[0117] The displacement calculation unit 334 acquires information on the motion of the imaging device 10 between frame images from the inertial information acquisition unit 331, and acquires information on the determination result as to whether the image is normal or not from the comparison unit 333. If the image is normal, the displacement calculation unit 334 calculates the displacement of the imaging device 10 between the frame images. The displacement calculation unit 334 outputs information on the displacement of the imaging device 10 between the frame images obtained as a result of the calculation to the reliability output unit 340. Note that if the image is not normal, the comparison unit 333 or the displacement calculation unit 334 may notify the reliability calculation unit 314, etc., to stop the calculation process of the reliability map.

[0118] The reliability output unit 340 acquires a reliability map from the transparent flag assignment unit 320, and acquires information related to the displacement of the imaging device 10 from the displacement calculation unit 334. The reliability output unit 340 associates the acquired information and stores it in the reliability storage unit 40. The reliability storage unit 40 stores the reliability map and information related to the displacement of the imaging device 10 for each frame image in association with each other. Note that the reliability storage unit 40 may also store an RGB image, a distance image, or the like corresponding to the reliability map in association with the reliability map, in addition to the reliability map and the information related to the displacement of the imaging device 10.

[0119] FIG. 16 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information creation device according to an embodiment. With reference to the same figure, an example of the functional configuration of a three-dimensional information creation device 50 will be described. The three-dimensional information creation device 50 includes a reliability information acquisition unit 51, a surface element identification unit 52, a similarity calculation unit 53, and a synthesis unit 55. Each of these functional units is realized, for example, by using an electronic circuit. Furthermore, each functional unit may include a storage means such as a semiconductor memory or a magnetic hard disk device inside as necessary. Furthermore, each function may be realized by a computer having a CPU and software.

[0120] The three-dimensional information creation device 50 generates three-dimensional information of the subject S based on the reliability of each of the multiple frame images stored in the reliability storage unit 40 and information related to the displacement of the imaging device 10. Specifically, the three-dimensional information creation device 50 compares the reliability at a first time point with the reliability at a second time point, and creates three-dimensional information based on a distance image with a high reliability. The three-dimensional information creation device 50 can generate highly reliable three-dimensional information by selectively using highly reliable distance information from the multiple frame images.

[0121] The reliability information acquisition unit 51 acquires the reliability map stored in the reliability storage unit 40. In addition, the reliability information acquisition unit 51 acquires information related to the displacement of the imaging device 10, information related to the RGB image, and the like, which are associated with the reliability map.

[0122] The surface element identifying unit 52 identifies surface elements based on a distance image corresponding to the acquired reliability map. The surface elements may be surfaces whose distance information is continuous (surfaces whose distance information is not discrete), for example. The surface element identifying unit 52 also identifies feature points for identifying the surfaces.

[0123] Similarity calculation section 53 calculates similarities based on feature points in the current frame image and feature points in the past frame images. Calculating similarities makes it possible to match the current frame image with the past frame images.

[0124] The synthesis unit 55 selectively extracts distance information about the faces from the frame image with the highest reliability among the multiple frame images according to the matching result, and synthesizes the distance information about the extracted faces to generate three-dimensional information of the subject S. The generated three-dimensional information is presented to the operator.

[0125] FIG. 17 is a block diagram showing an example of the internal configuration of the reliability calculation device 30 and the three-dimensional information creation device 50 according to this embodiment. The figure shows the internal configuration of either the reliability calculation device 30 or the three-dimensional information creation device 50. Both the reliability calculation device 30 and the three-dimensional information creation device 50 may have the internal configuration shown in the figure as a hardware configuration. At least some of the functions of the reliability calculation device 30 and the three-dimensional information creation device 50 can be realized using a computer. As shown in the figure, the computer is configured to include a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902, etc. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic operations and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices, etc. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data in the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0126] [Summary of the embodiment] According to the embodiment described above, the reliability calculation device 30 includes the distance image acquisition unit 313 to acquire a distance image having distance information at each coordinate of the two-dimensional coordinates, includes the IR image acquisition unit 312 to acquire an IR image having information corresponding to the distance image and information on the amount of infrared light received at each coordinate of the distance image, includes the ideal light amount calculation unit 3142 to calculate an ideal light amount based on the distance information at each coordinate of the acquired distance image by referring to the corresponding data, which is data in which the correspondence relationship between the distance and the amount of received light is measured in advance using an evaluation chart having a specific reflectance, and includes the calculation unit 3143 to calculate the reliability at each coordinate based on the result of comparing the ideal light amount obtained by calculation with the amount of infrared light received included in the IR image. By adopting such a configuration, the reliability calculation device 30 can calculate the reliability of the distance value for each pixel. By generating three-dimensional information based on the reliability calculated by the reliability calculation device 30, it is possible to generate three-dimensional information of the subject excluding inaccurate distance values, and therefore it is possible to generate highly accurate three-dimensional information that excludes abnormal values ​​due to the influence of noise, etc.

[0127] According to the above-described embodiment, the evaluation chart is a gray chart having a plurality of different gray levels, and the corresponding data is data in which the distance and the amount of received light are associated with each other for each gray level. The reliability calculation device 30 further includes a selection unit 3141, and selects a suitable gray level from among the plurality of gray levels based on distance information at each coordinate of the acquired distance image, and the ideal light amount calculation unit 3142 calculates the ideal light amount based on the data of the gray level selected by the selection unit 3141. Here, the amount of received light differs depending on the infrared reflectance of the subject S. According to the reliability calculation device 30, the reliability is calculated by comparing the ideal light amount for each infrared reflectance measured in advance with the amount of light obtained by imaging. Therefore, according to this embodiment, the reliability can be calculated with high accuracy.

[0128] According to the above-described embodiment, the three-dimensional information creation device 50 creates three-dimensional information using the reliability calculation device 30 as described above. The distance image acquisition unit 313 acquires a first distance image having distance information from a first point to the subject, and a second distance image having distance information from a second point, which is different from the first point, to the subject, and the IR image acquisition unit 312 acquires a first IR image corresponding to the first distance image and a second IR image corresponding to the second distance image. The three-dimensional information creation device 50 also compares the reliability based on the first distance image and the first IR image with the reliability based on the second distance image and the second IR image, and creates three-dimensional information based on the distance image with a high reliability. Therefore, the three-dimensional information creation device 50 can generate three-dimensional information of the subject based on the distance information with a high reliability, and therefore can generate highly accurate three-dimensional information that eliminates abnormal values ​​due to the influence of noise, etc.

[0129] According to the above-described embodiment, the three-dimensional information creation device 50 further includes a polarized image acquisition unit 311 to acquire a polarized image having information corresponding to the distance image and having the polarization information of the subject at each coordinate of the distance image, includes a mirror flag assignment unit 316 to assign a mirror flag to each coordinate based on the acquired polarized image, includes a noise removal unit 319 to remove distance information at the coordinate to which the mirror flag is assigned from the distance information that the distance image has at each coordinate, and includes a surface identification unit 317 to identify a surface based on the acquired polarized image. The three-dimensional information creation device 50 also creates three-dimensional information by smoothing the unevenness of the surface by complementing the removed distance information within the range identified as the surface. Therefore, according to this embodiment, accurate three-dimensional information of the subject S can be generated even if the subject S has a material that the ToF sensor is not good at detecting, such as a mirror surface.

[0130] The functions of the devices in the three-dimensional information generation system 1 in the above-described embodiment may be realized in whole or in part by recording a program for realizing these functions in a computer-readable recording medium, and reading and executing the program recorded in the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0131] In addition, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage units such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include those that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0132] Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications can be made without departing from the spirit of the present invention. In addition, the above-described embodiments may be combined as appropriate. [Explanation of symbols]

[0133] 1...3D information generation system, 10...imaging device, 20...inertial measurement device, 30...reliability calculation device, 40...reliability storage unit, 50...3D information creation device, 311...polarized image acquisition unit, 312...IR image acquisition unit, 313...distance image acquisition unit, 314...reliability calculation unit, 315...calibration information storage unit, 316...mirror flag assignment unit, 317...surface identification unit, 318...transparent material determination unit, 319...noise removal unit, 320...transparent flag assignment unit, 331...inertial information acquisition unit, 332...acceleration threshold information storage unit, 333...comparison unit, 334...displacement calculation unit, 340...reliability output unit, 3141...selection unit, 3142...ideal light amount calculation unit, 3143...calculation unit, 51...reliability information acquisition unit, 52...surface element identification unit, 53...similarity calculation unit, 55...combination unit, S...subject, MK...marker

Claims

1. a distance image acquisition unit for acquiring a distance image having distance information at each coordinate in a two-dimensional coordinate system; an IR image acquisition unit that acquires an IR image having information corresponding to the distance image, the IR image having information regarding an amount of infrared light received at each coordinate of the distance image; an ideal light amount calculation unit that refers to correspondence data that is data obtained by measuring a correspondence relationship between a distance to an evaluation chart and an amount of received light in advance using an evaluation chart having a specific reflectance, and calculates an ideal light amount based on distance information at each coordinate of the acquired distance image; a calculation unit that calculates a reliability for each coordinate based on a result of comparing the ideal light amount obtained by calculation with the amount of infrared light received included in the IR image for each coordinate; A reliability calculation device comprising:

2. The evaluation chart is a gray chart having a plurality of gray levels different from one another, the correspondence data is data in which distance and amount of received light are associated with each other for each gray level; a selection unit that selects a suitable gray level from among a plurality of gray levels based on distance information at each coordinate of the acquired distance image, The ideal light amount calculation unit calculates the ideal light amount based on the gray level data selected by the selection unit. The reliability calculation device according to claim 1 .

3. A three-dimensional information creation device that creates three-dimensional information using the reliability calculation device according to claim 1 or 2, the distance image acquisition unit acquires a first distance image having distance information from a first point to a subject, and a second distance image having distance information from a second point, which is different from the first point, to the subject; the IR image acquisition unit acquires a first IR image corresponding to the first distance image and a second IR image corresponding to the second distance image; A reliability based on the first distance image and the first IR image is compared with a reliability based on the second distance image and the second IR image, and three-dimensional information is generated based on the distance image having a higher reliability. Three-dimensional information creation device.

4. a polarization image acquisition unit that acquires information corresponding to the distance image, the polarization image having polarization information of a subject at each coordinate of the distance image; a mirror surface flag assigning unit that assigns a mirror surface flag to each coordinate based on the acquired polarized image; a noise removal unit that removes distance information at coordinates to which the mirror flag is assigned from distance information that the distance image has at each coordinate; a surface identification unit that identifies a surface based on the acquired polarized image, Within the range identified as a surface, the removed distance information is complemented to smooth out the unevenness of the surface, creating three-dimensional information. The three-dimensional information creation device according to claim 3.

5. a distance image acquisition step of acquiring a distance image having distance information at each coordinate of a two-dimensional coordinate system; an IR image acquisition step of acquiring information corresponding to the distance image, the IR image having information regarding an amount of infrared light received at each coordinate of the distance image; an ideal light amount calculation step of calculating an ideal light amount based on distance information at each coordinate of the acquired distance image by referring to correspondence data which is data obtained by previously measuring a correspondence relationship between a distance to a subject and an amount of received light using a subject having a specific reflectance; A calculation step of calculating a reliability at each coordinate based on a result of comparing the ideal light amount obtained by calculation with the amount of infrared light received included in the IR image at each coordinate; The reliability calculation method has the following structure:

Images