Modeling device, modeling method, and modeling program

Abstract

The purpose of the present invention is to provide a modeling device, a modeling method, and a modeling program that are capable of reducing the amount of information to be acquired or calculated, and achieve a high processing speed.　The modeling device is for creating polygon data pertaining to a point cloud comprising a plurality of points on the surface of an object, the modeling device comprising: an emission unit 1 that emits light to the object; an imaging unit 2 that detects two-dimensional image data of the object by imaging reflected light 15 obtained by the light that has been emitted being reflected by the object; and a computation means 3 that executes (1) a distance calculation step for calculating the distances to the respective points of the point cloud, (2) a three-dimensional coordinate calculation step for calculating three-dimensional coordinate data comprising the three-dimensional coordinates of the respective points on the basis of the distances to the respective points and the two-dimensional image data, and (3) a mesh creation step for creating polygon data by converting the point cloud into a mesh having planes of n-sided polygons obtained by connecting, with lines, points that are based on the three-dimensional coordinate data and close to each other.

Description

Modeling device, modeling method, and modeling program

[0001] The present invention relates to a modeling device, a modeling method, and a modeling program for creating polygon data of the surface shape of an object.

[0002] To create a three-dimensional model of an object, polygon data is used, which is a mesh of triangles or quadrangles formed by connecting three or more points with lines. The meshing requires three-dimensional coordinate data for each point. Conventionally, the object has been optically measured to obtain multiple two-dimensional image data, and the three-dimensional coordinate data is then calculated from the two-dimensional image data (see, for example, Patent Document 1).

[0003] JP 2003-58911

[0004] However, conventional methods have had problems with slow communication transfer speeds due to the large amount of information that must be measured. Another problem is that it takes a long time to calculate three-dimensional coordinate data from two-dimensional image data. These problems also slow down the processing speed for creating polygon data, resulting in a lack of real-time performance. Therefore, an object of the present invention is to provide a modeling device and modeling method that require less information to be acquired and calculated and that have high processing speeds.

[0005] In order to achieve the above-mentioned object, the modeling device of the present invention is for creating polygon data of a point cloud consisting of a plurality of points on the surface of an object, and is characterized by comprising: an illumination unit that illuminates the object with a predetermined pattern of light; an imaging unit that detects two-dimensional image data of the object by capturing an image of the light illuminated by the illumination unit reflected by the object; and calculation means that executes: (1) a distance calculation step that calculates the distance to each point of the point cloud using time data based on the time it takes for the light illuminated by the illumination unit to be reflected at each point and be captured by the imaging unit; (2) a three-dimensional coordinate calculation step that calculates three-dimensional coordinate data consisting of the three-dimensional coordinates of each point based on the distance to each point and the two-dimensional image data; and (3) a meshing step that creates polygon data by meshing the point cloud with an n-gonal plane in which adjacent points based on the three-dimensional coordinate data are connected by lines.

[0006] In this case, the irradiation unit may irradiate light in a dot pattern, and the three-dimensional coordinate calculation step may calculate the three-dimensional coordinate data of the point cloud by determining a z-coordinate for each point of the point cloud based on the distance to the point, and determining an x-coordinate and a y-coordinate based on the two-dimensional image data.

[0007] It is also preferable that the calculation means has, between the three-dimensional coordinate calculation step and the meshing step, a missing point completion step of calculating the three-dimensional coordinates of points on the surface of the object that could not be detected by the imaging unit based on the light distribution characteristics of the light irradiated by the irradiation unit and the three-dimensional coordinates, and adding the three-dimensional coordinate data to the three-dimensional coordinate data.

[0008] Furthermore, it is preferable that the imaging unit shortens the exposure time for imaging light or reduces the cumulative number of times the imaging unit images light as the distance to the object based on the three-dimensional coordinates becomes closer.

[0009] Furthermore, the modeling method of the present invention is a method for creating polygon data of a point cloud consisting of a plurality of points on the surface of an object, and is characterized by comprising the following calculation steps: an irradiation step of irradiating the object with a predetermined pattern of light from an irradiation unit; an imaging step of detecting two-dimensional image data of the object by imaging the light reflected by the object from the light irradiated from the irradiation unit with an imaging unit; (1) a distance calculation step of calculating the distance to each of the points using time data based on the time it takes for the light irradiated from the irradiation unit to be reflected at each point of the point cloud and to be imaged by the imaging unit; (2) a three-dimensional coordinate calculation step of calculating three-dimensional coordinate data consisting of the three-dimensional coordinates of each point based on the distance to each of the points and the two-dimensional image data; and (3) a meshing step of creating polygon data by meshing the point cloud with an n-gonal plane in which adjacent points based on the three-dimensional coordinate data are connected by lines.

[0010] In this case, the irradiation step may be to irradiate light in a dot pattern, and the three-dimensional coordinate calculation step may be to calculate the three-dimensional coordinate data of the point cloud by determining a z-coordinate for each point of the point cloud based on the distance to the point, and determining an x-coordinate and a y-coordinate based on the two-dimensional image data.

[0011] Furthermore, it is preferable that the calculation process includes, between the three-dimensional coordinate calculation step and the meshing step, a missing point completion step of calculating the three-dimensional coordinates of points on the surface of the object that could not be detected in the imaging step based on the light distribution characteristics of the light irradiated by the irradiation unit and the three-dimensional coordinates, and adding the three-dimensional coordinate data to the three-dimensional coordinate data.

[0012] Furthermore, it is preferable that the imaging process shortens the exposure time for the imaging unit to image light or reduces the cumulative number of times the imaging unit images light as the distance to the object based on the three-dimensional coordinates becomes shorter.

[0013] In addition, the modeling program of the present invention is for creating polygon data of a point cloud consisting of a plurality of points on the surface of an object, and is characterized in that it causes a computer to execute the following steps: a distance calculation step for calculating the distance to each point of the point cloud using time data based on the time it takes for light irradiated from an irradiation unit to be reflected at each point of the point cloud and to be captured by an imaging unit; a three-dimensional coordinate calculation step for calculating three-dimensional coordinate data consisting of the three-dimensional coordinates of each point based on the distance to each point and the two-dimensional image data captured by the imaging unit; and a meshing step for creating polygon data by meshing the point cloud with an n-sided plane in which adjacent points based on the three-dimensional coordinate data are connected by lines.

[0014] In this case, the computer may be configured to execute an irradiation step in which an irradiation unit irradiates the object with light of a predetermined pattern, and an imaging step in which an imaging unit images the light reflected by the object from the light irradiated from the irradiation unit, and detects two-dimensional image data of the object.

[0015] Furthermore, it is preferable that between the three-dimensional coordinate calculation step and the meshing step, the computer executes a missing point completion step of calculating the three-dimensional coordinates of points on the surface of the object that could not be detected in the imaging step based on the light distribution characteristics of the light irradiated by the irradiation unit and the three-dimensional coordinates, and adding the three-dimensional coordinate data to the missing point completion step.

[0016] It is also preferable that the imaging step be performed by a computer so that the closer the distance to the object based on the three-dimensional coordinates, the shorter the exposure time for the imaging unit to capture light, or the fewer the cumulative number of times the imaging unit captures light.

[0017] The present invention can provide a modeling device, a modeling method, and a modeling program that require a small amount of information to be acquired and calculated and have a high processing speed.

[0018] FIG. 1 is a schematic diagram showing the configuration of a modeling device of the present invention. FIG. 2 is a two-dimensional image photograph showing a pattern of light emitted by an irradiation unit according to the present invention. (a) and (b) are schematic diagrams of the irradiation unit according to the present invention, and (c) is a perspective view of a lens. FIG. 2 is two-dimensional image data showing an example of a point cloud. FIG. 3 is a schematic diagram showing an example of polygon data. FIG. 3 is two-dimensional image data showing an example of a point cloud where a point is missing. FIG. 4 is a schematic diagram showing polygon data to which a missing point has been added. FIG. 4 is a configuration diagram of a modeling system of the present invention. FIG. 5 is a flowchart showing each step of a modeling program of the present invention.

[0019] The modeling device of the present invention is described below. The modeling device of the present invention is used to create polygon data of a point cloud consisting of a plurality of points on the surface of an object 9, and is mainly composed of an illumination unit 1, an imaging unit 2, and a calculation means 3, as shown in FIG.

[0020] The irradiation unit 1 is for irradiating the object 9 with light of a predetermined pattern. The irradiation unit 1 may be any device capable of irradiating the object 9 with light of a predetermined pattern, and may, for example, be composed of a light source unit 11 and an optical element 12. The irradiation unit 1 is also connected to the calculation means 3 by wire or wirelessly, and can send information such as the time and length of light projection to the calculation means 3. The irradiation unit 1 may also be configured to receive information sent from the calculation means 3 by wire or wirelessly. This allows the calculation means 3 to control the irradiation unit 1 and adjust the light irradiation time and length.

[0021] The light source unit 11 may be any type that includes a light source 110 that irradiates light of wavelength λ onto the optical element 12. The light source unit 11 may include a single light source or multiple light sources. Alternatively, the light source unit 11 may include multiple light sources by passing light from a single light source through an aperture having multiple pores. When the light source unit 11 is configured with multiple light sources, it is preferable that the light sources 110 are formed on the same plane. Specific examples of the light source unit 11 include a vertical cavity surface emitting laser (VCSEL), which can achieve high output with low power consumption. VCSELs include single-emitter VCSELs that have one light source 110 that can irradiate light in a direction perpendicular to the light-emitting surface, and multi-emitter VCSELs that have multiple light sources 110. It is also preferable to form a light-absorbing film on areas other than the light source 110 to prevent noise due to reflected light.

[0022] The optical element 12 may be any type that can control and irradiate the light from the light source unit 11 onto a predetermined area of ​​the object 9. Specifically, the optical element 12 may be one that changes the light from the light source unit 11 into a predetermined pattern, such as one that changes the light from the light source unit 11 into a pattern of multiple dots spreading out (hereinafter referred to as a dot pattern) as shown in Fig. 2(a), one that changes the light from the light source unit 11 into a pattern of multiple lines spreading out (hereinafter referred to as a line pattern) as shown in Fig. 2(b), or one that diffuses the light from the light source unit 11 uniformly over a predetermined area as shown in Fig. 2(c).

[0023] An example of the optical element 12 that converts the light from the light source unit 11 into a dot pattern is shown in Figure 3. The optical element 12 has lenses 120 that transmit light of wavelength λ, periodically arranged in a square array. Here, the lenses 120 have a focal point at a predetermined distance f (f > 0) from the lens 120. The optical element 12 can improve contrast compared to conventional elements as the focal length f increases, such as 10 μm or more, 20 μm or more, 40 μm or more, or 60 μm or more.

[0024] The shape of the lens 120 can be freely designed according to the desired spread of the dots to be irradiated. For example, if it is desired to spread the dots in a circular shape, the shape of the lens 120 can be a spherical lens. On the other hand, if it is desired to spread the dots in a non-circular shape, the shape of the lens 120 can be an appropriately designed aspherical lens. Specific lens shapes include, for example, a convex lens and a concave lens. In the case of a convex lens, it is preferable that the convex lens portion faces the irradiation unit 1 side.

[0025] The periodic arrangement of the lenses 120 may be a quadrangular arrangement of lenses 120 that are square or rectangular in plan view, or a hexagonal arrangement of lenses 120 that are hexagonal in plan view. The lenses 120 may be any type that functions as a lens, and may be, for example, a Fresnel lens, a DOE lens, or a metalens. The lenses 120 are preferably provided with an anti-reflection coating that prevents reflection of light from the light source unit 11.

[0026] The light source section 11 and the optical element 12 may be arranged so that the optical axis direction of the light source 110 of the light source section 11 coincides with the optical axis direction of the lens 120 of the optical element 12 .

[0027] [Positional Relationship Between the Irradiation Unit 1 and the Optical Element 12] As shown in FIG. 3, the distance L between the irradiation unit 1 and the first focal plane 121 of the lens 120 is １ , the distance L to the second focal plane 122 ２ can convert incident light into a dot pattern with high contrast when the following formulas α and β are satisfied: where m and n are natural numbers equal to or greater than 1, １ is the pitch of the lens 120 in the x direction, P ２ is the pitch in the y direction, λ is the wavelength of the light incident from the irradiation unit 1, and f １ is the focal length of the lens 120 due to the cross-sectional shape perpendicular to the y direction, and f ２is the focal length according to the cross-sectional shape of the lens 120 perpendicular to the x-direction, and a, b, c, and d are coefficients indicating the allowable error. The first focal plane 121 refers to a plane that is perpendicular to the optical axis (z-direction) of the lens 120 and is located at the focal position according to the cross-sectional shape of the lens 120 perpendicular to the y-direction. The second focal plane 122 refers to a plane that is perpendicular to the optical axis (z-direction) of the lens 120 and is located at the focal position according to the cross-sectional shape of the lens 120 perpendicular to the x-direction. When focal points are on both sides of the lens 120, it is preferable to use the first focal plane 121 and the second focal plane 122 on the irradiation unit 1 side of the lens 120 as the reference.

[0028] Here, the smaller the coefficient a in formula α, the more preferable it is, such as a = 1, a = 0.75, a = 0.5, and a = 0.25. The smaller the coefficient b, the more preferable it is, such as b = 1, b = 0.75, b = 0.5, and b = 0.25. The smaller the coefficient c in formula β, the more preferable it is, such as c = 1, c = 0.75, c = 0.5, and c = 0.25. The smaller the coefficient d, the more preferable it is, such as d = 1, d = 0.75, d = 0.5, and d = 0.25. When the coefficients of formula α and formula β are a = b = c = d = 1, formula α and formula β become formula 1 and formula 2 below, respectively.

[0029] In addition, the distance L １ , L ２ can most effectively strengthen the light when the following formulas 3 and 4 are satisfied, where a=b=c=d=0.

[0030] Furthermore, diffraction becomes difficult to occur if the pitch P is too small compared to the wavelength λ of the light from the light source 110. Therefore, as long as the light distribution angle of the light source 110 includes enough lenses 120 to cause diffraction, the pitch P should be sufficiently larger than the wavelength λ of the light from the light source 110, for example, 5 times or more, and preferably 10 times or more.

[0031] Furthermore, when the light source unit 11 has a plurality of light sources 110, they must be arranged so that the number of light sources 110 for each lens 120 of the optical element 12 remains the same in plan view even when the light source unit 11 and the optical element 12 are moved in parallel relative to each other. Therefore, the light sources 110 of the irradiation unit 1 are arranged so that the size of the pitch in the x direction is P ｘ , the size of the pitch in the y direction is P ｙ In the case of an array such that j and k are natural numbers equal to or greater than 1, P ｘ =jp １ or jP ｘ =P １ and P ｙ = kP ２ or kP ｙ =P ２ It is better to satisfy

[0032] The optical element has a focal length f １ ya f ２ It is also possible to use wide-angle lenses with respective diameters smaller than 20 μm, 15 μm, and 10 μm, and narrow-angle lenses with respective diameters larger than 60 μm, 65 μm, and 70 μm.

[0033] The imaging unit 2 detects two-dimensional image data of the object 9 by capturing an image of the light irradiated from the irradiation unit 1 and reflected by the object 9. Here, the two-dimensional image data means digital data on a two-dimensional image of the reflected light captured by the imaging unit 2. For example, Fig. 4(b) shows two-dimensional image data captured by capturing reflected light 15 from the object 9 when dot pattern light is irradiated from the irradiation unit 1 onto the object shown in Fig. 4(a).

[0034] It is also preferable that the imaging unit 2 can detect imaging information such as the time and length of receiving reflected light, the light intensity and wavelength (color) of the received light, as digital data. The imaging unit 2 is connected to the calculation means 3 via a wired or wireless connection, and can send information such as the time of receiving light and the above-mentioned two-dimensional image data to the calculation means 3. The imaging unit 2 may also be configured to receive information sent from the calculation means 3 via a wired or wireless connection. This allows the calculation means 3 to control the imaging unit 2 and adjust the exposure time and number of integrations for the imaging unit to capture light. The imaging unit 2 may be any device that can detect reflected light and detect two-dimensional image data. For example, an existing image sensor such as a CMOS or CCD that can convert detected light into digital data may be used.

[0035] The calculation means 3 is used to create polygon data for a point cloud consisting of a plurality of points 6 on the surface of the object 9. The calculation means 3 is connected to the irradiation unit 1 and the imaging unit 2 via a wired or wireless connection, and can receive information such as the time when the irradiation unit 1 projected light, the time when the imaging unit 2 received the light, and two-dimensional image data. Then, based on this information, the calculation means 3 creates polygon data for a predetermined point cloud on the surface of the object 9. The calculation means 3 may be any device that can create polygon data for the point cloud based on the information received from the irradiation unit 1 and the imaging unit 2, and may, for example, be a central processing unit (CPU) or a computer including such a CPU. The calculation means 3 may also be configured to send information for controlling the irradiation unit 1 and the imaging unit 2.

[0036] Note that the points 6 constituting the point cloud may be selected in any manner. For example, if the light irradiated onto the object 9 by the irradiation unit 1 is a dot pattern, the multiple points 6 formed on the surface of the object 9 by the dot pattern can be directly used as the point cloud. Furthermore, if the light irradiated onto the object 9 by the irradiation unit 1 is a line pattern, multiple points 6 may be arbitrarily selected from multiple lines formed on the surface of the object 9 by the line pattern; for example, equally spaced points 6 on the lines may be used as the point cloud. Furthermore, if the light irradiated onto the object 9 by the irradiation unit 1 is uniformly diffused light, multiple points 6 may be arbitrarily selected in the X and Y directions from the pattern formed on the surface of the object 9 by the diffused light; for example, points 6 on a lattice may be used as the point cloud.

[0037] The calculation means 3 mainly executes (1) a distance calculation step, (2) a three-dimensional coordinate calculation step, and (3) a meshing step on the point group to create polygon data.

[0038] (1) The distance calculation step is for calculating the distance to each point 6 of the point cloud. The distance to each point 6 of the point cloud can be calculated using a time-of-flight (TOF) method. Specifically, the distance to each point 6 is calculated using time data based on the time it takes for light irradiated from the irradiation unit 1 to be reflected by each point 6 of the point cloud and captured by the imaging unit 2. The distance can be calculated using the product of the time and the speed of light.

[0039] (2) The three-dimensional coordinate calculation step is for determining the three-dimensional coordinates of each point 6 in the point cloud. Three-dimensional coordinate data consisting of the three-dimensional coordinates of each point 6 is calculated based on the distance to each point 6 calculated in (1) and the two-dimensional image data. For example, the calculation means 3 determines the z coordinate of each point 6 based on the distance to each point 6 in the point cloud. The calculation means 3 also determines two-dimensional coordinates such as x and y coordinates based on the two-dimensional image data. In this way, three-dimensional coordinate data consisting of the three-dimensional coordinates of each point 6 is calculated and created. Note that polar coordinates, etc., can also be used as the three-dimensional coordinates.

[0040] The two-dimensional coordinates refer to coordinates on a two-dimensional image captured by the imaging unit 2. The coordinates may be xy coordinates, consisting of an x ​​coordinate indicating a position in the x direction and a y coordinate indicating a position in the y direction. The x and y directions may be any directions as long as they do not coincide, but it is generally preferable to use directions that are orthogonal to each other. The x and y directions are also perpendicular to the z direction.

[0041] (3) The meshing step is for generating polygon data by meshing the point cloud, for example, as shown in FIG. 5 . Specifically, the point cloud is meshed with an n-sided plane in which adjacent points 6 based on the three-dimensional coordinate data calculated in (2) are connected by lines 7, such as straight lines or curves, to generate polygon data. Here, n is a natural number of 3 or more. Generally, a triangle or a quadrilateral is used as the n-sided polygon, but other polygons may also be used. A conventionally known method may be used to mesh the point cloud and generate polygon data, such as the Marching Cubes method or the Ball-Pivoting algorithm.

[0042] Note that the imaging unit 2 may be unable to detect the positions of some points in the point cloud due to saturation, reflection, proximity of dots, etc., as shown in Figure 6. In this case, the computing means 3 may include, between the above-mentioned (2) three-dimensional coordinate calculation step and (3) meshing step, a missing point completion step of calculating the three-dimensional coordinates of points on the surface of the object 9 that could not be detected by the imaging unit 2 (hereinafter referred to as missing points 6A) based on the light distribution characteristics and three-dimensional coordinates of the light irradiated by the irradiation unit 1, and adding the three-dimensional coordinate data to the missing points 6A. Here, the light distribution characteristics refer to information regarding the spread of light by the irradiation unit 1, light intensity, etc.

[0043] Any method may be used to calculate the three-dimensional coordinates of the defective point 6A that could not be detected by the imaging unit 2, as long as the error from the actual position of the defective point 6A is small.

[0044] As an example, first, the presence or absence of a missing point 6A is estimated. For example, if there are no significant irregularities around the missing point 6A on the surface of the object, the calculation means 3 can estimate that the missing point 6A is located in the periphery of the three-dimensional space formed by the adjacent points 6. Alternatively, the calculation means 3 may compare the light distribution characteristics with two-dimensional image data to estimate the presence or absence of the missing point 6A. If the missing point 6A is present, the provisional position of the missing point 6A is determined to be the midpoint of the three-dimensional coordinates of the two adjacent points sandwiching the missing point 6A, the position obtained by correcting the midpoint position using 3D light distribution data, or a position estimated by AI using a neural network or the like. This process is repeated for two adjacent points horizontally, vertically, diagonally, etc. around the missing point 6A to calculate the provisional position of the missing point 6A.

[0045] Next, the three-dimensional coordinates of one missing point 6A are calculated from the multiple virtual positions of the missing points 6A calculated in this manner. The three-dimensional coordinates of the missing point 6A can be calculated by any method as long as the error from the actual position of the missing point 6A is small. For example, the three-dimensional coordinates can be calculated by calculating the average position of the xyz positions of the multiple virtual missing points 6A, by using a regression method such as the least squares method, or by using AI such as a neural network. When using AI, it is also possible to use information about the surface condition of the object 9 that is known in advance, information about the surrounding measurement environment, etc. The measurement environment information refers to information that affects the two-dimensional image data captured by the imaging unit 2, such as the ambient brightness.

[0046] In this way, the calculation means 3 calculates the three-dimensional coordinates of the missing point 6A and adds them to the three-dimensional coordinate data calculated in the above-mentioned (2) three-dimensional coordinate calculation step to create new three-dimensional coordinate data.

[0047] Furthermore, the imaging unit 2 may be configured to shorten the exposure time for capturing light or reduce the number of integrations as the distance to the object 9 becomes shorter based on the three-dimensional coordinates. This makes it possible to prevent the influence of saturation. Such control of the imaging unit 2 may also be performed based on information sent from the calculation means 3.

[0048] Next, the modeling method of the present invention will be described using the modeling device described above. The modeling method of the present invention is a method for creating polygon data of the object 9, and is mainly composed of an illumination step, an imaging step, and a calculation step.

[0049] The irradiation process is a process of irradiating the object 9 with light of a predetermined pattern from the irradiation unit 1. In the irradiation process, any pattern may be used as long as the light from the irradiation unit 1 can be controlled and irradiated onto a predetermined area of ​​the object 9. Examples of the light pattern irradiated by the irradiation unit 1 include a dot pattern as shown in FIG. 2( a), a line pattern as shown in FIG. 2( b), and diffused light in which light is uniformly diffused over a predetermined range as shown in FIG. 2( c). In addition, in the irradiation process, information such as the time and length for which the light is projected is also acquired.

[0050] The imaging step is a step of detecting two-dimensional image data of the object 9 by using the imaging unit 2 to capture the light reflected by the object 9 from the light irradiated from the irradiation unit 1. In the imaging step, it is preferable to detect imaging information such as the time and length for which the reflected light is received, and the light intensity and wavelength (color) of the received light as digital data.

[0051] The calculation step is a step of creating polygon data for a point cloud consisting of a plurality of points 6 on the surface of the object 9. In the calculation step, polygon data is created for a predetermined point cloud on the surface of the object 9 based on information such as the time when the irradiation unit 1 projected light, the time when the imaging unit 2 received the light, and two-dimensional image data.

[0052] Note that the points 6 constituting the point cloud may be selected in any manner. For example, if the light irradiated onto the object 9 by the irradiation unit 1 is a dot pattern, the multiple points 6 formed on the surface of the object 9 by the dot pattern can be directly used as the point cloud. Furthermore, if the light irradiated onto the object 9 by the irradiation unit 1 is a line pattern, multiple points 6 may be arbitrarily selected from multiple lines formed on the surface of the object 9 by the line pattern; for example, equally spaced points 6 on the lines may be used as the point cloud. Furthermore, if the light irradiated onto the object 9 by the irradiation unit 1 is uniformly diffused light, multiple points 6 may be arbitrarily selected in the X and Y directions from the pattern formed on the surface of the object 9 by the diffused light; for example, points 6 on a lattice may be used as the point cloud.

[0053] The calculation process mainly involves executing (1) a distance calculation step, (2) a three-dimensional coordinate calculation step, and (3) a meshing step on the point group to create polygon data.

[0054] (1) The distance calculation step is for calculating the distance to each point 6 of the point cloud. The distance to each point 6 of the point cloud can be calculated using a time-of-flight (TOF) method. Specifically, the distance to each point 6 is calculated using time data based on the time it takes for light irradiated from the irradiation unit 1 to be reflected by each point 6 of the point cloud and captured by the imaging unit 2. The distance can be calculated using the product of the time and the speed of light.

[0055] (2) The three-dimensional coordinate calculation step is for determining the three-dimensional coordinates of each point 6 in the point cloud. Three-dimensional coordinate data consisting of the three-dimensional coordinates of each point 6 is calculated based on the distance to each point 6 calculated in (1) and the two-dimensional image data. For example, the calculation means 3 determines the z coordinate of each point 6 based on the distance to each point 6 in the point cloud. The calculation means 3 also determines two-dimensional coordinates such as x and y coordinates based on the two-dimensional image data. In this way, three-dimensional coordinate data consisting of the three-dimensional coordinates of each point 6 is calculated and created. Note that polar coordinates, etc., can also be used as the three-dimensional coordinates.

[0056] The two-dimensional coordinates refer to coordinates on a two-dimensional image captured by the imaging unit 2. The coordinates may be xy coordinates, consisting of an x ​​coordinate indicating a position in the x direction and a y coordinate indicating a position in the y direction. The x and y directions may be any directions as long as they do not coincide, but it is generally preferable to use directions that are orthogonal to each other. The x and y directions are also perpendicular to the z direction.

[0057] (3) The meshing step is for generating polygon data by meshing the point cloud, for example, as shown in FIG. 5 . Specifically, the point cloud is meshed with an n-gonal plane in which adjacent points 6 based on the three-dimensional coordinate data calculated in (2) are connected by lines such as straight lines and curves, thereby generating polygon data. Here, n is a natural number greater than or equal to 3. Generally, a triangle or a quadrilateral is used as the n-gon, but other polygons may also be used. A conventionally known method may be used to mesh the point cloud and generate polygon data, such as the Marching Cubes method or the Ball-Pivoting algorithm.

[0058] In the imaging process, the positions of some points 6 in the point cloud may not be detected due to saturation, reflection, proximity of dots, etc., as shown in Figure 6. In this case, the calculation process may include, between the above-mentioned (2) three-dimensional coordinate calculation step and (3) meshing step, a missing point completion step of calculating the three-dimensional coordinates of missing points 6A that could not be detected in the imaging process based on the light distribution characteristics and three-dimensional coordinates of the light irradiated by the irradiation unit 1, and adding the three-dimensional coordinate data to the missing point completion step. Here, the light distribution characteristics refer to information regarding the spread of light by the irradiation unit 1, light intensity, etc.

[0059] Any method may be used to calculate the three-dimensional coordinates of the defective point 6A that could not be detected in the imaging process, as long as the error from the actual position of the defective point 6A is small.

[0060] As an example, first, the presence or absence of a missing point 6A is estimated. For example, if there are no significant irregularities around the missing point 6A on the surface of the object, it can be estimated that the missing point 6A is located in the periphery of the three-dimensional space formed by the adjacent points 6. Alternatively, the light distribution characteristics may be compared with two-dimensional image data to estimate the position of the missing point 6A. If a missing point 6A is present, the provisional position of the missing point 6A may be the midpoint of the three-dimensional coordinates of the two adjacent points sandwiching the missing point 6A, the position obtained by correcting the midpoint position using 3D light distribution data, or a position estimated by AI using a neural network or the like. This process is repeated for two adjacent points horizontally, vertically, diagonally, etc. around the missing point 6A to calculate the provisional position of the missing point 6A.

[0061] Next, the three-dimensional coordinates of one missing point 6A are calculated from the multiple virtual positions of the missing points 6A calculated in this manner. The three-dimensional coordinates of the missing point 6A can be calculated by any method as long as the error from the actual position of the missing point 6A is small. For example, the three-dimensional coordinates can be calculated by calculating the average position of the x, y, and z positions of the multiple virtual missing points 6A, by using a regression method such as the least squares method, or by using AI such as a neural network. When using AI, it is also possible to use information about the surface condition of the object 9 that is known in advance, information about the surrounding measurement environment, etc. The measurement environment information refers to information that affects the two-dimensional image data captured in the imaging process, such as ambient brightness.

[0062] In this way, in the calculation process, the three-dimensional coordinates of the missing point 6A are calculated, and these are added to the three-dimensional coordinate data calculated in the above-mentioned (2) three-dimensional coordinate calculation step to create new three-dimensional coordinate data.

[0063] Furthermore, the imaging step may be such that the shorter the exposure time for imaging light by the imaging unit 2 or the fewer number of integrations is, the closer the distance to the object 9 is based on the three-dimensional coordinates. This makes it possible to prevent the effects of saturation.

[0064] Next, the modeling program of the present invention will be described. Here, the modeling program 4 refers to so-called software. The modeling program 4 of the present invention is for creating polygon data of a point cloud consisting of a plurality of points 6 on the surface of an object 9. FIG. 8 is a block diagram showing a schematic electrical configuration of a modeling system 5 in which the modeling program 4 of the present invention is installed. The modeling system 5 refers to hardware such as a computer in which the modeling program 4 is installed and its peripheral devices. The modeling system 5 can be used as the above-mentioned calculation means 3 or a modeling device equipped with the same.

[0065] The modeling system 5 is used to create polygon data of a point cloud consisting of a plurality of points 6 on the surface of an object 9, based on two-dimensional image data of the point cloud and distance data to each point 6 included in the point cloud. The hardware used in the modeling system 5, such as a computer and its peripheral devices, is mainly composed of a CPU 51, ROM 52, RAM 53, storage device 54, input device 56, display device 57, interface, etc., as shown in FIG.

[0066] The CPU 51 is a central processing unit that controls the modeling system 5 overall, and executes the various steps shown in the flowchart of Fig. 9 based on the modeling program 4. Information (data) determined by the CPU 51 is stored in the RAM 53 and storage device 54, which will be described later. The ROM 52 is a non-volatile memory that stores the BIOS executed by the CPU 51, etc. The RAM 53 is a volatile memory that temporarily stores the launched modeling program 4, information (data) required for various steps of the modeling program 4 executed by the CPU 51, information determined in various steps, etc.

[0067] The interface is for connecting the CPU 51 to external devices such as a ROM 52, a RAM 53, a storage device 54, an input device 56, a display device 57, a scanner, a printer, and a tablet via wired or wireless connection.

[0068] The storage device 54 refers to a rewritable nonvolatile memory such as a hard disk drive (HDD) or a solid state drive (SSD), and stores the modeling program 4, the database 41, etc. It may also store the modeling program 4, information (data) required for various steps of the modeling program 4 executed by the CPU 51, and information determined in various steps. Although not shown, the storage device 54 may also store an operating system (OS) such as Windows, or an interpreter such as Java or JRuby. The storage device 54 may be built into the computer or may be located on an external server. Using an external server allows multiple operators to use the modeling program 4 on multiple computers.

[0069] The modeling system 5 executes each step shown in the flowchart of Fig. 9 in accordance with the modeling program 4. Details of each step will be described later.

[0070] The database 41 stores various types of information used by the modeling system 5 and modeling program 4 of the present invention. For example, this information includes information on the surface condition of the object 9 and information on the surrounding measurement environment. Measurement environment information refers to information that affects the two-dimensional image data captured in the imaging process, such as information on the brightness of the surroundings. The database 41 also stores information for AI using neural networks, etc. Information not registered in the database 41 can also be registered in advance by an operator. The database 41 may be dedicated to the modeling program 4, or may be shared with databases of other software.

[0071] The input device 56 corresponds to, for example, the above-mentioned irradiation unit 1 or imaging unit 2, and is used to input time data, two-dimensional image data, etc. into the modeling system. The input device 56 may also be a keyboard, mouse, etc. that an operator uses to input time data, two-dimensional image data, etc. into the modeling system.

[0072] The display device 57 has a screen and, in response to signals from the CPU 51, displays and outputs the work screen in the modeling system 5 and the work status on the work screen, and is, for example, a liquid crystal display.

[0073] As shown in FIG. 9( a), the modeling program creates polygon data for a point cloud by causing the computer to execute mainly the following steps: (1) distance calculation step, (2) three-dimensional coordinate calculation step, and (3) meshing step.

[0074] (1) The distance calculation step is for calculating the distance to each point 6 in the point cloud. The distance to each point 6 in the point cloud can be calculated using a time-of-flight (TOF) method. Specifically, the distance to each point 6 is calculated using time data based on the time it takes for light irradiated from the irradiation unit 1 to be reflected by each point 6 in the point cloud and captured by the imaging unit 2. The distance can be calculated using the product of the time and the speed of light. Note that the time data can be data stored in advance in the storage device 54 by the operator as an input step, or digital data stored in the storage device 54 in the irradiation step or imaging step described below.

[0075] (2) The three-dimensional coordinate calculation step is for determining the three-dimensional coordinates of each point 6 in the point cloud. Three-dimensional coordinate data consisting of the three-dimensional coordinates of each point 6 is calculated based on the distance to each point 6 calculated in (1) and the two-dimensional image data. For example, the calculation means 3 determines the z-coordinate of each point 6 based on the distance to each point 6 in the point cloud. The calculation means 3 also determines two-dimensional coordinates such as x- and y-coordinates based on the two-dimensional image data. This calculates and creates three-dimensional coordinate data consisting of the three-dimensional coordinates of each point 6. Polar coordinates, etc., can also be used as the three-dimensional coordinates. The two-dimensional image data may be data stored in advance in the storage device 54 by the operator in the input step, or digital data stored in the storage device 54 in the imaging step described below.

[0076] Here, two-dimensional coordinates refer to coordinates on a two-dimensional image captured by the imaging unit 2. As the coordinates, xy coordinates consisting of an x ​​coordinate indicating a position in the x direction and a y coordinate indicating a position in the y direction may be used. The x direction and the y direction may be any direction as long as they do not coincide, but it is generally preferable to use directions that are orthogonal to each other. Furthermore, the x direction and the y direction are perpendicular to the z direction.

[0077] (3) The meshing step is for generating polygon data by meshing the point cloud, for example, as shown in FIG. 5 . Specifically, the point cloud is meshed with an n-gonal plane in which adjacent points 6 based on the three-dimensional coordinate data calculated in (2) are connected by lines such as straight lines and curves, thereby generating polygon data. Here, n is a natural number greater than or equal to 3. Generally, a triangle or a quadrilateral is used as the n-gon, but other polygons may also be used. A conventionally known method may be used to mesh the point cloud and generate polygon data, such as the Marching Cubes method or the Ball-Pivoting algorithm.

[0078] Furthermore, the modeling program may cause the computer to execute an irradiation step and an imaging step before the distance calculation step (1) described above in order to acquire time data and two-dimensional image data, as shown in FIG. 9(b).

[0079] The irradiation step is a step of causing the computer to execute the process so as to cause the irradiation unit 1 to irradiate the target object 9 with light of a predetermined pattern. Examples of the light pattern irradiated by the irradiation unit 1 include a dot pattern as shown in Fig. 2(a), a line pattern as shown in Fig. 2(b), and diffused light in which light is uniformly diffused over a predetermined range as shown in Fig. 2(c). Furthermore, in the irradiation step, the computer is executed to acquire information such as the time and length for which the light is projected and store this information as digital data in the storage device 54.

[0080] The imaging step causes the computer to capture an image of the light irradiated from the irradiation unit and reflected by the object, and to detect two-dimensional image data of the object. The imaging step also causes the computer to acquire imaging information such as the time and length for receiving the reflected light, and the light intensity and wavelength (color) of the received light, and store the information as digital data in the storage device 54.

[0081] In the imaging step, the positions of some points 6 in the point cloud may not be detected due to saturation, reflection, proximity of dots, etc., as shown in Figure 6. In this case, the modeling program may cause the computer to execute a missing point completion step between the above-mentioned (2) three-dimensional coordinate calculation step and (3) meshing step, in which the computer calculates the three-dimensional coordinates of missing points 6A that could not be detected in the imaging step based on the light distribution characteristics and three-dimensional coordinates of the light irradiated by the irradiation unit 1, and adds the three-dimensional coordinate data to the missing point completion step. Here, the light distribution characteristics refer to information regarding the spread of light by the irradiation unit 1, light intensity, etc.

[0082] Any method may be used to calculate the three-dimensional coordinates of the defective point 6A that could not be detected in the imaging step, as long as the error from the actual position of the defective point 6A is small.

[0083] As an example, first, the presence or absence of a missing point 6A is estimated. For example, if there are no significant irregularities around the missing point 6A on the surface of the object, it can be estimated that the missing point 6A is located in the periphery of the three-dimensional space formed by the adjacent points 6. Alternatively, the light distribution characteristics may be compared with two-dimensional image data to estimate the position of the missing point 6A. If a missing point 6A is present, the provisional position of the missing point 6A may be the midpoint of the three-dimensional coordinates of the two adjacent points sandwiching the missing point 6A, the position obtained by correcting the midpoint position using 3D light distribution data, or a position estimated by AI using a neural network or the like. This process is repeated for two adjacent points horizontally, vertically, diagonally, etc. around the missing point 6A to calculate the provisional position of the missing point 6A.

[0084] Next, the three-dimensional coordinates of one missing point 6A are calculated from the multiple virtual positions of the missing points 6A calculated in this manner. The three-dimensional coordinates of the missing points 6A can be calculated by any method as long as the error from the actual position of the missing points 6A is small. For example, the three-dimensional coordinates can be calculated by calculating the average position of the x, y, and z positions of the multiple virtual missing points 6A, by using a regression method such as the least squares method, or by using AI such as a neural network. When using AI, it is also possible to store information about the surface condition of the object 9 and information about the surrounding measurement environment, which are known in advance, in the database 41 and use this information.

[0085] In this way, the three-dimensional coordinates of the missing point 6A are calculated in the missing point completion step, and these are added to the three-dimensional coordinate data calculated in the above-mentioned (2) three-dimensional coordinate calculation step to create new three-dimensional coordinate data.

[0086] The imaging step may also be executed by a computer so that the exposure time for imaging unit 2 to capture light is shortened or the number of integrations is reduced as the distance to object 9 becomes closer based on the three-dimensional coordinates. This makes it possible to prevent the effects of saturation.

[0087] REFERENCE SIGNS LIST 1 Irradiation unit 2 Imaging unit 3 Calculation means 4 Modeling program 5 Modeling system 6 Point 6A Missing point 7 Line 9, 9A Object 11 Light source unit 12 Optical element 15 Reflected light 110 Light source 120 Lens 121 First focal plane 122 Second focal plane

Claims

1. A modeling apparatus for creating polygon data of a point group composed of a plurality of points on the surface of an object, comprising: an irradiation unit that irradiates the object with light of a predetermined pattern; an imaging unit that detects two-dimensional image data of the object by imaging reflected light of the light irradiated from the irradiation unit by the object; and calculation means that executes: (1) a distance calculation step of calculating a distance to each point based on time data based on the time from when the light irradiated from the irradiation unit is reflected by each point of the point group until it is imaged by the imaging unit; (2) a three-dimensional coordinate calculation step of calculating three-dimensional coordinate data composed of the three-dimensional coordinates of each point based on the distance to each point and the two-dimensional image data; and (3) a meshing step of creating polygon data by meshing the point group into a plane of an n-sided polygon connecting adjacent points based on the three-dimensional coordinate data. The modeling apparatus is characterized by comprising the above.

2. The irradiation unit irradiates light of a dot pattern, and the three-dimensional coordinate calculation step calculates the three-dimensional coordinate data of the point group by determining a z coordinate based on the distance to each point of the point group and determining an x coordinate and a y coordinate based on the two-dimensional image data for each point of the point group. The modeling apparatus according to claim 1 is characterized by this.

3. The calculation means has a missing point complementing step of calculating the three-dimensional coordinates of points on the surface of the object that could not be detected by the imaging unit based on the light distribution characteristics of the light irradiated by the irradiation unit and the three-dimensional coordinates, and adding them to the three-dimensional coordinate data, between the three-dimensional coordinate calculation step and the meshing step. The modeling apparatus according to claim 1 or 2 is characterized by this.

4. The imaging unit is such that the shorter the distance to the object based on the three-dimensional coordinates, the shorter the exposure time for the imaging unit to image light. The modeling apparatus according to claim 1 or 2 is characterized by this.

5. The imaging unit is such that the shorter the distance to the object based on the three-dimensional coordinates, the fewer the number of integration times for the imaging unit to image light. The modeling apparatus according to claim 1 or 2 is characterized by this.

6. A modeling method for creating polygon data of a point group composed of a plurality of points on the surface of an object, the method comprising: an irradiation step of irradiating the object with light of a predetermined pattern from an irradiation unit; an imaging step of detecting two-dimensional image data of the object by imaging, with an imaging unit, reflected light of the light irradiated from the irradiation unit by the object; and a calculation step including: (1) a distance calculation step of calculating a distance to each point based on time data based on a time until the light irradiated from the irradiation unit is reflected by each point of the point group and imaged by the imaging unit; (2) a three-dimensional coordinate calculation step of calculating three-dimensional coordinate data composed of three-dimensional coordinates of each point based on the distance to each point and the two-dimensional image data; and (3) a meshing step of creating polygon data by meshing the point group with a plane of an n-sided polygon in which adjacent points based on the three-dimensional coordinate data are connected by lines. The modeling method is characterized by having the calculation step.

7. The irradiation step irradiates light in a dot pattern. The three-dimensional coordinate calculation step calculates the three-dimensional coordinate data of the point group by determining a z coordinate based on the distance to each point of the point group and determining an x coordinate and a y coordinate based on the two-dimensional image data for each point of the point group. The modeling method according to claim 6 is characterized by this.

8. The calculation step has, between the three-dimensional coordinate calculation step and the meshing step, a missing point complementing step of calculating three-dimensional coordinates of points on the surface of the object that could not be detected in the imaging step based on the light distribution characteristics of the light irradiated by the irradiation unit and the three-dimensional coordinates, and adding the three-dimensional coordinates to the three-dimensional coordinate data. The modeling method according to claim 6 or 7 is characterized by this.

9. The imaging step is characterized in that, based on the three-dimensional coordinates, the shorter the distance to the object, the shorter the exposure time for the imaging unit to image light. The modeling method according to claim 6 or 7 is characterized by this.

10. The imaging step is characterized in that, based on the three-dimensional coordinates, the shorter the distance to the object, the fewer the number of integration times for the imaging unit to image light. The modeling method according to claim 6 or 7 is characterized by this.

11. A modeling program for creating polygon data of a point group consisting of a plurality of points on the surface of an object, the program causing a computer to execute: a distance calculation step of calculating the distance to each point of the point group based on time data based on the time from when the light irradiated from the irradiation unit is reflected by each point of the point group until it is imaged by the imaging unit; a three-dimensional coordinate calculation step of calculating three-dimensional coordinate data consisting of the three-dimensional coordinates of each point based on the distance to each point and the two-dimensional image data imaged by the imaging unit; and a meshing step of creating polygon data by meshing the point group into a plane of an n-sided polygon connecting adjacent points based on the three-dimensional coordinate data.

12. The modeling program according to claim 11, the program causing a computer to execute: an irradiation step of irradiating the object with light of a predetermined pattern by the irradiation unit; and an imaging step of imaging the reflected light of the light irradiated from the irradiation unit by the object by the imaging unit and detecting two-dimensional image data of the object.

13. The modeling program according to claim 12, the program causing a computer to execute, between the three-dimensional coordinate calculation step and the meshing step, a missing point complementing step of calculating the three-dimensional coordinates of points on the surface of the object that could not be detected in the imaging step based on the light distribution characteristics of the light irradiated by the irradiation unit and the three-dimensional coordinates, and adding them to the three-dimensional coordinate data.

14. The modeling program according to claim 12 or 13, the program causing a computer to execute the imaging step so that the exposure time for the imaging unit to image light is shorter as the distance to the object based on the three-dimensional coordinates is closer.

15. The modeling program according to claim 12 or 13, the program causing a computer to execute the imaging step so that the number of integration times for the imaging unit to image light is smaller as the distance to the object based on the three-dimensional coordinates is closer.

Images