Shape evaluation device, shape evaluation method, and shape evaluation program

Abstract

To provide a shape evaluation device, shape evaluation method, and shape evaluation program that can perform shape evaluation of laminated structures with minimal effort and high accuracy. [Solution] The shape evaluation device comprises a reference plane calculation unit, a cross-sectional point cloud calculation unit, a cross-sectional shape detection unit, and a detection result aggregation unit. The reference plane calculation unit calculates a reference plane that serves as the basis for cutting from 3D measurement data of a stacked structure composed of multiple objects. The cross-sectional point cloud calculation unit cuts the 3D measurement data with a plane parallel to the reference plane and calculates cross-sectional point cloud data representing each cut surface. The cross-sectional shape detection unit detects the shape of each cut surface from the respective cross-sectional point cloud data. The detection result aggregation unit aggregates the detection results of the shape of each cut surface and evaluates the displacement of each cut surface as the displacement of objects in the stacked structure.

Description

Technical Field

[0001] Embodiments relate to a shape evaluation apparatus, a shape evaluation method, and a shape evaluation program.

Background Art

[0002] In a laminated structure formed by stacking a plurality of objects having similar shapes, it is required to accurately evaluate whether each object is stacked in alignment without displacement. As a method for evaluating the shape of the laminated structure for this purpose, for example, a method can be considered in which cross-sections are manually cut out one by one from three-dimensional measurement data obtained by three-dimensionally measuring the laminated structure, and the degree of displacement of each cross-section with respect to other cross-sections is evaluated. However, in such a shape evaluation method, the number of processing steps tends to increase when the number of cross-sections is large, that is, when the number of objects to be stacked is large.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Embodiments provide a shape evaluation apparatus, a shape evaluation method, and a shape evaluation program that can perform shape evaluation of a laminated structure with a small number of steps and with high accuracy.

Means for Solving the Problems

[0005] One embodiment of the shape evaluation device comprises a reference plane calculation unit, a cross-sectional point cloud calculation unit, a cross-sectional shape detection unit, and a detection result aggregation unit. The reference plane calculation unit calculates a reference plane that serves as the basis for cutting from 3D measurement data of a stacked structure composed of multiple objects. The cross-sectional point cloud calculation unit cuts the 3D measurement data with a plane parallel to the reference plane and calculates cross-sectional point cloud data representing each cut surface. The cross-sectional shape detection unit detects the shape of each cut surface from the respective cross-sectional point cloud data. The detection result aggregation unit aggregates the detection results of the shapes of each cut surface and evaluates the displacement of each cut surface as the displacement of objects in the stacked structure. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 shows an example configuration of a measurement system according to the embodiment. [Figure 2] Figure 2 shows an example of the hardware configuration of a shape evaluation device. [Figure 3] Figure 3 is a flowchart showing the operation of the shape evaluation device. [Figure 4] Figure 4 is a diagram illustrating an example of the calculation of a reference plane. [Figure 5] Figure 5 is a flowchart showing an example of the process for acquiring cross-sectional point cloud data. [Figure 6] Figure 6 is a conceptual diagram of the setting of the cutting surface. [Figure 7] Figure 7 is a flowchart showing the shape detection process for the first example. [Figure 8] Figure 8 is a diagram illustrating the detection of circular elements. [Figure 9] Figure 9 is a flowchart showing the shape detection process for the second example. [Figure 10] Figure 10 is a diagram illustrating the detection of linear elements. [Figure 11] Figure 11 shows an example of a cross-section with a protrusion. [Figure 12] Figure 12 shows a first example of display on a display device. [Figure 13] Figure 13 shows a second example of display on a display device. [Modes for carrying out the invention]

[0007] Embodiments will be described below with reference to the drawings. Figure 1 is a diagram showing the configuration of an example of a measurement system according to the embodiment. As shown in Figure 1, the measurement system 1 includes a camera 10 and a shape evaluation device 20. The shape evaluation device 20 is configured to communicate with a display device 30. Communication between the shape evaluation device 20 and the display device 30 may be performed wirelessly or by wire.

[0008] Camera 10 is a camera configured to measure information related to the three-dimensional shape of the object to be measured O. In this embodiment, the object to be measured O is a stacked structure formed by stacking multiple objects of similar shapes. The object to be measured O could be a stack of books, a stack of cardboard boxes, a forklift pallet, a stack of ship containers, etc. Camera 10 is, for example, an RGB-D camera. An RGB-D camera is a camera configured to measure an RGB-D image as information related to the three-dimensional shape of the object to be measured O. An RGB-D image includes a depth image and an RGB color image. The depth image is an image in which the depth of each point of the object to be measured is the value of the pixel. The color image is an image in which the RGB values ​​of each point of the object to be measured are the values ​​of the pixel. The RGB-D camera measures information related to the three-dimensional shape of the object to be measured O while moving in the circumferential and height directions of the object to be measured O. Note that camera 10 does not have to be an RGB-D camera as long as it can measure information related to the three-dimensional shape of the object to be measured O. For example, camera 10 may be a stereo camera or the like.

[0009] The shape evaluation device 20 is a computer such as a personal computer or a tablet terminal, and has a measurement data generation unit 21, a measurement database (DB) 22, a reference plane calculation unit 23, a cross-sectional point cloud calculation unit 24, a cross-sectional point cloud database (DB) 25, a cross-sectional shape detection unit 26, a shape detection result database (DB) 27, a detection result aggregation unit 28, and a display control unit 29.

[0010] The measurement data generation unit 21 generates 3D measurement data of the measurement object O from information related to the 3D shape of the measurement object O obtained by the camera 10. For example, the measurement data generation unit 21 generates point cloud data from the depth image of the measurement object O measured by the camera 10, and generates polygon data as 3D measurement data from the point cloud data. The point cloud data is data composed of 3D information corresponding to the xyz coordinates of the camera 10 based on the depth measured by the camera 10, for example. Also, the polygon data is data in, for example, the STL (stereolithography) format. For example, the measurement data generation unit 21 can generate polygon data by attaching a triangular mesh to the point cloud data. The polygon data is not limited to data in the STL format.

[0011] The measurement DB 22 is a database for storing the 3D measurement data generated by the measurement data generation unit 21.

[0012] The reference plane calculation unit 23 calculates a reference plane from the 3D measurement data stored in the measurement DB 22. The reference plane is a plane that serves as a reference when cutting out cross-sectional point cloud data from the 3D measurement data, and is, for example, a plane parallel to the plane on which the measurement object O is placed. The reference plane can be data of a cross-sectional point cloud obtained by cutting the 3D measurement data at a predetermined position in the height direction, that is, the z direction. The predetermined position can be, for example, the topmost position, the bottommost position, or the central position of the measurement object O. The predetermined position may also be a position specified by the user.

[0013] The cross-sectional point group calculation unit 24 calculates a plurality of cross-sectional point group data based on a reference plane. The cross-sectional point group can be data of a cross-sectional point group representing a cross-sectional plane parallel to the reference plane, obtained by cutting at a plurality of cutting positions in the height direction parallel to the reference plane. The cutting position is a position in the height direction based on the reference plane. The cutting positions may be set at equal intervals or unequal intervals from the reference plane. The cutting position may be a position specified by the user. Also, the cutting position may be determined from the three-dimensional measurement data. As will be described later, in the embodiment, the positional deviation and angular deviation of each object constituting the laminated structure are evaluated. For the evaluation, when the locations with large positional deviation and angular deviation can be specified from the three-dimensional measurement data, the interval between the cutting positions near the locations with large positional deviation and angular deviation may be set to be narrower than the cutting positions at other locations.

[0014] The cross-sectional point group DB 25 is a database for storing the cross-sectional point group data of each cutting position calculated by the cross-sectional point group calculation unit 24. The cross-sectional point group DB 25 may be a database constructed in the same storage as the measurement DB 22, or may be a database constructed in a different storage.

[0015] The cross-sectional shape detection unit 26 detects the cross-sectional shape of the object represented by each cross-sectional point group data from the cross-sectional point group data of each cutting position stored in the cross-sectional point group DB 25. The cross-sectional shape detection unit 26 detects the cross-sectional shape by fitting to the cross-sectional point group data.

[0016] The shape detection result DB 27 is a database for storing the detection results of the cross-sectional shapes detected by the cross-sectional shape detection unit 26. The shape detection result DB 27 may be a database constructed in the same storage as the measurement DB 22 and the cross-sectional point group DB 25, or may be a database constructed in a different storage.

[0017] The detection result aggregation unit 28 evaluates the positional and angular displacement of each object constituting the laminated structure by aggregating the detection results of each cross-sectional shape stored in the shape detection result DB 27.

[0018] The display control unit 29 is an output control unit that displays the evaluation results obtained by the detection result aggregation unit 28 on the display device 30. For example, the display control unit 29 displays the amount of positional displacement of an object in a visual way.

[0019] Figure 2 shows an example of the hardware configuration of the shape evaluation device 20. The shape evaluation device 20 can be various terminal devices such as a personal computer (PC) or a tablet terminal. As shown in Figure 2, the shape evaluation device 20 has a processor 201, a ROM 202, a RAM 203, a storage 204, an input interface 205, and a communication device 206 as hardware.

[0020] The processor 201 is a processor that controls the overall operation of the measurement system 1. The processor 201 operates as the measurement data generation unit 21, the reference plane calculation unit 23, the cross section point cloud calculation unit 24, the cross section shape detection unit 26, the detection result aggregation unit 28, and the display control unit 29 by executing a program stored in the storage 204, for example. The processor 201 is, for example, a CPU (Central Processing Unit). The processor 201 may also be an MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), etc. The processor 201 may be a single CPU, etc., or multiple CPUs, etc.

[0021] ROM (Read Only Memory) 202 is a non-volatile memory. ROM 202 stores the startup program for the measurement system 1, etc. RAM (Random Access Memory) 203 is a volatile memory. RAM 203 is used, for example, as working memory during processing in the processor 201.

[0022] Storage 204 is, for example, a hard disk drive or a solid-state drive. Storage 204 stores various programs executed by the processor 201, such as a shape evaluation program. Furthermore, storage 204 may be configured to store 3D measurement data as measurement DB 22, or to store cross-sectional point cloud data as cross-sectional point cloud DB 25, or to store cross-sectional shape detection results as shape detection result DB 27.

[0023] The input interface 205 includes input devices such as a touch panel, keyboard, and mouse. When an input device of the input interface 205 is operated, a signal corresponding to the operation is input to the processor 201. The processor 201 performs various processes according to this signal. The input interface 205 can be used, for example, when a user specifies a reference plane.

[0024] The communication device 206 is a communication device that allows the shape evaluation device 20 to communicate with external devices such as the camera 10 and the display device 30. The communication device 206 may be a communication device for wired communication or a communication device for wireless communication.

[0025] Next, the operation of the shape evaluation device 20 in the embodiment will be described. Figure 3 is a flowchart showing the operation of the shape evaluation device 20. The process in Figure 3 is executed by the processor 201. Here, it is assumed that prior to the process in Figure 3, 3D measurement data of the object to be measured O has been acquired. The 3D measurement data of the object to be measured O may be acquired by having the processor 201 perform measurement of the object to be measured O with the camera 10, or it may be acquired by a user holding the camera 10 in their hand and performing measurement of the object to be measured O. For example, when the processor 201 performs measurement of the object to be measured O with the camera 10, the processor 201 performs 3D measurement with the camera 10 while moving the camera 10 in the circumferential and height directions of the object to be measured O. Then, the processor 201 generates 3D measurement data from the information related to the 3D shape of the object to be measured O acquired at each position of the camera 10, and stores the generated 3D measurement data in, for example, the storage 204.

[0026] In step S1, the processor 201 calculates the reference plane. After calculating the reference plane, the process moves to step S2. For example, if the camera 10 is moved by the control of the processor 201 so that the top surface of the object to be measured O is measured, the 3D measurement data D will include data of the top surface F0, as shown in Figure 4. Therefore, if the top surface is to be used as the reference plane, the processor 201 can calculate the reference plane simply by extracting the point cloud data of the top surface F0 from the 3D measurement data. As a method for detecting the point cloud data of the top surface F0 from the 3D measurement data, for example, plane RANSAC (Random Sample Consensus) can be used. In plane RANSAC, three points are randomly selected from the point cloud, and a score based on the number of neighboring points of the plane containing these three points is repeatedly counted, and the plane with the highest score is used as the final plane. In the calculation of the reference plane, one point P0 of the top surface F0 may be specified by the user. In this case, the processor 201 may fix point P0 specified by the user and randomly select the remaining two points to perform plane RANSAC. Since point P0 specified by the user is a point on the reference plane, it is expected that the calculation of the reference plane will be completed earlier than if all three points were randomly selected. Alternatively, the calculation of the reference plane may be performed by virtually drawing a plane based on a landmark shape element such as a sphere. In other words, the calculation of the reference plane may be performed by any method.

[0027] In step S2, the processor 201 acquires N sets of cross-sectional point cloud data from the 3D measurement data. After acquiring the cross-sectional point cloud data, the process moves to step S3. The process of acquiring the cross-sectional point cloud data is described below. Figure 5 is a flowchart of an example of the process of acquiring cross-sectional point cloud data.

[0028] In step S101, the processor 201 acquires 3D measurement data from, for example, the storage 204.

[0029] In step S102, the processor 201 sets the parameter i, which represents the cutting surface to be cut, to an initial value of 0.

[0030] In step S103, the processor 201 sets the cutting plane. Setting the cutting plane involves, for example, the normal vector n of the cutting plane. i and a point P on the cross-section i This is done by setting the normal vector n of the cross-section. i This is equal to the normal vector n0 of the reference plane. On the other hand, a point P on the cross-section i This is a predetermined distance d in the height direction from a point on the immediately preceding cross-section. i It is a point located at a distance of d. In the first case, the cross-section is the reference plane F0. Figure 6 is a conceptual diagram of the setting of the cross-section. As shown in Figure 6, for the next cross-section F0 of the reference plane F0, a normal vector n1 and point P1 are set. Normal vector n1 is equal to normal vector n0. On the other hand, a point P1 on the cross-section F1 is a point located at a predetermined distance d0 from, for example, point P0 on the reference plane F0 in the direction of normal vector n0. The size of the stacked objects is not taken into consideration when the distance d i If this is determined, it is possible that two or more cross-sections may be set for a single object. Also, if point cloud data of the reference plane has already been obtained in step S2, the setting of cross-sections for the reference plane may be omitted.

[0031] In step S104, the processor 201 extracts planar data of the cross-section from the 3D measurement data. For example, the processor 201 extracts point P i It passes through and its normal vector is n i The planar data of (=n0) is extracted from the 3D measurement data.

[0032] In step S105, the processor 201 calculates the coordinates of each point in the cross-sectional point cloud included in the extracted planar data of the cross-section. For example, the processor 201 calculates the coordinates of the intersection points between the point cloud data constituting the 3D measurement data and the planar data of the cross-section extracted in step S104. Regarding the processing in steps S104 and S105, for example, if the 3D measurement data is STL data, the shape of the object is represented by a set of triangles. In this case, the coordinates of each point in the cross-sectional point cloud can be calculated by extracting the triangles that intersect the cross-section from the triangles constituting the 3D measurement data, and finding the coordinates of the intersection points between the extracted triangles and the cross-section.

[0033] In step S106, the processor 201 calculates the coordinates of the centroid of the cross-section point cloud. The coordinates of each point in the cross-section point cloud calculated in step S105 are the coordinates in the coordinate system of the camera 10. Shape detection is preferably performed after converting the coordinates of each point in the cross-section point cloud to the coordinates in the coordinate system of the corresponding cutting plane. The coordinates of the centroid of the cross-section point cloud are used as the coordinates of the origin in the coordinate system of the cutting plane when converting the coordinates of each point in the cross-section point cloud from the coordinate system of the camera 10 to the coordinate system of the cutting plane.

[0034] In step S107, the processor 201 converts the coordinates of each point in the cross-sectional point cloud calculated in step S104 into coordinates in the cross-sectional plane coordinate system. If the coordinate axes of the cross-sectional plane coordinate system are the ξ axis and the η axis, then the ξ axis is the axis parallel to the horizontal direction of the cross-sectional plane, and the η axis is the axis parallel to the vertical direction of the cross-sectional plane. The origin of the cross-sectional plane coordinate system is the centroid of the cross-sectional point cloud. The coordinate transformation is performed, for example, by applying a transformation matrix to the coordinates of each point that corresponds to the positional and orientational shifts between the camera coordinate system and the cross-sectional plane coordinate system.

[0035] In step S108, the processor 201 stores the coordinates of each point calculated in step S107 as the i-th cross-sectional point cloud data, for example, in RAM 203.

[0036] In step S109, the processor 201 determines whether i is less than N. N is the number of sets of cross-sectional point cloud data, which can be set in advance. N may also be set by the user. If it is determined in step S109 that i is less than N, the process proceeds to step S110. If it is determined in step S109 that i is not less than N, the process in Figure 5 ends.

[0037] In step S110, processor 201 increments i by 1. Then, the process returns to step S103.

[0038] Now, let's return to the explanation of Figure 3. In step S3, after acquiring the cross-sectional point cloud data, the processor 201 initializes i to 0.

[0039] In step S4, the processor 201 determines whether i is less than N. If it is determined in step S4 that i is less than N, the process proceeds to step S5. If it is determined in step S4 that i is not less than N, the process proceeds to step S8.

[0040] In step S5, the processor 201 detects the shape of the cross-section based on the i-th cross-sectional point cloud data. After detecting the shape of the cross-section, the process proceeds to step S6. The cross-section shape detection process is described below.

[0041] Figure 7 is a flowchart of the shape detection process in the first example. The shape detection process in the first example is a process for detecting circular elements in the cross-section. Figure 7 shows the detection process using RANSAC.

[0042] In step S201, the processor 201 initializes the parameter j, which represents the number of RANSAC iterations, to 0.

[0043] In step S202, the processor 201 initializes the parameters s_max, c, and r. s_max represents the maximum score, which will be explained later. c represents the coordinates of the center of the circle. r represents the radius of the circle. The processor 201 initializes s_max to, for example, 0. The processor 201 also initializes c to, for example, (0,0). The processor 201 also initializes r to, for example, 0.

[0044] In step S203, the processor 201 determines whether j is less than M. M is the maximum number of RANSAC iterations and can be set in advance. If it is determined in step S203 that j is less than M, the process proceeds to step S204. If it is determined in step S203 that j is not less than M, the process in Figure 7 ends. At this time, the processor 201 outputs the current pts_extr, c, and r as return values. Here, pts_extr is a parameter that indicates the coordinates of the point cloud representing the circular elements.

[0045] In step S204, the processor 201 randomly selects three points from the i-th cross-sectional point cloud data. Figure 8 is a diagram illustrating the detection of circular elements. For example, the processor 201 selects points p1, p2, and p3 from the cross-sectional point cloud data pc shown in Figure 8.

[0046] In step S205, the processor 201 finds the center and radius of the circle passing through the three selected points, stores the coordinates of the center in c_new, and stores the value of the radius in r_new. In the example in Figure 8, the processor 201 finds the center and radius of circle C passing through points p1, p2, and p3.

[0047] In step S206, the processor 201 determines whether r_new is between r_min and r_max. r_min and r_max are predetermined constants. If it is determined in step S206 that r_new is between r_min and r_max, the process proceeds to step S207. If it is determined in step S206 that r_new is not between r_min and r_max, the process proceeds to step S210. In other words, if it is determined that r_new is not between r_min and r_max, i.e., if a circle that is too small or too large is detected, the detection of a circle element is considered unsuccessful.

[0048] In step S207, the processor 201 calculates the coordinates of the neighboring points of the circle and the number of neighboring points, stores the coordinates of each calculated neighboring point in pts_new, and stores the number of calculated neighboring points in s_new. Neighboring points are, for example, points whose distance in the normal direction from each point on the circumference is within a threshold.

[0049] In step S208, the processor 201 determines whether s_new is greater than s_max. If it is determined in step S208 that s_new is greater than s_max, the process proceeds to step S209. If it is determined in step S208 that s_new is not greater than s_max, the process proceeds to step S210.

[0050] In step S209, the processor 201 stores s_new in s_max, pts_new in pts_extr, r_new in r, and c_new in c. Then, the process proceeds to step S210.

[0051] In step S210, processor 201 increments j by 1. Then, the process returns to step S203.

[0052] Figure 9 is a flowchart of the shape detection process in the second example. The shape detection process in the second example is a process for detecting rectangular elements in a cross-section, that is, elements formed by multiple line segments. Figure 9 shows the detection process using RANSAC.

[0053] In step S301, the processor 201 initializes the parameter j, which represents the number of RANSAC iterations, to 0.

[0054] In step S302, the processor 201 initializes the parameters pts and pts_extr, respectively. pts is a parameter that represents the state of the cross-sectional point cloud data. pts_extr is a parameter that indicates the coordinates of the point cloud representing the four sides of the rectangular element. The processor 201 stores the i-th cross-sectional point cloud data in pts. The processor 201 also initializes each element of pts_extr to, for example, (0,0).

[0055] In step S303, the processor 201 determines whether j is less than 4. If it is determined in step S303 that j is less than 4, the process proceeds to step S304. If it is determined in step S303 that j is not less than 4, the process proceeds to step S307.

[0056] In step S304, the processor 201 detects straight lines in the cross-sectional point cloud data using Line RANSAC and stores the coordinates of the neighboring points of the detected straight lines in pts_extr. Figure 10 is a diagram illustrating the detection of straight line elements. In Line RANSAC, the processor 201 randomly selects two points from the cross-sectional point cloud data pc. In Figure 10, it is assumed that the processor 201 has selected p1 and p2. Next, the processor 201 detects a straight line passing through the two selected points and counts the number of neighboring points of that straight line as a score. In the example in Figure 10, the processor 201 counts the number of neighboring points of the straight line L as a score. The processor 201 repeats this process while changing the two points selected, and stores the coordinates of the neighboring points of the straight line that obtains the highest score as a straight line constituting one side of a rectangle in pts_extr.

[0057] In step S305, the processor 201 deletes the data of neighboring points of the line detected in step S304 from pts.

[0058] In step S306, processor 201 increments j by 1. Then, the process returns to step S303.

[0059] In step S307, the processor 201 divides the coordinates of the four point clouds that make up the four sides of the rectangle into pairs of opposite sides.

[0060] In step S308, the processor 201 calculates the width w, height h, center position c, and inclination angle θ of the rectangle by least-squares fitting with constraints that the rectangle is parallel to opposite sides and perpendicular to adjacent sides for the two paired point clouds. After that, the processing in Figure 9 is completed. At this time, the processor 201 outputs w, h, c, and θ as return values. In the example in Figure 10, the processor 201 calculates the width w, height h, center position c, and inclination angle θ of the rectangle R by least-squares fitting. The width w is the horizontal length of the rectangle R. The height h is the vertical length of the rectangle R. The center position c is the coordinates of the center of the rectangle R, i.e., the intersection of the diagonals. The inclination angle θ is the rotation angle of the rectangle R with respect to the ξ axis and the η axis.

[0061] In the example shown in Figure 9, by changing the number of lines to be detected and the constraint conditions, polygons other than quadrilaterals can also be detected. Furthermore, by combining the detection of circular elements described in the first example and the detection of rectangular elements described in the second example, the shape of a cross-section containing both circular and rectangular elements can also be detected.

[0062] However, it is desirable that the cross-section in the embodiment includes a specific shape element. This specific shape element is a shape element that can be represented by a piecewise linear function, such as a polygon in a plane Cartesian coordinate system, or a shape element that can be represented by a quadratic form, such as a circle or ellipse in a plane Cartesian coordinate system, and whose free parameter value can be uniquely determined from the coordinate values ​​of one or more points extracted from the point cloud, up to a number of free parameters, and which is a shape element for which a distance function that determines whether a point on an arbitrary plane is an ε-neighbor of the shape can be explicitly expressed, excluding piecewise boundaries. If it includes a specific shape element, the cross-section may have protrusions, indentations, deformations, or defects. For example, the cross-section may have a protrusion pr as shown in Figure 11. With shape detection by RANSAC, the shape of the cross-section can be detected even if there is a protrusion pr.

[0063] Furthermore, in the example shown in Figure 4, since objects of substantially identical shape are stacked, the cross-sections also have substantially identical shapes. On the other hand, the shapes of the objects being stacked do not necessarily have to be identical. Also, objects of the same shape but with different cross-sectional areas may be stacked. For example, the stacked structure may be a conical stacked structure in which the cross-sectional area decreases towards the top.

[0064] Now, let's return to the explanation of Figure 3. In step S6, after the detection of the cross-sectional shape, the processor 201 saves the detection result of the cross-sectional shape for the i-th cross-sectional point cloud data, for example, to the storage 204. If the detected cross-sectional shape contains circular elements, the processor 201 saves pts_extr, c, and r. Also, if the detected cross-sectional shape contains rectangular elements, the processor 201 saves w, h, c, and θ.

[0065] In step S7, processor 201 increments i by 1. Then the process returns to step S4.

[0066] In step S8, if it is determined in step S4 that i is not less than N, the processor 201 aggregates the detection results. For example, as part of the aggregation of detection results, the processor 201 calculates the deviation of the center position c of each cross-section relative to the reference plane, based on the detection result of the center position c of each cross-sectional shape. The processor 201 also calculates the deviation of the inclination angle θ of each cross-section relative to the reference plane, based on the detection result of the inclination angle θ of each cross-sectional shape. Here, both the center position c and the inclination angle θ are calculated in the coordinate system of the corresponding cross-section. Therefore, the processor 201 converts the center position c and inclination angle θ for each cross-section to values ​​in the camera's coordinate system before performing the aggregation. Furthermore, when aggregating the detection results, the processor 201 may calculate the deviation relative to an average value, etc., instead of the deviation relative to the reference plane.

[0067] In step S9, the processor 201 displays the evaluation results on the display device 30. After the display is finished, the process shown in Figure 3 is completed. Here, instead of or in addition to displaying the evaluation results on the display device 30, the processor 201 may perform various output processes, such as sending the evaluation results to a server not shown in the figure. Furthermore, when performing the output process, the processor 201 may output a file that includes 3D measurement data in addition to the evaluation results.

[0068] Figure 12 shows a first example of the display on the display device 30. In the display screen 300 of the first example, a graph 301 is displayed that shows, for example, the displacement or rotational displacement of the center position of each cross-section. The graph 301 in Figure 12 is a graph that shows the rotational displacement. The horizontal axis of the graph in Figure 12 is the z coordinate, that is, the height of each cross-section as seen in the coordinate system of the camera 10. On the other hand, the vertical axis of the graph in Figure 12 is, for example, the rotation angle of each cross-section with respect to a reference plane. The graph 301 in Figure 12 allows the user to easily understand how much an object at each height position in the stacked structure is rotated.

[0069] Furthermore, the display screen 300 may also display a positional misalignment button 302 and a rotational misalignment button 303. The positional misalignment button 302 is a button for displaying a graph of the misalignment of the center position of the cross-section as graph 301. When the positional misalignment button 302 is selected, graph 301 switches to a graph representing the misalignment of the center position of the cross-section. In the graph representing the misalignment of the center position of the cross-section, the horizontal axis is the z-coordinate, and the vertical axis is a graph of the misalignment of the center position of each cross-section relative to, for example, a reference plane. With such a graph, the user can easily understand how much the object at each height position in the laminated structure is misaligned. The rotational misalignment button 303 is a button for displaying a graph of the rotational misalignment of the cross-section as graph 301. When the rotational misalignment button 303 is selected, graph 301 switches to a graph representing the rotational misalignment shown in Figure 12.

[0070] Figure 13 shows a second example of the display on the display device 30. On the second display screen 300, a three-dimensional model 311 of the object to be measured O is displayed based on the three-dimensional measurement data of the object to be measured O. When any part of the three-dimensional model 311 of the object to be measured O is selected, for example, by the cursor 312, a display 313 is shown indicating the displacement and rotational displacement of the center position of the cross-section corresponding to the z coordinate of the selected part. With such a display, the user can intuitively grasp the displacement of the desired height position.

[0071] As described above, according to the embodiment, point cloud data of cross-sections are extracted from 3D measurement data at equal or unequal intervals, the shape of each cross-section represented by the extracted point cloud data is detected, and the deviation between the detected cross-section shapes is evaluated. In this way, in the embodiment, the positional and rotational deviations of objects constituting the laminated structure can be evaluated without the user having to cut out cross-sections one by one from the 3D measurement data. In other words, in the embodiment, the shape evaluation of the laminated structure can be performed with less effort and with high accuracy.

[0072] (Variation 1) Modifications of the embodiment will now be described. In this embodiment, RANSAC is used to detect the cross-sectional shape. However, methods other than RANSAC may be used to detect the cross-sectional shape. For example, the least squares method or the weighted least squares method may be used to detect the cross-sectional shape. Alternatively, pattern matching using known shape patterns such as circles and rectangles may be used to detect the cross-sectional shape.

[0073] (Modification 2) In this embodiment, polygon data such as STL data is generated as 3D measurement data, and point cloud data of the cross-section is extracted from the polygon data. Alternatively, the 3D measurement data may be point cloud data. In this case, the extraction of the cross-section can also be performed by fitting using RANSAC or the like.

[0074] (Other variations) While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0075] 1 Measurement system, 10 Camera, 20 Shape evaluation device, 21 Measurement data generation unit, 22 Measurement database (DB), 23 Reference plane calculation unit, 24 Cross section point cloud calculation unit, 25 Cross section point cloud database (DB), 26 Cross section shape detection unit, 27 Shape detection result database (DB), 28 Detection result aggregation unit, 29 Display control unit, 30 Display device, 201 Processor, 202 ROM, 203 RAM, 204 Storage, 205 Input interface, 206 Communication device.

Claims

1. A reference plane calculation unit calculates a reference plane that serves as the basis for cutting from 3D measurement data of a stacked structure composed of multiple objects, A cross-sectional point cloud calculation unit that cuts the three-dimensional measurement data with a plane parallel to the reference plane and calculates cross-sectional point cloud data representing each cut surface, A cross-sectional shape detection unit that detects the shape of each of the aforementioned cross-sectional planes from each of the cross-sectional point cloud data, A detection result aggregation unit aggregates the detection results of the shape of each of the aforementioned cross-sections and evaluates the displacement of each of the aforementioned cross-sections as the displacement of the object in the laminated structure, A shape evaluation device equipped with the following.

2. The reference plane calculation unit calculates the reference plane using a planar RANSAC on the three-dimensional measurement data, or calculates a plane that is virtually drawn based on a landmark shape element in the three-dimensional measurement data as the reference plane. The shape evaluation apparatus according to claim 1.

3. The cross-section includes a shape element that can be represented by a piecewise linear function or a shape element that can be represented by a quadratic form in a plane orthogonal coordinate system, wherein the value of the free parameter can be uniquely determined from the coordinate values ​​of one or more points extracted from a point cloud, and the distance function includes a shape element that can be explicitly represented excluding the piecewise boundary. The shape evaluation apparatus according to claim 1.

4. This involves calculating a reference plane for cutting from 3D measurement data of a stacked structure composed of multiple objects, and The three-dimensional measurement data is cut by a plane parallel to the aforementioned reference plane, and cross-sectional point cloud data representing each cut surface is calculated. The shape of each cross-section is detected from the respective cross-sectional point cloud data, The detection results for the shape of each of the aforementioned cross-sections are aggregated, and the displacement of each of the aforementioned cross-sections is evaluated as the displacement of the object in the laminated structure, A shape evaluation method comprising the following.

5. This involves calculating a reference plane for cutting from 3D measurement data of a stacked structure composed of multiple objects, and The three-dimensional measurement data is cut by a plane parallel to the aforementioned reference plane, and cross-sectional point cloud data representing each cut surface is calculated. A cross-sectional shape detection unit that detects the shape of each of the aforementioned cross-sectional planes from each of the cross-sectional point cloud data, The detection results for the shape of each of the aforementioned cross-sections are aggregated, and the displacement of each of the aforementioned cross-sections is evaluated as the displacement of the object in the laminated structure, A shape evaluation program to be executed by the processor.

Images