Component inspection method and device

Abstract

In order to inspect a minimum distance between an ignition coil (3) and a fuel tube (4), the present invention involves: generating point cloud data through scanning by a three-dimensional laser scanner (5) (S1); and positioning reference point cloud data based on CAD data (S4-S14). After the positioning, the measured point cloud data is interpolated using the reference point cloud data (S17), and a minimum distance between given two points is calculated (S19). When the reliability level is low (S15, S16), a loss region is determined (S17) and additional scanning is performed. In this case, a multitude of virtual viewpoints that are set around the region are assessed and measurement parameters for a trajectory and the like are calculated so as to connect advantageous points (S23), and an instruction for additional scanning along the trajectory is given (S24).

Description

[Technical Field]

[0001] The present invention relates to a part inspection method and apparatus for inspecting one or more parts in an assembly formed by assembling a plurality of parts in relation to their spatial arrangement. [Background technology]

[0002] For example, in the inspection process for completed automobiles, inspectors may visually or use a ruler to inspect the positions of numerous parts assembled in the engine compartment (e.g., inspecting the gaps between adjacent parts). In recent years, attempts have been made to inspect the mounting state or spatial position of specific parts in such assemblies by acquiring three-dimensional data of the external shapes of each part through measurements using a three-dimensional sensor (e.g., Patent Document 1).

[0003] In an assembly where multiple parts are assembled in three dimensions, when measuring from the outside with a 3D sensor, there may be areas of the target parts where data cannot be acquired, known as missing areas, due to factors such as improper orientation of the 3D sensor or obstruction by other parts. These missing areas can reduce the accuracy of identifying the shape and position of the parts.

[0004] Patent Document 1 discloses that the missing areas are used as features to estimate the three-dimensional position and orientation of a target object and to identify the object, but does not disclose anything about reducing the missing areas during measurement. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-174891 Summary of the Invention

[0006] The present invention provides a part inspection method for inspecting the spatial arrangement of one or more target parts in an assembly formed by assembling a plurality of parts, comprising the steps of: The area including the target part is scanned from the outside with a 3D sensor to obtain measurement data including part of the outer surface of the target part, Acquire design data including the individual external shapes of the target parts and the positional relationships of the parts within the assembly; determining a missing area in the measurement data of the target part by comparing with design data of the target part; Determine the measurement parameters of the 3D sensor suitable for acquiring measurement data of this defect area, Additional scanning is performed along these measurement parameters.

[0007] According to this invention, by determining measurement parameters for a 3D sensor suitable for obtaining measurement data for a missing area and performing additional scanning in accordance with these parameters, the missing area can be made smaller, thereby improving the measurement accuracy that is a prerequisite for inspecting the target part. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is an explanatory diagram showing various components in an engine compartment, which is an assembly according to an embodiment, viewed from above. [Figure 2] FIG. 2 is a functional block diagram of the gap inspection device according to the embodiment. [Figure 3] Point cloud data of the engine compartment and its internal parts scanned using a 3D laser scanner. [Figure 4] Point cloud data of the engine bay and its internal components at 40% completion. [Figure 5] FIG. 10 is an explanatory diagram of reliability. [Figure 6] 1 is a flowchart showing the process flow of a gap inspection method according to an embodiment. [Figure 7] FIG. 10 is an explanatory diagram of the movement trajectory of a three-dimensional sensor during scanning. [Figure 8]FIG. 10 is an explanatory diagram showing the deviation between the instruction trajectory and the actual movement trajectory during scanning. [Figure 9] An explanatory diagram of additional scanning of a defect area. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment of the present invention will now be described with reference to the drawings. In this embodiment, the present invention is applied to a gap inspection process in which the spatial positions of two parts in an engine compartment are identified and the minimum gap between them is calculated during the inspection process of a completed automobile.

[0010] FIG. 1 is a top view of some of the various components in an automobile engine compartment 1, which is an assembly of one embodiment, showing the area near the engine intake manifold 2. An ignition coil 3, which is part of the engine's ignition system, is located below the intake manifold 2. A fuel tube 4, which supplies fuel to the engine, is located to the side of the intake manifold 2. The fuel tube 4 is made of a metal tube for oil resistance and fire resistance, and has a circular cross section along its radial direction. As shown in FIG. 1, the fuel tube 4 has a straight portion 4a that extends generally linearly along the side of the intake manifold 2 and an inclined portion 4b that slopes from one end of the straight portion 4a toward the ignition coil 3. The fuel tube 4, made of a metal tube, undergoes some deformation during installation and is therefore considered a non-rigid component. On the other hand, the ignition coil 3 is considered a rigid component whose external shape does not change.

[0011] The minimum spatial distance (minimum gap) between the ignition coil 3 and the fuel tube 4 is generally regulated by law, and therefore inspection of this minimum gap is required during the finished vehicle inspection process. In many cases, the design minimum distance between the ignition coil 3 and the fuel tube 4 is regulated at a position on the outer surface of at least one of the ignition coil 3 or the fuel tube 4 that is not visible from the outside, and it is generally difficult for an inspector to measure using a ruler. In this embodiment, the gap between these two components is determined by external measurement using a three-dimensional sensor and calculation processing.

[0012] As shown in Figure 2, one embodiment of a gap inspection device used in finished vehicle inspections includes a three-dimensional sensor, such as a three-dimensional laser scanner 5, a database 6, a control device 7, and one or more displays 8. In the finished vehicle inspection process, multiple inspections are performed sequentially in a predetermined order. The gap inspection device of one embodiment is configured as part of the inspection equipment used in the finished vehicle inspection process.

[0013] The 3D laser scanner 5 can acquire the 3D coordinates of the surface shape of the measurement target by irradiating the target with a laser and measuring the reflection time. By performing measurements at a high speed of approximately tens of thousands of points per second, high-density point cloud data can be obtained. While various sizes and types of 3D laser scanners are known, one embodiment uses a type that can be held by an operator and used to scan the target. A specific area of ​​the engine compartment 1, including the ignition coil 3, the fuel tube 4, and other components, is measured by manually scanning the target area from the outside along a specified movement trajectory. Scanning with this 3D laser scanner 5 obtains point cloud data for the entire area, including portions of the outer surfaces of the ignition coil 3 and the fuel tube 4, as shown in FIG. 3.

[0014] Although point cloud data is actually acquired as time-series data for each frame by scanning, point cloud data for the entire area can be acquired by overlaying the data. It is also desirable to perform noise reduction processing on the acquired point cloud data to remove noise caused by dust or other particles that may have been introduced during scanning. The 3D laser scanner may also require markers to serve as references for identifying the shape and position of the target part during scanning.

[0015] Here, "scanning" in the present invention is a concept that includes either or both of the planar scanning function of the 3D sensor (e.g., 3D laser scanner 5) itself and the movement of the 3D sensor along a certain trajectory by, for example, an operator. In one embodiment, scanning is performed by using a 3D laser scanner 5 with a planar scanning function and further moving this 3D laser scanner 5 along a predetermined trajectory. Depending on the size and shape of the target area to be measured or the type of 3D sensor, it is also possible, for example, to arrange multiple 3D sensors fixedly at appropriate positions in space and acquire point cloud data of the target area using the scanning function of each 3D sensor.

[0016] The database 6 stores all design data for the automobile to be inspected, and therefore stores shape data for the ignition coil 3 and fuel tube 4, which are the targets of the gap inspection, as design data, along with data indicating their spatial positional relationship. The database 6 also stores shape data and positional relationship data for other components present within the area. The design data is stored in the database 6 in the form of CAD data constituting a mesh, and the necessary design data is read from the database 6 to the control device 7 during the gap inspection. After the design data is read into the control device 7, known hidden surface removal may be performed on areas other than those involved in the gap between the ignition coil 3 and the fuel tube 4 to reduce the data size.

[0017] The control device 7 includes a first alignment unit 7a, a progress calculation unit 7b, a second alignment unit 7c, a third alignment unit 7d, a reliability calculation unit 7e, a first interpolation unit 7f, a second interpolation unit 7g, a distance calculation unit 7h, an information output unit 7i, a missing area determination unit 7j, a measurement parameter calculation unit 7k, and a movement trajectory determination unit 7m. Although not shown, the control device 7 also includes a measurement data acquisition unit that acquires measurement data including a portion of the outer surface of the target part by scanning with the 3D laser scanner 5, as described above, and a design data acquisition unit that acquires design data including the individual outer shapes of the target part and the positional relationship of the part within the engine room 1 from the database 6.

[0018] The first alignment unit 7a roughly aligns the entire target area (the so-called target area in alignment) including the ignition coil 3 and the fuel tube 4 in the measurement data acquired by the 3D laser scanner 5 with the entire area (the so-called reference area in alignment) including the ignition coil 3 and the fuel tube 4 in the design data stored in the database 6. For example, the first alignment unit 7a uses the FPFH algorithm to search for keypoints from point cloud data of the entire target area and describes the characteristics of these keypoints, such as the normal vectors of the keypoints and the relative angles of the surroundings. Furthermore, the first alignment unit 7a converts the design data of the entire reference area into point cloud data using a well-known conversion method, and then similarly searches for keypoints using the FPFH algorithm and describes the characteristics of these keypoints, such as the normal vectors of the keypoints and the relative angles of the surroundings. The first alignment unit 7a then compares the normal vectors between the target keypoints and the reference keypoints and the relative angles between the surrounding point clouds to search for combinations of keypoints that form pairs whose normal vectors and relative angles approximately match. Then, the reference point cloud data is roughly aligned with the target point cloud data using a combination of paired keypoints. Note that during this rough alignment, the non-rigid fuel tube 4 is considered to be a rigid body, and as a result, an average alignment is performed within the length of the fuel tube 4. The rough alignment of the target region by the first alignment unit 7a is performed in parallel with the progress of scanning of the target region. The first alignment unit 7a may also perform the rough alignment of the target region using a known algorithm other than the FPFH algorithm.

[0019] The progress calculation unit 7b calculates the progress of the scanning required in the target area after the rough alignment by the first alignment unit 7a and before the detailed alignment of the ignition coil 3 and the fuel tube 4 (described later). For example, the progress calculation unit 7b calculates the progress of the scanning from the ratio between the number of points in the point cloud data of the target area based on the reference design data and the number of points in the point cloud data that match between the reference and the target through the rough alignment by the first alignment unit 7a. The progress calculation unit 7b further compares the calculated progress with a predetermined progress threshold (e.g., 40% in this embodiment) and determines in real time whether the progress, which gradually increases as the scanning progresses, has exceeded the progress threshold. For example, the information output unit 7i displays the current progress and information on whether the progress has exceeded the progress threshold on the display 8, allowing the operator to continue scanning using the 3D laser scanner 5 according to the display. In other words, scanning by the 3D laser scanner 5 (generation of point cloud data), rough alignment, and progress calculation are repeated in real time until a predetermined progress threshold is exceeded. For example, the area surrounded by the dashed line in Figure 4 corresponds to a progress of 40%.

[0020] In addition to the ratio of the number of data points as described above, the degree of progress may also be evaluated based on the number of viewpoints passed by the 3D laser scanner 5, the number of key points used in rough alignment, etc.

[0021] The second alignment unit 7c performs detailed spatial alignment for each component, provided that the progress exceeds a threshold. Specifically, the second alignment unit 7c searches for paired points in the reference point cloud data for every point in the target point cloud data obtained by measuring the ignition coil 3 and the fuel tube 4, and performs detailed alignment of the reference point cloud data with the target point cloud data. The non-rigid fuel tube 4 is considered to be a rigid body here, and as a result, average alignment is performed within the length of the fuel tube 4. The detailed alignment in the second alignment unit 7c can be performed using an appropriate known algorithm.

[0022] After the detailed alignment by the second alignment unit 7c, the third alignment unit 7d performs so-called non-rigid alignment of non-rigid parts (fuel tube 4 in this embodiment) taking deformation into account. Any known algorithm can be used here. For example, the third alignment unit 7d searches for nearest neighbor pairs from reference point cloud data and target point cloud data, which have been aligned in advance assuming a rigid body, and calculates rotation, scaling, and translation parameters to bring the paired points closer to each other. For example, the outer shape of the fuel tube 4 is downsampled to form an outer surface composed of multiple triangles with vertices and edges. Using the vertices representing the point clouds (clusters) of each downsampled region, i.e., each region containing multiple adjacent triangles, the deformation of the cluster is decomposed into rotation, scaling, and translation parameters under the constraint that the edge length does not change (strictly speaking, the length change is minimized). The deformation of the entire fuel tube 4 is obtained as a collection of such cluster-based deformations. The third alignment unit 7d aligns the reference point cloud data for the fuel tube 4 with the target point cloud data in space while deforming the reference point cloud data for the fuel tube 4 using a non-rigid alignment method that takes such deformation into consideration.

[0023] The reliability calculation unit 7e calculates the reliability of the alignment of each target component (i.e., the ignition coil 3 and the fuel tube 4). In other words, the reliability calculation unit 7e calculates the reliability indicating how close the target point cloud data is to the aligned reference point cloud data for each of the ignition coil 3 and the fuel tube 4.

[0024] FIG. 5 is an explanatory diagram that schematically shows point cloud data of a fuel tube 4 after alignment to explain reliability. For simplicity, it is assumed here that the circular outer surface of a cross section of the fuel tube 4 is formed by 13 pieces of point cloud data. The 13 pieces of point cloud data Dr arranged in a circle are reference point cloud data based on design data, and the seven pieces of point cloud data Dt arranged in a semicircle are target point cloud data based on measurement data. For example, if scanning using the 3D laser scanner 5 is performed only from the top of the figure, the bottom half will be a so-called missing area, and the target point cloud data Dt will be arranged in a semicircle.

[0025] The reliability is expressed, for example, as the ratio between the number of points in the reference point cloud data Dr and the number of points in the target point cloud data Dt that fall within a radius L from each point in the reference point cloud data Dr. In the figure, the range of radius L from the numerous points in the reference point cloud data Dr is represented by an outer circle C1 and an inner circle C2, each indicated by a dashed line. In the example shown in FIG. 5( a), four pieces of target point cloud data Dt fall within the range R of the circles C1 and C2, resulting in a reliability of 4 / 13. In the example shown in FIG. 5( b), two pieces of target point cloud data Dt fall within the range R of the circles C1 and C2, resulting in a reliability of 2 / 13. In the example shown in FIG. 5( c), all of the target points, i.e., seven pieces of point cloud data Dt, fall within the range R, resulting in a reliability of 7 / 13. Thus, the reliability is affected by both the accuracy of alignment and the size or proportion of missing areas in the measurement data.

[0026] The first interpolation unit 7f uses reference point cloud data to interpolate missing portions that have not been scanned in the measurement data of the ignition coil 3, i.e., the target point cloud data. In other words, the first interpolation unit 7f interpolates the outer surface portion of the back (lower) side of the ignition coil 3, which is hidden from view from above the engine compartment 1, using the reference point cloud data to generate point cloud data of the ignition coil 3 including the hidden portion. Then, the first interpolation unit 7f converts the generated point cloud data into mesh data that constitutes a surface using a well-known conversion method.

[0027] Similarly, the second interpolation unit 7g interpolates missing portions that have not been scanned in the measurement data of the fuel tube 4, i.e., the target point cloud data, using the reference point cloud data. In other words, the outer surface portion of the back side (lower side) of the fuel tube 4, which is hidden from view from above the engine compartment 1, is interpolated using the reference point cloud data to generate point cloud data of the fuel tube 4 including the hidden portion. Then, the second interpolation unit 7g converts the generated point cloud data into mesh data that constitutes a surface using a well-known conversion method.

[0028] The distance calculation unit 7h calculates the distance from each point on the surface of the ignition coil 3 to each point on the surface of the fuel tube 4 based on the mesh data of the ignition coil 3 acquired by the first interpolation unit 7f and the mesh data of the fuel tube 4 acquired by the second interpolation unit 7g. Furthermore, the distance calculation unit 7h obtains the minimum distance by comparing the calculated distances with each other. Note that the distance calculation unit 7h may calculate the distance from each point on the ignition coil 3 to each point on the fuel tube 4 based on each point cloud data without converting the point cloud data of the ignition coil 3 and the point cloud data of the fuel tube 4 into mesh data.

[0029] The information output unit 7i generates various images to be displayed on the display 8 and data to be output as sound from a speaker (not shown). For example, the finally determined minimum distance, the progress rate, etc. are displayed on one or more displays 8 that can be visually confirmed by the worker.

[0030] The missing area determination unit 7j determines missing portions, i.e., missing areas, in the measurement data for each target component (ignition coil 3 and fuel tube 4). For example, the surface to be scanned in the design data of each component is divided into multiple grids, and the proportion of scanned points for each grid (the ratio of the number of points actually scanned to the number of points to be scanned) is calculated. If this proportion is below a predetermined threshold, the grid is determined to be a missing area. This allows the determination of which portions of the target component's design data are missing areas.

[0031] Note that surface areas that are not originally targeted for scanning (for example, areas that are surrounded by other parts and cannot be seen linearly) are not subject to determination of missing areas. In other words, if the scanning operation is performed appropriately, measurement data can be acquired, and even if the area is expected to have measurement data, a missing area is a part where measurement data is missing.

[0032] For the defective areas determined by the defective area determination unit 7j, the measurement parameter calculation unit 7k calculates the measurement parameters of the measurement by the 3D laser scanner 5 required to acquire measurement data for these defective areas. The measurement parameters refer to the position and orientation direction (in other words, angle) of the 3D laser scanner 5, as well as the measurement mode, such as laser intensity, if the measurement mode can be changed. In one embodiment, the worker performs scanning while holding the 3D laser scanner 5 in his hand and moving it, so the position and orientation direction of the 3D laser scanner 5 required to acquire measurement data for the defective areas are generated as a continuous movement trajectory.

[0033] The measurement parameters thus generated (for example, the necessary movement trajectory and measurement mode) are displayed on the display 8 via the information output unit 7i. The position and range of the missing area may also be displayed. The operator will perform additional scanning according to the display on the display 8.

[0034] FIG. 9 is an explanatory diagram illustrating the principle of measurement parameter calculation in the measurement parameter calculation unit 7k. In this simplified view, the entire area requiring measurement, such as the engine room 1 described above, is shown as the measurement object OJ. The measurement object OJ includes an area OJa for which measurement data has already been acquired and a missing area OJb for which measurement data is missing. The measurement parameter calculation unit 7k first sets multiple virtual viewpoints around the measurement object OJ, as shown in FIG. 9(a). The rectangular pyramid in the figure schematically represents the 3D laser scanner 5, with the orientation of the rectangular base representing the direction of orientation of the 3D laser scanner 5. Multiple virtual viewpoints with different positions and orientation directions are set, as indicated by the reference symbols 5a, 5b, 5c, etc. Furthermore, if the 3D laser scanner 5 has two switchable measurement modes with different measurement depths, a virtual viewpoint is set for each mode. Next, for each virtual viewpoint, the quality of data acquisition for the missing area OJb is evaluated. For example, it is evaluated whether the target defective area can be seen linearly from the virtual viewpoint, whether the distance to the defective area is appropriate for measurement, whether multiple defective areas can be measured at once, etc.

[0035] Then, as shown in FIG. 9(b), multiple relatively favorable virtual viewpoints having the same measurement mode are interpolated and connected into a single continuous line to generate a preferred movement trajectory TR11 for the 3D laser scanner 5, including the pointing direction. Here, the condition is that the pointing direction does not change suddenly. The symbol S indicates the start point of scanning, and the symbol E indicates the end point. If additional scanning in a different measurement mode is required as a result of the evaluation of the virtual viewpoints, a preferred movement trajectory is generated for each mode. The preferred movement trajectory TR11 thus generated is displayed on the display 8 as an instruction trajectory for the additional scanning, for example, as an image including the pointing direction, as shown in the figure.

[0036] The movement trajectory determination unit 7m estimates the movement trajectory of the 3D laser scanner 5 while the worker is scanning based on data acquired by the 3D laser scanner 5 and determines whether this movement trajectory deviates from the specified instruction trajectory. For example, as shown in FIG. 7, instruction trajectories TR1 and TR2 (trajectories along which the worker should operate the 3D laser scanner 5), including the direction of orientation of the 3D laser scanner 5, are set in advance for a specific measurement object OJ (shown simplified in the figure), such as an assembly in the engine compartment 1. These instructions are given to the worker, for example, by displaying them on the display 8 or by printed matter. The worker moves the 3D laser scanner 5 along these instruction trajectories TR1 and TR2 to perform scanning. Note that the instruction trajectory TR1 is the instruction trajectory for scanning when the 3D laser scanner 5 is set to the first measurement mode, and the instruction trajectory TR2 is the instruction trajectory for scanning when the 3D laser scanner 5 is set to the second measurement mode. If the worker's actual scanning operation does not accurately follow the instruction trajectories, the missing areas increase, which is undesirable. Therefore, the movement trajectory determination unit 7m determines whether the actual movement trajectory deviates from the instructed trajectory, and if so, notifies the worker of this fact and prompts him or her to correct the movement trajectory.

[0037] FIG. 8 is an explanatory diagram illustrating the principle of trajectory determination. Since the basic shape or configuration of the area to be scanned is known as design data obtained from the database 6, the spatial position and orientation of the 3D laser scanner 5 can be estimated based on point cloud data sequentially obtained from the 3D laser scanner 5. This allows the trajectory of the 3D laser scanner 5, including its orientation, to be estimated. The trajectory TR1' estimated in this manner is sequentially compared with the instruction trajectory TR1. For example, as shown in FIG. 8, the angle difference and distance difference between the vector pairs connecting the two nearest points from the coordinate sequences of points P1, P2, P3, etc. that make up the instruction trajectory TR1 and points P11, P12, P13, etc. that make up the trajectory TR1' are calculated, and these are compared to determine the degree of local agreement between the instruction trajectory TR1 and the trajectory TR1'. Note that other known appropriate algorithms can be used as the algorithm for determining the deviation of the trajectories.

[0038] When the actual movement trajectory deviates from the instructed trajectory, the worker can be notified by a display on the display 8 or by audio warning or instruction. In a preferred embodiment, however, the 3D laser scanner 5 has a built-in vibrator, and the worker is warned or notified by vibrating the 3D laser scanner 5 held by the worker. In this case, it is preferable that the vibrator emits vibrations of a strength corresponding to the degree of deviation as information to prompt correction. In other words, when the actual movement trajectory deviates from the instructed trajectory during scanning, the 3D laser scanner 5 begins to vibrate. The vibrations become stronger as the actual movement trajectory deviates from the instructed trajectory, and then weaken as the worker approaches the instructed trajectory. This makes it easy for the worker to scan along the instructed trajectory even if he or she does not have a precise grasp of the instructed trajectory in space.

[0039] Next, the operation of the gap inspection device of this embodiment will be described with reference to the flowchart of FIG.

[0040] First, in step S1, an operator operates the 3D laser scanner 5 to scan a predetermined area in the engine compartment 1 from the outside, including the target components, the ignition coil 3 and the fuel tube 4, to obtain point cloud data including a portion of the outer surface of each component. This scanning and generation of point cloud data progresses gradually as the scanning operation proceeds.

[0041] Next, in step S2, CAD data of the target components, ignition coil 3 and fuel tube 4, and other peripheral components are obtained from database 6, along with CAD data indicating their spatial positional relationships.

[0042] Then, in step S3, the CAD data is converted into point cloud data using a known appropriate conversion method.

[0043] Next, in step S4, the first alignment unit 7a performs rough alignment between the entire target area and the entire reference area. As described above, rough alignment is performed by aligning the point cloud data of the entire area including the ignition coil 3, the fuel tube 4, and other components with the point cloud data of the entire area including the ignition coil, the fuel tube, and other components in the design data stored in the database 6 by searching for key points and combining paired key points.

[0044] After the rough alignment in step S4, in step S5, the progress calculation unit 7b calculates the progress of scanning from the ratio between the number of points in the point cloud data of the reference area and the number of matching points in the point cloud data of the scanned area.

[0045] Then, in step S6, it is determined whether this progress rate exceeds a predetermined progress rate threshold (for example, 40%). If the progress rate is equal to or less than the threshold, the process proceeds to step S7, where the progress rate is displayed on the display 8 and the worker continues scanning. In other words, the process from step S1 onwards is repeated.

[0046] Here, in step S8 following step S7, the movement trajectory determination unit 7m described above estimates the actual movement trajectory of the 3D laser scanner 5 by the worker, and then in step S9, it is determined whether the estimated movement trajectory is along the correct instructed trajectory. If it is along the instructed trajectory, the process returns to step S1, and scanning is repeated. If the estimated movement trajectory deviates from the correct instructed trajectory, the process notifies the worker in step S10, prompting him to correct the trajectory, and then the process returns to step S1, and scanning continues. The worker is preferably notified by a vibrator built into the 3D laser scanner 5, with a strength corresponding to the degree of deviation, as described above.

[0047] Therefore, basically, the processes of steps S1 to S10 are repeated from the start of scanning until the progress reaches a predetermined progress threshold. If the progress threshold is not reached even after the entire instruction trajectory has been traversed, multiple scanning operations along the same instruction trajectory are required.

[0048] If the progress rate exceeds the threshold in step S6, the process proceeds from step S6 to step S11, where point cloud data of the target components, the ignition coil 3 and the fuel tube 4, is extracted from the point cloud data of the measurement data for the entire region and the point cloud data of the design data. The point cloud data extracted from the measurement data becomes the so-called target, and the point cloud data extracted from the design data becomes the so-called reference. The point cloud data of the target component that serves as the reference may be generated from CAD data of the component alone.

[0049] Next, in step S12, the second alignment unit 7c uses a known appropriate algorithm to perform detailed alignment of the ignition coil 3 and the fuel tube 4. Here, the fuel tube 4 is considered to be a rigid body. For example, as described above, paired points are searched for between the point cloud data of the target that has undergone rough alignment and the point cloud data of the reference, and detailed alignment is performed so that the reference approaches the target.

[0050] Next, in step S13, data on the fuel tube 4 is downsampled to perform non-rigid registration of the fuel tube 4, which is a non-rigid body. Then, in step S14, non-rigid registration is performed taking into account the deformation of the fuel tube 4. That is, the reference point cloud data is deformed while being registered to the target position. In this non-rigid registration, a nearest neighbor pair is searched for from the reference point cloud data and target point cloud data, which have been registered assuming a rigid body as described above, and rotation, enlargement, and translation are calculated as parameters for bringing them closer to each other.

[0051] Next, the process proceeds to step S15, where the reliability of the alignment is calculated for each of the ignition coil 3 and the fuel tube 4. The reliability is expressed, for example, as the ratio between the number of points in the reference point cloud data and the number of points in the target point cloud data that are included within a predetermined radius from each point in the reference point cloud data.

[0052] Then, in step S16, it is determined whether or not the reliability of the alignment of the ignition coil 3 and the reliability of the alignment of the fuel tube 4 each satisfy a predetermined reliability. If both reliability levels satisfy the predetermined reliability, it is determined that alignment is complete and the process proceeds to step S17.

[0053] In step S17, the first interpolation unit 7f and the second interpolation unit 7g interpolate unscanned portions of the target point cloud data of the ignition coil 3 and the fuel tube 4 with reference point cloud data aligned with the targets. As a result, point cloud data of the ignition coil 3 and the fuel tube 4 including unscanned portions (i.e., hidden portions) is generated at the positions of each target.

[0054] Then, in step S18, a known conversion method is used to convert the point cloud data including the unscanned portions of both the ignition coil 3 and the fuel tube 4 into mesh data that constitutes a surface.

[0055] Next, in step S19, distance calculation unit 7h calculates the minimum distance between ignition coil 3 and fuel tube 4 based on the mesh data of ignition coil 3 and the mesh data of fuel tube 4. That is, the distance between any two points on each surface is found, and the minimum value among these is set as the minimum distance.

[0056] Then, in step S20, the inspection results including this minimum distance and other necessary information are displayed on the display 8. The calculated minimum distance may be compared with a threshold value, and if it is less than the threshold value, some kind of warning may be displayed.

[0057] On the other hand, if it is determined in step S16 that the alignment reliability is insufficient for any of the components, the process proceeds from step S16 to step S21, where a determination is made as to whether the component has a missing area for which the reliability is insufficient. As described above, the surface to be scanned in the design data of each component is divided into multiple grids, and the proportion of scanned points for each grid (the ratio of the number of points actually scanned to the number of points to be scanned) is calculated. If this proportion is equal to or less than a predetermined threshold, the grid is determined to be a missing area.

[0058] If the result in step S21 is NO, this means that the lack of reliability is not due to a missing area, and for example, the target part has not been correctly recognized, and the process ends with a warning or the like being displayed on the display 8. In this case, for example, the process may be restarted from the initial scanning.

[0059] If it is determined in step S21 that there is a missing area (in other words, that the missing area is due to insufficient reliability), the process proceeds from step S21 to step S22, where multiple virtual viewpoints are set around the target area (engine compartment 1) to acquire data on the missing area, and the advantages of each are evaluated. Then, in step S23, measurement parameters required for additional scanning (such as the movement trajectory, pointing direction, and measurement mode of the 3D laser scanner 5) are calculated, and in step S24, the operator is instructed to perform additional scanning using these measurement parameters. This is done, for example, by displaying on the display 8, providing audio instructions, etc. The necessary movement trajectory calculated as a parameter is preferably displayed on the screen as an instruction trajectory for the additional scanning. Then, the process returns from step S24 to step S1, and rescanning is performed. Note that in the illustrated example, the movement trajectory is also determined and corrected in steps S8 to S10 during additional scanning. The additional scanning is continued until the reliability of the registration satisfies a predetermined reliability (step S16) or until it is determined that there is no missing area (for example, a certain percentage or less) (step S17).

[0060] In this way, in the above embodiment, a missing area is determined for a portion from which data would be obtained if scanned properly, the measurement parameters necessary to obtain data for the missing area are calculated, and the operator is prompted to perform additional scanning in accordance with these measurement parameters, thereby enabling efficient acquisition of 3D measurement data. Furthermore, various degradations in inspection accuracy due to the presence of missing areas during scanning can be suppressed. In particular, because missing areas are determined only for parts with low alignment reliability and additional scanning is performed to fill in the missing areas of those parts, the additional scanning is efficient.

[0061] Furthermore, since the determination of the missing area is performed after the reference point cloud data is aligned with the target point cloud data, it is possible to more accurately determine which part of the target part is the missing area.

[0062] In the above embodiment, it is determined whether the movement trajectory of the 3D laser scanner 5 operated by the worker is along a predetermined instructed trajectory, and the worker is prompted to correct it so that it follows the instructed trajectory. This makes scanning easier and more appropriate, and reduces the work time.

[0063] While the above describes one embodiment in which the present invention is applied to inspecting the gap between an automobile ignition coil 3 and a fuel tube 4, the present invention is not limited to such an application and can be widely applied to inspections related to the spatial arrangement of target parts, such as their position and orientation in space, in an assembly including multiple parts. Note that in the above embodiment, two parts are used as target parts in order to inspect the distance between the two parts, but the present invention can also be applied to a single target part.

[0064] In the above embodiment, an example has been described in which the 3D laser scanner 5 is handheld and operated by an operator, but the above-described processing of missing areas and estimation of movement trajectories can also be applied when scanning is performed by a robot using a 3D sensor. In this case, additional scanning is possible by controlling the robot arm according to measurement parameters calculated to compensate for the missing area.

[0065] The three-dimensional sensor is not limited to the three-dimensional laser scanner 5 of the above embodiment, and any type of sensor may be used as long as it can acquire and generate three-dimensional point cloud data. A wide range of sensors can be used, including ToF sensors and triangulation sensors such as stereo cameras.

Claims

1. 1. A part inspection method for inspecting a spatial arrangement state of one or more target parts in an assembly formed by assembling a plurality of parts, comprising: Scanning an area including the target part from the outside with a three-dimensional sensor to obtain measurement data including a portion of the outer surface of the target part; Acquire design data including the individual external shapes of the target parts and the positional relationships of the parts within the assembly; determining a missing area in the measurement data of the target part by comparing with design data of the target part; determining measurement parameters of a three-dimensional sensor suitable for acquiring measurement data of the defect area; Perform additional scanning along these measurement parameters, Part inspection methods.

2. 2. The part inspection method according to claim 1, wherein the defective area is determined after aligning the point cloud data of the target part based on the design data with the point cloud data of the target part extracted from the measurement data.

3. performing alignment for each of the plurality of target parts and calculating the reliability of the alignment for each target part based on the number of corresponding points in the point cloud; Determining missing areas for target parts with low alignment reliability; The component inspection method according to claim 2 .

4. Evaluating the quality of data acquisition of the missing area when data is acquired from each of the plurality of virtual viewpoints using a three-dimensional sensor; A trajectory connecting a plurality of advantageous virtual viewpoints and an angle of the 3D sensor required at each virtual viewpoint are determined as measurement parameters. The component inspection method according to claim 1 .

5. Estimating a movement trajectory of the three-dimensional sensor during scanning based on data acquired by the three-dimensional sensor; Determine whether this movement trajectory deviates from the specified trajectory; If there is a discrepancy, information prompting correction is output. The component inspection method according to claim 1 .

6. The system repeatedly performs rough alignment by using key points to align the point cloud data of the area including the target part acquired by scanning with the point cloud data of the area based on the design data, while estimating the movement trajectory of the 3D sensor, determining deviations, and outputting information to prompt corrections.

6. The component inspection method according to claim 5.

7. The three-dimensional sensor operated by the worker is equipped with a vibrator, As information to prompt correction, a vibrator emits vibrations of a strength according to the degree of deviation.

6. The component inspection method according to claim 5.

8. The progress of scanning is calculated based on the matching state of the two roughly aligned point cloud data. When this progress reaches a predetermined level, the target part is aligned. The component inspection method according to claim 2 .

9. A part inspection device that performs inspection related to the spatial arrangement state of one or more target parts in an assembly formed by assembling a plurality of parts, a three-dimensional sensor that scans an area including the target part from the outside; a measurement data acquisition unit that acquires measurement data including a portion of the outer surface of the target part by scanning using the three-dimensional sensor; a design data acquisition unit that acquires design data including the external shapes of individual target parts and the positional relationships of the parts within an assembly from a database; a loss area determination unit that determines a loss area in the measurement data of the target part by comparing with design data of the target part; a measurement parameter calculation unit for calculating measurement parameters of a three-dimensional sensor suitable for acquiring measurement data of the missing region; an information output unit that outputs the measurement parameters to an operator for further scanning; A component inspection device comprising:

Images