Component inspection method and device

Abstract

In order to check the minimum distance between an ignition coil (3) and a fuel pipe (4), point cloud data (S1) is generated by scanning with a three-dimensional laser scanner (5), and point cloud data (S2, S3) of a reference based on CAD data is registered (S4-S14). After registration, the measured point cloud data is filled with reference point cloud data (S17), and the minimum distance between any two points is calculated (S19). When the reliability is low (S15, S16), a missing region is determined (S17), and additional scanning is performed. At this time, a large number of virtual viewpoints set in the surroundings are evaluated, measurement parameters such as trajectories are calculated so as to connect favorable points (S23), and on the basis thereof, additional scanning is instructed to be performed in accordance with the measurement parameters (S24).

Description

Technical Field

[0001] The present invention relates to a component inspection method and apparatus for inspecting the spatial arrangement state of one or more components as a target in an assembly formed by assembling a plurality of components. Background Art

[0002] For example, in the whole vehicle inspection process of an automobile, an inspector sometimes visually inspects the positions of a large number of components assembled in an engine compartment (e.g., inspects the gaps between adjacent components) using a ruler. In recent years, attempts have been made to obtain three-dimensional data of the external shape of each component by measuring with a three-dimensional sensor, thereby inspecting the installation state or spatial position of a specific component in such an assembly (e.g., Patent Document 1).

[0003] In an assembly composed of multiple parts assembled three-dimensionally, when measuring from the outside using a three-dimensional sensor, an area where data cannot be obtained may occur in a part of the target part due to reasons such as inappropriate orientation of the three-dimensional sensor or shielding by other parts. This missing area causes a decrease in the accuracy of determining the shape and position of the part.

[0004] Patent Document 1 discloses that the three-dimensional position and orientation of a target object is estimated and the object is recognized by using the missing region as a feature, but does not disclose anything about reducing the missing region during measurement.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2011-174891 Summary of the invention

[0008] According to the present invention, in a component inspection method for inspecting a state of spatial arrangement of one or more components as a target in an assembly formed by assembling a plurality of components,

[0009] Scanning an area including the object component from the outside using a three-dimensional sensor to obtain measurement data of a portion of the outer surface of the object component,

[0010] Acquire design data including the external shape of each target component and the positional relationship of the target component in the assembly,

[0011] Compare the measured data of the target part with the design data of the target part to determine the missing area in the measured data of the target part.

[0012] determining measurement parameters of a three-dimensional sensor suitable for acquiring measurement data of the missing area,

[0013] An additional scan is performed according to the measurement parameters.

[0014] According to the present invention, measurement parameters of the three-dimensional sensor suitable for acquiring measurement data of the missing area are determined, and additional scanning is performed according to the measurement parameters, thereby making the missing area smaller and improving the measurement accuracy that is a prerequisite for inspecting the target component. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is an explanatory diagram showing various components in an engine compartment as an assembly in one embodiment as viewed from above.

[0016] Figure 2 is a functional block diagram of a gap checking device according to an embodiment.

[0017] Figure 3 It is the point cloud data of the engine compartment and its internal components obtained by scanning with a 3D laser scanner.

[0018] Figure 4 The point cloud data of the engine compartment and its internal components is 40% of the progress.

[0019] Figure 5 This is a diagram to illustrate reliability.

[0020] Figure 6 Detailed description of the invention is a flowchart showing the flow of processing of a gap checking method according to one embodiment.

[0021] Figure 7 This is a diagram illustrating the movement trajectory of the three-dimensional sensor during scanning.

[0022] Figure 8 It is an explanatory diagram showing the deviation between the instruction trajectory and the actual movement trajectory during scanning.

[0023] Fig. 9 This is a diagram illustrating additional scanning of the missing area. DETAILED DESCRIPTION

[0024] One embodiment of the present invention is described below with reference to the accompanying drawings. One embodiment is obtained by applying the present invention to a clearance inspection for determining the spatial positions of two components located in an engine compartment and calculating the minimum clearance between the two components in a vehicle inspection process.

[0025] Figure 1This is a diagram of a portion of various components in an engine compartment 1 of an automobile as an assembly as viewed from above, showing an area near an intake manifold 2 of the engine. An ignition coil 3 as a part of the ignition device of the engine is arranged on the lower side of the intake manifold 2. In addition, a fuel pipe 4 as a pipe for supplying fuel to the engine is located on the side of the intake manifold 2. The fuel pipe 4 is formed of a metal pipe in consideration of oil resistance and fire resistance, and has a circular cross section along the radial direction. Figure 1 As shown in FIG. 1 , the fuel pipe 4 has a straight portion 4a extending substantially straight along the side of the intake manifold 2, and an inclined portion 4b inclined from one end of the straight portion 4a toward the ignition coil 3. The fuel pipe 4 made of a metal pipe will deform slightly during installation, and is therefore considered a non-rigid component. On the other hand, the ignition coil 3 is considered a rigid component whose outer shape does not change.

[0026] The minimum distance (minimum gap) in space between the ignition coil 3 and the fuel pipe 4 is generally regulated by laws and regulations, and therefore, the minimum gap needs to be checked during the vehicle inspection process. Here, in most cases, the minimum distance in design between the ignition coil 3 and the fuel pipe 4 is specified at a position on the outer surface of at least one of the ignition coil 3 and the fuel pipe 4 that cannot be visually confirmed from the outside, and it is generally difficult for an inspector to measure it using a ruler. This embodiment obtains the gap between the two components by measuring and calculating from the outside using a three-dimensional sensor.

[0027] like Figure 2 As shown, a gap inspection device of one embodiment used in the whole vehicle inspection of an automobile is composed of a three-dimensional sensor such as a three-dimensional laser scanner 5, a database 6, a control device 7, and one or more displays 8. In addition, a large number of inspections are performed in a prescribed order in the whole vehicle inspection process. The gap inspection device of one embodiment is constituted as a part of the inspection device in the whole vehicle inspection process.

[0028] The three-dimensional laser scanner 5 can obtain the three-dimensional coordinates of the surface shape of the measurement object by irradiating the measurement object with laser light and measuring the reflection time, and obtain high-density point cloud data by measuring at a high speed of about tens of thousands of points per second. As a three-dimensional laser scanner, various sizes and forms of three-dimensional laser scanners are known, but in one embodiment, a three-dimensional laser scanner in a form that an operator can hand-hold to scan the measurement object is used. A specific area of ​​the engine compartment 1 including the ignition coil 3, the fuel pipe 4 and other components is manually scanned from the outside along the indicated movement trajectory to measure the object area. By scanning with such a three-dimensional laser scanner 5, as Figure 3 As shown, point cloud data of the entire region including a portion of the outer surface of the ignition coil 3 and the fuel pipe 4 is obtained.

[0029] In addition, the point cloud data is actually acquired as time series data for each frame by scanning, and the point cloud data of the entire region can be acquired by superposition. In addition, it is desirable to perform the following noise removal processing on the acquired point cloud data: for example, noise caused by dust entering during scanning is removed. In addition, as a three-dimensional laser scanner, a reference mark for identifying the shape and position of the target component during scanning may also be required.

[0030] Here, the "scanning" in the present invention is a concept that includes either or both of the plane scanning function of the three-dimensional sensor (e.g., the three-dimensional laser scanner 5) itself and the movement of the three-dimensional sensor, for example, by an operator along a certain trajectory. In one embodiment, scanning is performed by using a three-dimensional laser scanner 5 having a plane scanning function and moving the three-dimensional laser scanner 5 along a prescribed trajectory. Depending on the size and shape of the object area to be measured, or the form of the three-dimensional sensor, for example, multiple three-dimensional sensors can be fixedly arranged at appropriate positions in the space, and the scanning function of each three-dimensional sensor can be used to obtain point cloud data of the object area.

[0031] The database 6 accumulates all the design data of the automobile to be inspected, and therefore, the shape data of the ignition coil 3 and the shape data of the fuel pipe 4, which are the objects of the gap inspection, and the data indicating the positional relationship between the two in space are stored as design data. In addition, the database 6 also accumulates the shape data and positional relationship data of other components existing in the area. The design data is stored in the database 6 in the form of CAD data constituting a mesh, and the required design data is read from the database 6 to the control device 7 during the gap inspection. In addition, for the design data, after reading the control device 7, the well-known hidden surface processing can be performed on the parts other than the part related to the gap between the ignition coil 3 and the fuel pipe 4 to reduce the data size.

[0032] The control device 7 is configured to include a first registration unit 7a, a progress calculation unit 7b, a second registration unit 7c, a third registration unit 7d, a reliability calculation unit 7e, a first filling unit 7f, a second filling unit 7g, a distance calculation unit 7h, an information output unit 7i, a missing area determination unit 7j, a measurement parameter calculation unit 7k, and a movement trajectory determination unit 7m. In addition, although not shown in the figure, as described above, the control device 7 includes: a measurement data acquisition unit that acquires measurement data including a part of the outer surface of the target component by scanning using the three-dimensional laser scanner 5; and a design data acquisition unit that acquires design data including the outer shape of each target component and the positional relationship of the target component in the engine compartment 1 from the database 6.

[0033] The first registration unit 7a performs a rough registration of the entire object area (the area serving as the so-called target in the registration) including the ignition coil 3 and the fuel pipe 4 in the measurement data acquired by the three-dimensional laser scanner 5 with the entire area (the area serving as the so-called reference in the registration) including the ignition coil 3 and the fuel pipe 4 in the design data accumulated in the database 6. For example, the first registration unit 7a uses the FPFH algorithm to search for key points from the point cloud data of the entire area serving as the target, and describes the characteristics of the key points, such as the normal vector of the key points and the relative angles around them. In addition, on the basis of transforming the design data of the entire area serving as the reference into point cloud data using a well-known transformation method, the FPFH algorithm is similarly used to search for key points, and the characteristics of the key points, such as the normal vector of the key points and the relative angles around them are described. Then, the first registration unit 7a compares the normal vector between the key point related to the target and the key point related to the reference, and the relative angle with the surrounding point cloud, to search for a combination of key points constituting a pair whose normal vectors and relative angles are roughly consistent. Then, the reference point cloud data is roughly registered with the target point cloud data using the combination of paired key points. In addition, in this rough registration, the fuel pipe 4, which is a non-rigid body, is regarded as a rigid body, and as a result, an average registration is performed within the length range of the fuel pipe 4. The rough registration of the object area by the first registration unit 7a is performed in parallel with the progress of the scanning of the object area. The first registration unit 7a may also use a known algorithm other than the FPFH algorithm to perform the rough registration of the object area.

[0034] After the rough alignment performed by the first alignment unit 7a and before the detailed alignment of the ignition coil 3 and the fuel pipe 4 described later, the progress calculation unit 7b calculates the progress indicating the extent to which the required scanning in the object area has progressed. For example, the progress calculation unit 7b calculates the progress of the scanning based on the ratio of the number of points in the point cloud data of the object area based on the design data used as the reference to the number of points in the point cloud data that matches the reference and the target through the rough alignment performed by the first alignment unit 7a. The progress calculation unit 7b further compares the progress calculated in this way with a prescribed progress threshold (for example, 40% in this embodiment) and determines in real time whether the progress that gradually increases with the progress of the scanning exceeds the progress threshold. For example, the progress at this time and the information on whether the progress exceeds the progress threshold are displayed on the display 8 via the information output unit 7i, so the operator continues to scan using the three-dimensional laser scanner 5 according to the display. In other words, scanning (generation of point cloud data), rough alignment, and progress calculation using the three-dimensional laser scanner 5 are repeated in real time until the prescribed progress threshold is exceeded. For example, at Figure 4 The area enclosed by the dotted line corresponds to 40% of the progress.

[0035] In addition, the progress may be evaluated based on the number of viewpoints that the three-dimensional laser scanner 5 has passed, the number of key points used in rough registration, or the like, in addition to the evaluation based on the ratio of the number of data points as described above.

[0036] The second registration unit 7c performs detailed spatial registration of each component on the condition that the progress exceeds the threshold. That is, the second registration unit 7c searches for paired points from the reference point cloud data for all points of the target point cloud data obtained by measuring the ignition coil 3 and the fuel pipe 4, and performs detailed registration of the reference point cloud data with the target point cloud data. The fuel pipe 4, which is a non-rigid body, is regarded as a rigid body, and as a result, an average registration is performed within the length range of the fuel pipe 4. The detailed registration of the second registration unit 7c can be performed using a known appropriate algorithm.

[0037] After the detailed registration performed by the second registration unit 7c, the third registration unit 7d performs so-called non-rigid body registration for the non-rigid body component (the fuel pipe 4 in this embodiment) in consideration of deformation. Here, a known appropriate algorithm can be used, but for example, the nearest neighbor pair is searched from the reference point cloud data and the target point cloud data that have been pre-registered as a rigid body, and the parameters of rotation, enlargement, and translation are obtained as parameters that make the two points of the pair approach each other. For example, the outer shape of the fuel pipe 4 is downsampled to form a shape in which the outer surface is composed of a plurality of triangles having vertices and edges, and the vertices of the point cloud (cluster) representing each area obtained by downsampling, that is, each area including a plurality of adjacent triangles are used, and the constraint that the length of the edge does not change (strictly speaking, the length change is minimized) is added, and the deformation of the cluster is decomposed into parameters of rotation, enlargement, and translation. The overall deformation of the fuel pipe 4 is obtained as a collection of such deformations in cluster units. The third registration unit 7 d deforms the reference point cloud data on the fuel pipe 4 by using such a non-rigid registration method taking deformation into consideration, and registers the data with the target point cloud data in space.

[0038] The reliability calculation unit 7e calculates the reliability of the registration for each target component (i.e., the ignition coil 3 and the fuel pipe 4). In other words, the reliability calculation unit 7e calculates the reliability of the point cloud data indicating how many targets are included in the vicinity of the reference point cloud data after registration for each of the ignition coil 3 and the fuel pipe 4.

[0039] Figure 5This is an explanatory diagram schematically showing the point cloud data of the fuel pipe 4 after registration in order to explain the reliability. For the sake of simplicity, it is assumed that the circular outer surface at a certain cross section of the fuel pipe 4 is formed by 13 point cloud data. The 13 point cloud data Dr arranged in a circle are reference point cloud data based on design data, and the 7 point cloud data Dt arranged in a semicircle are target point cloud data based on measurement data. For example, the 3D laser scanner 5 is scanned only from the upper side of the figure, and as a result, the lower half becomes a so-called missing area, and the target point cloud data Dt is arranged in a semicircle.

[0040] Reliability is represented, for example, by the ratio of the number of points of the reference point cloud data Dr to the number of points of the target point cloud data Dt included within a radius L from each point of the reference point cloud data Dr. In addition, in the figure, the range of radius L from a large number of points of the reference point cloud data Dr is represented by an outer circle C1 and an inner circle C2 represented by dotted lines. Figure 5 In the example (a), the four point cloud data Dt of the target are included in the range R of the circles C1 and C2, so the reliability is 4 / 13. Figure 5 In the example shown in (b), there are two target point cloud data Dt included in the range R of circles C1 and C2, so the reliability is 2 / 13. Figure 5 In the example shown in (c), all the target points, that is, 7 point cloud data Dt are included in the range R, and the reliability is 7 / 13. As described above, the reliability is affected by both the accuracy of registration and the size or ratio of the missing area in the measurement data.

[0041] The first filling unit 7f fills in the missing part that is not scanned in the measurement data of the ignition coil 3, that is, the target point cloud data, with the reference point cloud data. In other words, the outer surface part of the back side (lower side) of the ignition coil 3 that is hidden and cannot be observed from the upper side of the engine compartment 1 is filled in with the reference point cloud data to generate point cloud data including the hidden part of the ignition coil 3. Then, the first filling unit 7f transforms the generated point cloud data into mesh data constituting a surface using a well-known transformation method.

[0042] Similarly, the second filling unit 7g fills in the missing portion that is not scanned in the measurement data of the fuel pipe 4, that is, the target point cloud data, using the reference point cloud data. In other words, the outer surface portion of the back side (lower side) of the fuel pipe 4 that is hidden and cannot be observed from the upper side of the engine compartment 1 is filled in with the reference point cloud data to generate point cloud data including the hidden portion of the fuel pipe 4. Then, the second filling unit 7g transforms the generated point cloud data into mesh data constituting a surface using a well-known transformation method.

[0043] The distance calculation unit 7h calculates the distance from each point on the surface of the ignition coil 3 to each point on the surface of the fuel pipe 4 based on the mesh data of the ignition coil 3 obtained by the first filling unit 7f and the mesh data of the fuel pipe 4 obtained by the second filling unit 7g. And the distance calculation unit 7h finds the minimum distance by comparing the calculated distances. In addition, instead of converting the point cloud data of the ignition coil 3 and the point cloud data of the fuel pipe 4 into mesh data, the distance calculation unit 7h calculates the distance from each point of the ignition coil 3 to each point of the fuel pipe 4 based on each point cloud data.

[0044] The information output unit 7i generates various images to be displayed on the display 8, data to be output as sound from a speaker (not shown), etc. For example, the final minimum distance, the above-mentioned progress, etc. are displayed on one or more displays 8 so that the operator can visually confirm.

[0045] The missing area determination unit 7j determines the missing part in the measurement data, that is, the missing area, for each target component (ignition coil 3 and fuel pipe 4). For example, the surface to be scanned in the design data of each component is divided into a large number of grids, and the ratio of the scanned point cloud (the ratio of the number of points actually scanned to the number of points to be scanned) is calculated for each grid. If the ratio is below a predetermined threshold, the grid is determined to be a missing area. In this way, which part of the design data of the target component is the missing area is calculated.

[0046] In addition, a surface portion that is not originally a scanning target (e.g., a portion that is surrounded by other components and cannot be visually confirmed by a straight line, etc.) is not a target for determination of a missing area. In other words, a portion that can acquire measurement data if the scanning operation is performed properly but currently lacks measurement data becomes a missing area, even if the portion is a portion where measurement data acquisition is scheduled.

[0047] The measurement parameter calculation unit 7k calculates the measurement parameters required for obtaining the measurement data of the missing areas determined by the missing area determination unit 7j using the three-dimensional laser scanner 5. The measurement parameters refer to the position of the three-dimensional laser scanner 5 and its pointing direction (in other words, angle), and in the case where the measurement mode such as the intensity of the laser can be changed, the measurement parameters refer to the measurement mode, etc. In one embodiment, the operator scans while holding the three-dimensional laser scanner 5 and moving it, so the position of the three-dimensional laser scanner 5 and its pointing direction required for obtaining the measurement data of the missing areas are generated as a continuous movement trajectory.

[0048] The measurement parameters generated in this way (such as the required movement trajectory, measurement mode, etc.) are displayed on the display 8 via the information output unit 7i. The position and range of the missing area may also be displayed. The operator performs additional scanning according to the display of the display 8.

[0049] Fig. 9 7k is an explanatory diagram for explaining the principle of measurement parameter calculation by the measurement parameter calculation unit 7k. Fig. 9 In the embodiment, for example, the entire area to be measured in the engine compartment 1 is represented as the measurement object OJ. The measurement object OJ includes an area OJa for which measurement data has been acquired, and a missing area OJb for which measurement data is currently missing. The measurement parameter calculation unit 7k first calculates the measurement parameters of the measurement object OJ. Fig. 9 As shown in (a), a large number of virtual viewpoints are set around the measured object OJ. The quadrangular pyramid in the figure schematically represents the three-dimensional laser scanner 5, and the direction of the bottom surface of the quadrilateral is the pointing direction of the three-dimensional laser scanner 5. As shown as the reference numerals 5a, 5b, 5c..., a large number of virtual viewpoints with different positions and pointing directions are set. In addition, in the case where the three-dimensional laser scanner 5 has two switchable measurement modes with different measurement depths, virtual viewpoints are set according to each mode. Next, for each virtual viewpoint, it is evaluated whether the data of the missing area OJb is well obtained. For example, it is evaluated whether the missing area as the object can be confirmed by straight-line vision from the virtual viewpoint, whether the distance to the missing area is appropriate for measurement, whether a large number of missing areas can be measured intensively, etc.

[0050] Then, if Fig. 9 As shown in (b), multiple relatively advantageous virtual viewpoints having the same measurement mode are filled in to form a continuous line, and a preferred movement trajectory TR11 of the three-dimensional laser scanner 5 including the pointing direction is generated. Here, it is conditioned that the pointing direction does not change sharply. Reference symbol S represents the starting point of the scan, and reference symbol E represents the ending point. In addition, if the result of the evaluation of the virtual viewpoint is that additional scanning under different measurement modes is required, a preferred movement trajectory is generated for each mode. The preferred movement trajectory TR11 generated in this way is displayed as an image including the pointing direction on the display 8 as an indication trajectory for additional scanning, for example, as shown in the figure.

[0051] The movement trajectory determination unit 7m estimates the movement trajectory of the three-dimensional laser scanner 5 during scanning by the operator based on the data acquired by the three-dimensional laser scanner 5, and determines whether the movement trajectory deviates from the indicated trajectory. Figure 7As shown, for a specific measurement object OJ (simplified in the figure) such as an assembly in the engine compartment 1, an indication trajectory (a trajectory along which the operator should operate the three-dimensional laser scanner 5) TR1, TR2 including a pointing direction of the three-dimensional laser scanner 5 is pre-set, and the indication is provided to the operator, for example, through a display on the display 8 or a printed material. The operator moves the three-dimensional laser scanner 5 along the indication trajectory TR1, TR2 to perform scanning. In addition, the indication trajectory TR1 is an indication trajectory for scanning performed by setting the measurement mode of the three-dimensional laser scanner 5 to the first mode, and the indication trajectory TR2 is an indication trajectory for scanning performed by setting the measurement mode of the three-dimensional laser scanner 5 to the second mode. When the actual scanning operation performed by the operator does not correctly follow the indication trajectory, the missing area increases, which is not preferred. Therefore, the movement trajectory determination unit 7m determines whether the actual movement trajectory deviates from the indication trajectory, and in the case of deviation, notifies the operator of the meaning to urge the movement trajectory to be corrected.

[0052] Figure 8 This is an explanatory diagram for explaining the principle of moving trajectory determination. The basic shape or structure of the area to be scanned is known as design data obtained from the database 6. Therefore, the position of the three-dimensional laser scanner 5 in space and its pointing direction can be estimated based on the point cloud data successively obtained from the three-dimensional laser scanner 5, and further, its moving trajectory can be estimated in a manner including the pointing direction of the three-dimensional laser scanner 5. The moving trajectory TR1' estimated in this way is successively compared with the indicated trajectory TR1. For example Figure 8 As shown, based on the coordinate sequence of points P1, P2, P3, ... constituting the indication trajectory TR1 and points P11, P12, P13, ... constituting the moving trajectory TR1', the angle difference and distance difference of the vector pair connecting the two closest points are calculated and compared, thereby determining the degree of consistency between the local indication trajectory TR1 and the moving trajectory TR1'. In addition, as an algorithm for determining the deviation of the moving trajectory, other known appropriate algorithms can be used.

[0053] When the actual moving trajectory deviates from the indicated trajectory, the notification to the operator can be a warning or indication through a display or sound on the display 8, but in a preferred embodiment, the three-dimensional laser scanner 5 is built with a vibrator, and the operator is warned or notified by vibrating the three-dimensional laser scanner 5 held by the operator. At this time, it is preferred that the vibrator emits a vibration of an intensity corresponding to the degree of deviation as information to prompt correction. That is, when the actual moving trajectory is about to deviate from the indicated trajectory during the scanning operation, the three-dimensional laser scanner 5 begins to vibrate, and the farther away from the indicated trajectory, the stronger the vibration. Thereafter, if it approaches the indicated trajectory, the vibration becomes weaker. Therefore, even if the operator does not strictly grasp the indicated trajectory in the space, it is easy to perform scanning operations along the indicated trajectory.

[0054] Next, based on Figure 6 The operation of the gap inspection device of this embodiment is described with reference to the flowchart of FIG.

[0055] First, as step S1, the operator operates the three-dimensional laser scanner 5 to scan a predetermined area including the ignition coil 3 and the fuel pipe 4 as target components in the engine compartment 1 from the outside to obtain point cloud data including a portion of the outer surface of each component. The scanning and generation of point cloud data gradually progresses along with the scanning operation.

[0056] Next, in step S2 , CAD data of the ignition coil 3 , the fuel pipe 4 , and other peripheral components as target components, and CAD data indicating the spatial positional relationship between them are acquired from the database 6 .

[0057] Then, in step S3, the CAD data is transformed into point cloud data using a known appropriate transformation method.

[0058] Next, in step S4, the first registration unit 7a performs a rough registration of the entire target region with the entire reference region. As described above, the point cloud data of the entire region including the ignition coil 3, the fuel pipe 4 and other components is registered with the point cloud data of the entire region including the ignition coil, the fuel pipe and other components in the design data accumulated in the database 6 by searching for key points and combining paired key points, thereby performing a rough registration.

[0059] After the rough registration in step S4, in step S5, the progress calculation unit 7b as described above calculates the progress of the scan based on the ratio between the number of points in the point cloud data of the reference area and the number of matching points in the point cloud data of the scanned area.

[0060] Then, in step S6, it is determined whether the progress exceeds a predetermined progress threshold (for example, 40%). If the progress is below the threshold, the process proceeds to step S7, where the progress is displayed on the display 8 and the operator continues scanning. That is, the process after step S1 is repeated.

[0061] Here, in step S8 following step S7, as the aforementioned movement trajectory determination unit 7m, the actual movement trajectory of the three-dimensional laser scanner 5 moved by the operator is estimated, and in step S9, it is determined whether the estimated movement trajectory is along the correct indicated trajectory. If it is along the indicated trajectory, the process returns to step S1 and the scanning is repeated as usual. If the estimated movement trajectory deviates from the correct indicated trajectory, the operator is notified of this fact in step S10 to prompt the trajectory to be corrected, and on this basis, the process returns to step S1 and the scanning is continued. Regarding the notification to the operator, it is preferred that, as described above, a vibration of an intensity corresponding to the degree of deviation is used by a vibrator built into the three-dimensional laser scanner 5.

[0062] Therefore, basically, the processing of steps S1 to S10 is repeated from the start of scanning until the progress reaches a predetermined progress threshold. If the progress threshold is not reached even after the entire instruction trajectory is passed, it is necessary to perform multiple scans along the same instruction trajectory.

[0063] If the progress exceeds the threshold in step S6, the process moves from step S6 to step S11, and the point cloud data of the ignition coil 3 and the fuel pipe 4 as the target parts are extracted from the point cloud data of the measurement data and the point cloud data of the design data about the entire region. The point cloud data extracted from the measurement data becomes the so-called target, and the point cloud data extracted from the design data becomes the so-called reference. The point cloud data of the target part as a reference can also be generated based on the CAD data of the component itself.

[0064] Next, in step S12, as the second registration unit 7c, a known appropriate algorithm is used to perform detailed registration of the ignition coil 3 and the fuel pipe 4. Here, the fuel pipe 4 is regarded as a rigid body. For example, as described above, a pair of points is searched between the point cloud data of the target that has undergone rough registration and the point cloud data of the reference, and detailed registration is performed in a manner that the reference and the target are close.

[0065] Next, in step S13, the data of the fuel pipe 4 is downsampled to perform non-rigid registration of the fuel pipe 4 as a non-rigid body. Then, in step S14, non-rigid registration of the fuel pipe 4 is performed in consideration of deformation. That is, the reference point cloud data is deformed while being registered to the position of the target. In this non-rigid registration, the nearest neighbor pair is searched from the reference point cloud data and the target point cloud data registered as a rigid body as described above, and rotation, magnification, and translation as mutually close parameters are obtained.

[0066] Next, the process proceeds to step S15 to calculate the reliability of the registration for the ignition coil 3 and the fuel pipe 4. The reliability is expressed, for example, by the ratio of the number of points in the reference point cloud data to the number of points in the target point cloud data within a predetermined radius from each point in the reference point cloud data.

[0067] Then, in step S16, it is determined whether the reliability of the registration of the ignition coil 3 and the reliability of the registration of the fuel pipe 4 satisfy a predetermined reliability. If both reliability satisfy the predetermined reliability, the registration is considered complete and the process proceeds to step S17.

[0068] In step S17, the first filling unit 7f and the second filling unit 7g fill in the unscanned parts of the point cloud data of the ignition coil 3 and the fuel pipe 4 with the reference point cloud data registered with the target. Thus, the point cloud data of the ignition coil 3 and the fuel pipe 4 including the unscanned parts (that is, the hidden parts) at the positions of the respective targets are generated.

[0069] Then, in step S18 , the point cloud data including the unscanned portions of both the ignition coil 3 and the fuel pipe 4 are converted into mesh data constituting a surface using a known conversion method.

[0070] Next, in step S19, the distance calculator 7h calculates the minimum distance between the ignition coil 3 and the fuel pipe 4 based on the mesh data of the ignition coil 3 and the mesh data of the fuel pipe 4. That is, the distance between any two points on each surface is obtained, and the minimum value is set as the minimum distance.

[0071] Then, in step S20, the inspection result including the minimum distance and other necessary information are displayed on the display 8. The calculated minimum distance may also be compared with a threshold value, and some warning may be displayed if the minimum distance is less than the threshold value.

[0072] On the other hand, if it is determined in step S16 that the reliability of registration is insufficient for any component, the process proceeds from step S16 to step S21, and a missing area is determined for the component with insufficient reliability. As described above, the surface to be scanned in the design data of each component is divided into a large number of grids, and the ratio of the scanned point cloud (the ratio of the number of points actually scanned to the number of points to be scanned) is calculated for each grid. If the ratio is below a predetermined threshold, the grid is determined to be a missing area.

[0073] In the case of "No" in step S21, it means that the lack of reliability is not caused by the missing area, for example, the target component is not correctly recognized, so some warning is displayed on the display 8 and the processing is terminated. In this case, for example, the scan is restarted from the first scan.

[0074] If it is determined in step S21 that there is a missing area (in other words, insufficient reliability is caused by the missing area), proceed from step S21 to step S22, set multiple virtual viewpoints around the target area (engine compartment 1) in order to obtain data of the missing area and evaluate the advantages of each virtual viewpoint. Then, in step S23, calculate the measurement parameters required for the additional scan (moving trajectory, pointing direction, measurement mode, etc. of the three-dimensional laser scanner 5), and in step S24, instruct the operator to use these measurement parameters for the additional scan. The instruction is performed, for example, by display on the display 8, sound instruction, etc. It is desired that the required moving trajectory calculated as a parameter is displayed on the screen as an instruction trajectory for performing additional scanning. Then, return from step S24 to step S1 and rescan. In addition, in the illustrated example, the moving trajectory of steps S8 to S10 is also determined and corrected during the additional scan. Additional scanning is continued until the reliability of registration satisfies a predetermined reliability (step S16 ) or until it is determined that there is no missing area (for example, less than a certain ratio) (step S17 ).

[0075] As described above, in the above embodiment, the missing area is determined for the part where data should be acquired if the scan is performed properly, the measurement parameters required to acquire the data of the missing area are calculated, and the operator is prompted to perform additional scanning according to the measurement parameters, so that three-dimensional measurement data can be acquired efficiently. In addition, it is possible to suppress the reduction of various inspection accuracies caused by the existence of the missing area during scanning. In particular, the missing area is determined only for the parts with low reliability of registration, and additional scanning is performed to make up for the missing area of ​​the part, so the additional scanning efficiency is high.

[0076] Furthermore, since the missing area is determined after the reference point cloud data and the target point cloud data are registered, it is possible to more accurately understand which part of the target component is the missing area.

[0077] In the above embodiment, it is determined whether the movement trajectory of the three-dimensional laser scanner 5 operated by the operator follows a predetermined designated trajectory, and correction is prompted to follow the designated trajectory. Therefore, scanning becomes easier and more appropriate, and the operation time can be shortened.

[0078] In the above, an embodiment of applying the present invention to the inspection of the gap between the ignition coil 3 and the fuel pipe 4 of the automobile is described, but the present invention is not limited to such an application, and can be widely applied to the inspection associated with the spatial configuration state of the position, posture, etc. of the target component in the assembly including multiple components. In addition, in the above embodiment, two components are used as the target components in order to inspect the distance between the two components, but the present invention can be applied even if there is only one target component.

[0079] In addition, in the above embodiment, an example is described in which an operator holds the three-dimensional laser scanner 5 for operation, but when a robot uses a three-dimensional sensor to scan, the above-mentioned missing area processing, movement trajectory estimation, etc. can also be applied. In this case, additional scanning can be performed by controlling the robot arm according to the measurement parameters calculated to supplement the missing area.

[0080] In addition, the three-dimensional sensor is not limited to the three-dimensional laser scanner 5 of the above embodiment, and can be any type of three-dimensional sensor as long as it can acquire and generate three-dimensional point cloud data. Three-dimensional sensors in the form of ToF, three-dimensional sensors in the form of triangulation such as stereo cameras, etc. can be widely used.

Claims

1. A component inspection method for inspecting a spatial arrangement state of one or more components as a target in an assembly formed by assembling a plurality of components, wherein: Scanning an area including the object component from the outside using a three-dimensional sensor to obtain measurement data of a portion of the outer surface of the object component, Acquire design data including the external shape of each target component and the positional relationship of the target component in the assembly, Compare the measured data of the target part with the design data of the target part to determine the missing area in the measured data of the target part. determining measurement parameters of a three-dimensional sensor suitable for acquiring measurement data of the missing area, An additional scan is performed according to the measurement parameters.

2. The component inspection method according to claim 1, wherein: After the point cloud data of the target component based on the design data and the point cloud data of the target component extracted from the measurement data are registered, the missing area is determined.

3. The component inspection method according to claim 2, wherein: Each of the plurality of object parts is registered, and for each object part, the reliability of the registration is calculated based on the number of corresponding points in the point cloud. For target parts with low registration reliability, missing area determination is performed.

4. The component inspection method according to claim 1, wherein: For each of the plurality of virtual viewpoints, it is evaluated whether the data of the missing area can be well acquired when the three-dimensional sensor acquires data from the virtual viewpoints. The trajectory connecting the plurality of virtual viewpoints that are advantageous and the angle of the three-dimensional sensor required at each virtual viewpoint are determined as measurement parameters.

5. The component inspection method according to claim 1, wherein: estimating a movement trajectory of the three-dimensional sensor during scanning based on data acquired by the three-dimensional sensor, Determine whether the moving trajectory deviates from the indicated trajectory, In the event of a deviation, information is output to prompt a correction.

6. The component inspection method according to claim 5, wherein: While repeatedly performing a rough alignment using key points to align the point cloud data of an area including an object part based on the design data with the point cloud data of the area acquired by scanning, the movement trajectory of the three-dimensional sensor is estimated, deviations are determined, and information prompting corrections is output.

7. The component inspection method according to claim 5, wherein: The 3D sensor operated by the operator is equipped with a vibrator. The vibrator generates vibrations of an intensity corresponding to the degree of deviation as information to prompt correction.

8. The component inspection method according to claim 2, wherein: The progress of the scan is determined based on the matching status of the two roughly registered point cloud data, and the registration of the target component is performed when the progress reaches a predetermined level.

9. A component inspection device for performing inspection related to the spatial arrangement state of one or more components as a target in an assembly formed by assembling a plurality of components, the component inspection device comprising: a three-dimensional sensor that scans an area including the object part from the outside; a measurement data acquisition unit that acquires measurement data including a portion of an outer surface of a target component by scanning using the three-dimensional sensor; A design data acquisition unit that acquires design data including the outer shape of each target component and the positional relationship of the target component in the assembly from a database; a missing region determination unit that compares the measurement data of the target component with the design data of the target component to obtain a missing region in the measurement data of the target component; a measurement parameter calculation unit for obtaining a measurement parameter of a three-dimensional sensor suitable for acquiring measurement data of the missing area; as well as The information output unit outputs the measurement parameters to the operator to perform additional scanning.

Images