Inspection assistance system, inspection assistance method, and inspection assistance program

Abstract

An inspection assistance system according to the present invention identifies the shape of an inspection object on the basis of a two-dimensional real image obtained as a result of the inspection object being imaged by an imaging device. A two-dimensional simulated image corresponding to the two-dimensional real image is extracted from a three-dimensional CAD model on the basis of the identified shape. A projection range designated from the two-dimensional real image is projected onto the three-dimensional CAD model by being adapted to the two-dimensional simulated image.

Description

Inspection support system, inspection support method, and inspection support program

[0001] The present disclosure relates to an inspection support system, an inspection support method, and an inspection support program.

[0002] When inspectors visually inspect objects (including products, machine parts, and intermediate products in the manufacturing process), determine the dimensions of defects detected in the objects, and enter the information in a report, this is time-consuming and results in inaccuracies that vary depending on the inspector's abilities.To solve this problem, in recent years, technologies have been proposed that enable the determination of defect dimensions by analyzing two-dimensional actual images obtained by capturing images of the objects to be inspected with an imaging device such as a camera.

[0003] For example, Patent Literature 1 discloses a method for supporting inspection work by analyzing a two-dimensional actual image obtained by capturing an image of an inspection object for which a three-dimensional CAD model exists in advance, thereby determining the dimensions of defects in the inspection object. In this document, the shape of the inspection object contained in the two-dimensional actual image is identified, and a reference portion contained in the shape is compared with a reference portion contained in a two-dimensional simulated image extracted from a three-dimensional CAD model corresponding to the inspection object, thereby depicting the defects contained in the two-dimensional actual image on the three-dimensional CAD model. By depicting the defects on the three-dimensional CAD model in this manner, the dimensions of the defects can be determined on the three-dimensional CAD model.

[0004] Japanese Patent Application Laid-Open No. 2021-21669

[0005] In the above-mentioned Patent Document 1, only defects identified based on predetermined criteria are depicted in a 3D CAD model, and elements other than defects in the 2D actual image are discarded. However, in order to perform high-quality inspection, information on elements that cannot be determined as defects based on the predetermined criteria, such as the texture of the surface condition of the inspection object, is also important.

[0006] At least one embodiment of the present disclosure has been made in consideration of the above-mentioned circumstances, and aims to provide an inspection support system, an inspection support method, and an inspection support program that can perform high-quality inspections of objects to be inspected.

[0007] In order to solve the above problem, an inspection support system according to at least one embodiment of the present disclosure comprises: a shape identification unit for identifying a shape of an object to be inspected contained in a two-dimensional actual image based on a two-dimensional actual image obtained by imaging the object to be inspected with an imaging device; a two-dimensional simulated image extraction unit for extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from a three-dimensional CAD model of the object to be inspected based on the shape identified by the shape identification unit; a projection range identification unit for identifying a projection range of the two-dimensional actual image; and an image projection unit for projecting the projection range onto the three-dimensional CAD model by fitting the projection range to the two-dimensional simulated image.

[0008] In order to solve the above problem, an inspection support method according to at least one embodiment of the present disclosure includes the steps of: identifying a shape of an object to be inspected contained in a two-dimensional actual image based on a two-dimensional actual image obtained by capturing an image of the object to be inspected with an imaging device; extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from a three-dimensional CAD model of the object to be inspected based on the shape; identifying a projection range of the two-dimensional actual image; and projecting the projection range onto the three-dimensional CAD model by fitting the projection range to the two-dimensional simulated image.

[0009] In order to solve the above problem, an inspection assistance program according to at least one embodiment of the present disclosure causes a computer device to execute the following steps: identifying the shape of an object to be inspected contained in a two-dimensional actual image based on a two-dimensional actual image obtained by capturing an image of the object to be inspected with an imaging device; extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from a three-dimensional CAD model of the object to be inspected based on the shape; identifying a projection range of the two-dimensional actual image; and projecting the projection range onto the three-dimensional CAD model by fitting the projection range to the two-dimensional simulated image.

[0010] According to at least one embodiment of the present disclosure, it is possible to provide an inspection support system, an inspection support method, and an inspection support program that are capable of performing high-quality inspections of inspection objects.

[0011] 1 is a block diagram showing a schematic configuration of an inspection support system according to an embodiment. It is a schematic diagram showing the hardware configuration of the inspection support system of FIG. 1. It is an explanatory diagram of distortion correction processing by the distortion correction unit of FIG. 1. It is an example of a two-dimensional actual image to which fast Fourier transform processing has been applied. It is another example of a two-dimensional actual image to which fast Fourier transform processing has been applied. It is a conceptual diagram for explaining a method of detecting a reference portion by the shape identification unit of FIG. 1. It is a diagram conceptually showing the correspondence between a first coordinate system and a second coordinate system using coordinate transformation parameters estimated by PNP. It is a flowchart showing an inspection support method according to an embodiment. It is an example of a two-dimensional actual image captured by an imaging device. It is a schematic diagram showing the default posture of a three-dimensional CAD model. It is a schematic diagram showing the posture of the three-dimensional CAD model of FIG. 9 after at least one of the position and the direction has been changed by a three-dimensional CAD model modification unit. It is a modified example of FIG.

[0012] Hereinafter, several embodiments of the present disclosure will be described with reference to the accompanying drawings. However, the dimensions, materials, shapes, relative arrangements, etc. of components described as embodiments or shown in the drawings are merely illustrative examples and are not intended to limit the scope of the present disclosure.

[0013] FIG. 1 is a block diagram showing a schematic configuration of an inspection support system 1 according to one embodiment. The inspection support system 1 is a system for supporting the detection of defects in an inspection object 2 and the creation of a report related to the inspection work when an inspection work is performed on the inspection object 2. The user of the inspection support system 1 may be a worker who performs the inspection work, a user who uses the inspection object 2, or another third party. The defects detected by the inspection support system 1 are defects that can be identified from the appearance, and the types thereof include, for example, cracks, dents, scratches, poor coating, oxidation, thinning, dirt, etc.

[0014] The inspection object 2 may be the entire product, or may be a component (e.g., a machine part) that constitutes the product. The inspection object 2 may be a new product, a repaired product, or an existing facility. The inspection object 2 may be, for example, a moving blade, a stationary blade, a ring segment, a combustor, or the like of a gas turbine.

[0015] The inspection support system 1 includes at least one computer device. The inspection support system 1 may be configured as a single device, but in FIG. 1 it includes a client terminal 6 and a server 8, which are communication terminals that can communicate with each other via a communication network 4, and the client terminal 6 and the server 8 work together to realize the functions of the inspection support system 1. The communication network 4 may be a WAN (World Area Network) or a LAN (Local Area Network), and may be wireless or wired.

[0016] Fig. 2 is a schematic diagram showing the hardware configuration of the inspection support system 1 of Fig. 1. The client terminal 6 includes a communication unit 10 for communicating with the server 8 via the communication network 4, a memory unit 11 for storing various data, an output unit 12 for outputting various information, an input unit 13 for accepting user input, and a calculation unit 14 for performing various calculations. The server 8 includes a communication unit 15 for communicating with the client terminal 6 via the communication network 4, a memory unit 16 for storing various data, and a calculation unit 18 for performing various calculations. The internal configurations of the client terminal 6 and the server 8 are connected to each other via a bus line.

[0017] The communication units 10 and 15 are communication interfaces equipped with a NIC (Network Interface Card controller) for wired or wireless communication, and enable communication between the client terminal 6 and the server 8 via the communication network 4 .

[0018] The storage units 11 and 16 are composed of RAM (Random Access Memory), ROM (Read Only Memory), etc., and store programs (e.g., inspection assistance programs, trained models described later) for executing various control processes performed by the client terminal 6 and the server 8, respectively, and data necessary for the various control processes.

[0019] The various data stored in the storage units 11 and 16 include multiple learning models and multiple 3D CAD models of objects used in various processes described below. In this embodiment, as described below with reference to FIG. 7 and other figures, these learning models and 3D CAD models are stored in the storage unit 16 of the server 8. Since these models generally require a large storage capacity, storing them in the storage unit 16 on the server 8 side can prevent the storage capacity on the client terminal 6 side from being overwhelmed. In this case, the client terminal 6 can be realized as a small terminal or a portable terminal such as a laptop computer, which is effective in improving convenience.

[0020] The multiple objects include the inspection object 2 described above. A 3D CAD model is a 3D CG (computer graphic) representation of the object in a 3D virtual space with actual dimensions. The surface of this 3D CG has a mesh structure consisting of points, lines, and faces, and a 2D simulated image can be obtained by generating (rendering) an image from any viewpoint. The viewpoint of the 2D simulated image can be changed by moving the viewpoint (camera) position within the 3D virtual space. Therefore, by acquiring a 2D simulated image after moving the viewpoint position, the obtained 2D simulated image can be effectively rotated, enlarged, and reduced. Note that the storage units 11 and 16 may be configured as a single storage device or as multiple storage devices. Furthermore, the storage units 11 and 16 may be external storage devices.

[0021] The output unit 12 is configured by an output device such as a display device, a speaker device, etc. The output unit 12 is an output interface for presenting various information to the user.

[0022] The input unit 13 is an input interface for inputting information necessary for carrying out various processes from the outside, and is composed of input devices such as operation buttons, a keyboard, a pointing device, a microphone, etc. Such information includes, in addition to instructions from the user, data on a two-dimensional actual image acquired by an imaging device, as will be described later, and the like.

[0023] The calculation units 14 and 18 are configured with processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The calculation units 14 and 18 execute programs stored in the storage units 11 and 16 to control the operation of the entire system.

[0024] Next, a specific description will be given of the functional configurations of the client terminal 6 and the server 8 that constitute the inspection support system 1. As shown in Fig. 1, the client terminal 6 includes an image acquisition unit 30, a distortion correction unit 31, a projection range specification unit 32, a shape identification unit 33, a defect detection unit 34, and a report creation unit 36. The server 8 includes a coordinate transformation parameter estimation unit 40, a 3D CAD model modification unit 41, an image extraction unit 42, an image projection unit 43, and a defect depiction unit 44.

[0025] In this embodiment, the functional configurations are arranged across the client terminal 6 and the server 8, but they may be arranged in either the client terminal 6 or the server 8. For example, by arranging the configuration shown on the server 8 side in FIG. 1 in the client terminal 6, this embodiment may be realized by the client terminal 6 alone without using the server 8. Furthermore, the functional configurations provided in either the client terminal 6 or the server 8 may be arranged in the other as appropriate. In this way, the layout of the functional configurations provided in the inspection support system 1 can be changed as appropriate depending on the application.

[0026] The image acquisition unit 30 is configured to acquire a two-dimensional real image of the inspection object 2 captured by an imaging device 50. The imaging device 50 is, for example, a camera compatible with visible light, and is configured to acquire a two-dimensional real image, which is an image captured of the inspection object 2. The acquisition of the two-dimensional real image by the imaging device 50 may be performed at the same location as the inspection support system 1, particularly the client terminal 6 to which the two-dimensional real image is input, or may be performed at a different location (remote location). Data related to such two-dimensional real image is acquired by inputting it to the input unit 13 of the client terminal 6.

[0027] The imaging device 50 for capturing a 2D real image may have a standard lens, a wide-angle lens, an ultra-wide-angle lens (or a fisheye lens), or a telephoto lens as an imaging lens. A standard lens has an angle of view ranging from 50 degrees to 25 degrees, including 45 degrees, which is generally considered to be the same angle of view as human vision. A wide-angle lens has a wider angle of view than a standard lens, and generally has an angle of view of 60 degrees or more (more preferably, in the range of 100 degrees to 60 degrees). An ultra-wide-angle lens (or a fisheye lens) has an even wider angle of view than a wide-angle lens, and its maximum angle of view may be 180 degrees or more. A telephoto lens has a narrower angle of view than a standard lens, and generally has an angle of view of 30 degrees or less (more preferably, in the range of 15 degrees to 10 degrees).

[0028] For example, if the imaging device 50 uses a wide-angle lens as an imaging lens, the imaging device 50 has a wide imaging range, so that even a relatively large inspection object 2 can be included in a single two-dimensional actual image. Furthermore, the imaging device 50 may have an imaging lens with an ultra-wide angle of view, such as an endoscope camera (borescope camera), and its use is not limited.

[0029] The distortion correction unit 31 is configured to correct distortions contained in the two-dimensional real image acquired by the image acquisition unit 30. Distortions contained in the two-dimensional real image are a factor that reduces inspection accuracy. The distortion correction unit 31 performs a correction process on such two-dimensional real image to reduce (or remove) the distortion contained in the two-dimensional real image and improve inspection accuracy. In particular, when the imaging device 50 is equipped with a wide-angle lens, the two-dimensional real image acquired by the imaging device 50 contains a large amount of distortion. Therefore, by correcting the distortion contained in the two-dimensional real image by the distortion correction unit 31, inspection accuracy based on the two-dimensional real image captured using the wide-angle lens can be effectively improved. Note that the distortion correction process does not need to be performed on two-dimensional real images with no or little distortion.

[0030] The distortion correction process by the distortion correction unit 31 will now be described in detail with reference to Fig. 3. Fig. 3 is an explanatory diagram of the distortion correction process by the distortion correction unit 31 of Fig. 1.

[0031] The distortion correction unit 31 captures a test image having a predetermined depiction pattern BP using the imaging device 50, and is able to determine the distortion correction parameter Pa based on the image data obtained as a result. The distortion correction data Pa thus obtained can be used to correct distortion in a two-dimensional actual image. The shape pattern included in the depiction pattern BP may be any shape pattern, but FIG. 3 illustrates a depiction pattern BP consisting of mutually orthogonal grid lines. Note that if specification data for the imaging lens of the imaging device 50 is available in advance, the distortion correction parameter Pa may be calculated numerically based on the specification data.

[0032] The projection range identification unit 32 is configured to identify the range of the two-dimensional actual image acquired by the image acquisition unit 30 (or the two-dimensional actual image after distortion correction processing, if distortion correction is performed by the distortion correction unit 31) to be projected onto the three-dimensional CAD model as the projection range.

[0033] In one aspect, the projection range specification unit 32 may specify the projection range manually by the operator operating the input unit 13 while displaying the target 2D actual image on the screen. In this case, it is possible to specify any range intended by the operator as the projection range.

[0034] In another aspect, the projection range specifying unit 32 may automatically specify the projection range based on a predetermined algorithm. In this case, the projection range is specified without relying on the subjective judgment of the operator, and therefore the projection range can be specified with stable quality.

[0035] The algorithm may also be configured to remove out-of-focus areas of the actual two-dimensional image from the projection area. In this case, the algorithm may perform the following processing, for example:

[0036] First, the 2D real image to be used to identify the projection range is obtained as a data point cloud containing, for example, information about RGB or brightness. A depth map is estimated based on this data point cloud. The depth map is estimated, for example, by converting the RGB or brightness at each point in the 2D real image using deep learning, such as an autoencoder. By operating the input unit 13, the operator can click on any point (e.g., a point on the inspection target 2) in the 2D real image. Areas other than those with a depth close to the point (e.g., areas where the depth difference is less than a threshold) are excluded from the projection range. In other words, by masking areas with large depth differences from the point selected by the operator, a projection range with out-of-focus areas removed can be identified. With this algorithm, the operator can identify an appropriate projection range with out-of-focus areas removed simply by clicking on any point.

[0037] In another aspect, the algorithm may be configured to automatically identify the projection range by applying a fast Fourier transform (FFT) process to the two-dimensional real image. The two-dimensional real image to which the fast Fourier transform has been applied is converted into a set of two-dimensional position coordinates and frequency values ​​associated with the coordinates. This conversion process is performed by selecting a frequency band with high intensity based on a frequency spectrum (i.e., an intensity distribution decomposed into each frequency component contained in an arbitrary signal) obtained as an output result of the fast Fourier transform, and associating the frequency band with the coordinate.

[0038] 4A and 4B show examples of two-dimensional real images to which fast Fourier transform processing has been applied. In FIGS. 4A and 4B, the points constituting the set are plotted in a gradational pattern, with the color approaching black as the frequency decreases and approaching white as the frequency increases. The degree of blur can be determined, for example, based on whether the average frequency value in the entire set or a portion of the set exceeds a predetermined threshold. FIG. 4A shows a normal set corresponding to a two-dimensional real image with little blur. On the other hand, FIG. 4B shows a set corresponding to a two-dimensional real image with a lot of blur, which appears whitish overall due to a higher average frequency than FIG. 4A.

[0039] If the actual 2D image is determined to be normal (less out-of-focus) by such a determination, the coordinate with the largest frequency value in the set is estimated to be the focal position. In this case, by specifying the range with the frequency value equal to or greater than a predetermined threshold value centered on the focal position as the projection range, it is possible to exclude areas with significant out-of-focus from the projection range.

[0040] In addition, if this algorithm determines that the actual two-dimensional image is out of focus and not normal (if the reference frequency is not met throughout the actual two-dimensional image), the algorithm may prompt the user to re-acquire the actual two-dimensional image.

[0041] The shape identification unit 33 is configured to identify the two-dimensional shape of the inspection object 2 in the two-dimensional actual image. The shape identification unit 33 may be configured to identify the shape of the inspection object 2 included in the two-dimensional actual image by detecting a plurality of reference portions of the inspection object 2 in the two-dimensional actual image.

[0042] Here, Fig. 5 is a conceptual diagram for explaining a method for detecting a reference portion by the shape identification unit 33 in Fig. 1. Note that Fig. 5 illustrates a case where the inspection object 2 has a triangular shape when viewed from the front, but the shape is not limited thereto.

[0043] First, the shape identification unit 33 is configured to detect a characteristic part of the inspection target object 2 as a reference part. In one embodiment, the characteristic part is a corner. The characteristic part may be any part that is useful for identifying the shape, and may be, for example, a landmark, a trademark, a keyhole, a button, or the like.

[0044] In one embodiment, machine learning (e.g., SSD (Single Shot MultiBox Detector) technology) is used to detect the reference portion. SSD technology is known as a method for detecting objects in an image using deep learning. Specifically, as shown in FIG. 5 , the shape identification unit 33 searches the two-dimensional actual image P1 using a default box B0 and detects a corner of the inspection object 2 as the reference portion. The shape of the reference portion is identified based on the RGB values ​​indicating the red, green, and blue components of each pixel, the hue compared to other pixels, and brightness information.

[0045] As a result, default boxes B1, B2, and B3 corresponding to the three corners are detected as reference portions. Furthermore, by including brightness information in the input, the shape identification unit 33 can detect reference portions even when there is some brightness variation, for example, because a portion of the actual 2D image P1 is not blurred. Detection of reference portions based on such brightness information can be performed using an estimation model constructed by machine learning using images with brightness variations as training data, making it possible to detect reference portions from the actual 2D image P1 that includes some brightness variation.

[0046] The shape identification unit 33 detects at least three (preferably four or more) reference portions. The greater the number of reference portions detected by the shape identification unit 33, the more accurate the estimation of coordinate transformation parameters by the coordinate transformation parameter estimation unit 40 (described later) can be. However, for example, if the reference portions are specified manually by the user, the operational burden on the user increases accordingly. Therefore, the number of reference portions detected by the shape identification unit 33 should be set in consideration of the balance between estimation accuracy and operational burden.

[0047] In this way, the shape identification unit 33 detects the reference part based on RGB values, color tone, and brightness information, thereby reducing detection errors due to brightness fluctuations. Furthermore, when identifying the shape of the reference part using machine learning based on deep learning, it is possible to automate the operator's work and reduce the work time compared to when the operator identifies the reference part based on subjective judgment.

[0048] In other embodiments, other machine learning techniques may be used to detect the reference portion, such as Region-Based Convolutional Neural Networks (R-CNN) or YOLO (You Only Look Once).

[0049] The defect detection unit 34 is configured to detect defects of the inspection object 2 contained in the two-dimensional actual image. The defect detection unit 34 may detect defects of the inspection object 2 using a trained model that has been machine-learned to understand the relationship between the two-dimensional actual images of each of a plurality of objects including the inspection object 2 and images of defects that may occur in the plurality of objects. For example, the defect detection unit 34 may be configured to analyze the RGB values ​​of each pixel of the two-dimensional actual image and determine defects by pattern classification of the contrast and color tone.

[0050] The trained model used by the defect detection unit 34 to detect defects from the actual 2D image can be created by deep learning or machine learning. In this case, a large amount of training data that associates the relationship between the actual 2D image and the defect area is used for training, and the trained model outputs an estimation result of the defect area as pixel coordinate data of the actual 2D image.

[0051] The coordinate transformation parameter estimation unit 40 is configured to estimate coordinate transformation parameters for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to the 2D actual image based on the shape identified by the shape identification unit 32 and the 3D CAD model of the inspection target object 2. The coordinate transformation parameters are parameters for transforming coordinate points in the first coordinate system and the second coordinate system, and one of the estimation methods is the Perspective-Points Problem (PNP). In other words, the coordinate transformation parameters are parameters for estimating the position and orientation that define the viewpoint information of the imaging device in the first coordinate system based on n-point (n is any natural number equal to or greater than 3) reference points (reference points detected by the shape identification unit 33 as described above) represented by three-dimensional coordinates in the first coordinate system corresponding to the 3D CAD model handled on the 3D CG software and those reference points represented by two-dimensional coordinates in the second coordinate system corresponding to the 2D actual image.

[0052] 6 is a conceptual diagram showing the correspondence between the first coordinate system and the second coordinate system based on the coordinate transformation parameters estimated by the PNP. In the PNP, coordinate transformation parameters (such as a translation vector or a rotation matrix) are estimated as parameters for converting between the three-dimensional coordinates (X1, X2, ...) of n points in the first coordinate system, which is a world coordinate system corresponding to a three-dimensional CAD model handled on 3D graphics software, and the two-dimensional coordinates (x1, x2, ...) of n points in the second coordinate system corresponding to the viewpoint of the imaging device 50 that captured the two-dimensional actual image containing those points.

[0053] The PNP estimates the coordinate transformation parameters, for example, so that corresponding reference portions in the first coordinate system and the second coordinate system coincide with each other. Specifically, the coordinate transformation parameter estimation unit 40 identifies multiple first reference portions (multiple coordinate points in the second coordinate system) of the inspection object 2 in the 2D actual image based on the shape identified by the shape identification unit 33, for example, by automatic detection using machine learning or manual input by the user, and pre-registers multiple second reference portions (multiple coordinate points in the first coordinate system) in the 3D CAD model. The second reference portions are registered by pre-registering the second reference portions in a database or the like. Such registration of the second reference portions may be performed, for example, by displaying the 3D CAD model on a screen and having the operator specify the second reference portions on the screen using a cursor, pointer, or the like. Then, the coordinate transformation parameters are estimated so that the first reference portions coincide with the second reference portions.

[0054] The coordinate transformation parameters include external parameters of the image capture device 50. The external parameters are parameters necessary to define the position and orientation (six degrees of freedom) of the image capture device 50 in the first coordinate system. In other words, the external parameters are parameters for reproducing, in the first coordinate system, the same location as the image capture position of the two-dimensional actual image, rather than a relative position, and are expressed by, for example, a translation vector or a rotation matrix.

[0055] The coordinate transformation parameters may also include internal parameters of the image capture device 50. The internal parameters of the image capture device 50 are parameters related to the main body (lens) of the image capture device 50, such as the focal length f and the optical center. The optical center corresponds to the origin of the second coordinate system, is a lens-specific parameter, and is expressed as two-dimensional coordinates (Cu, Cv). In this case, the relationship between the three-dimensional coordinates (Xw, Yw, Zw) of the feature point in the first coordinate system corresponding to the 3D CAD model and the two-dimensional coordinates (u, v) of the feature point in the second coordinate system corresponding to the 2D actual image is expressed as follows:

[0056] 1 , the 3D CAD model modification unit 41 modifies the position and direction of the viewpoint information of the 3D CAD model of the inspection object 2 using the coordinate transformation parameters estimated by the coordinate transformation parameter estimation unit 40. As a result, the viewpoint information of the 3D CAD model shown from the viewpoint of a camera located at a specific position and direction on the 3D CAD model rendering software is modified to correspond to the position and direction corresponding to the viewpoint of the imaging device 50 that captured the 2D actual image.

[0057] Based on the identification result by the shape identification unit 32, the image extraction unit 42 refers to the 3D CAD model whose viewpoint information has been changed by the 3D CAD model modification unit 41, and extracts a 2D simulated image corresponding to the 2D actual image from the 3D CAD model.

[0058] The image projection unit 43 is configured to adjust the projection range specified by the projection range specifying unit 32 in the 2D actual image so that it matches the 2D simulated image, and project the adjusted projection range onto the 3D CAD model. This adjustment may be performed before or when projecting the projection range onto the 3D CAD model, or may be performed on the already projected projection range after it has been projected onto the 3D CAD model. The 3D CAD model with the projection range projected may be displayed on the display device of the output unit 12.

[0059] For example, the image projection unit 43 uses a geometric transformation (plane transformation), such as an affine transformation, to convert the projection range specified in the 2D actual image into the coordinate system of the 2D simulated image using a reference portion (multiple coordinate points) of the target image (projection range) as an input value, thereby projecting it onto the 3D CAD model. At this time, since the viewpoint information of the 3D CAD model is changed as described above, the image projection unit 43 defines the projection range according to the converted coordinates and projects it onto the 3D CAD model.

[0060] For example, the image projection unit 43 may compare the lengths of the sides determined from the positional relationships of multiple reference parts between the actual 2D image and the simulated 2D image, and expand or reduce the projection range based on the similarity ratio. The image projection unit 43 may compare the positional relationships of multiple reference parts between the actual 2D image and the simulated 2D image, and adjust the aspect ratio of the projection range, translate, linearly transform, etc.

[0061] The defect depiction unit 44 is configured to depict defects detected by the defect detection unit 34 on a three-dimensional CAD model. The defect depiction unit 44 depicts defects detected from the two-dimensional actual image, adjusts the depiction of the defect to fit the two-dimensional simulated image, similar to the projection of the projection range by the image projection unit 43 described above, and depicts the adjusted depiction of the defect on the three-dimensional CAD model. For example, the defect depiction unit 44 depicts the defect depicted on the two-dimensional actual image on the three-dimensional CAD model by converting the defect depicted on the two-dimensional actual image into the coordinate system of the two-dimensional simulated image using a geometric transformation (plane transformation), for example, an affine transformation, or other transformation method.

[0062] The defect depiction by the defect depiction unit 44 is performed programmatically on the operation screen of the three-dimensional CG software using the point sequence coordinates of the defect area converted into the coordinates of the two-dimensional simulated image.

[0063] As a supplementary explanation, in the above-mentioned projection of the projection range or depiction of the defect, there are mainly two methods for converting the projection range or the defect into coordinates of the 3D CG software.

[0064] The first method is geometric transformation. Assume that the target size of the coordinate transformation is 6000 × 4000 and the screen size of the 3D CG software is 1200 × 800. The geometric transformation is performed by rotating, translating, shearing, and scaling each feature point (paired feature points). When a defect, which is a point sequence coordinate, is to be transformed, a geometric transformation is performed using the screen size (1200 × 800) of the 3D CG software, thereby converting it into a point sequence coordinate that can be processed on the 3D CG software screen. On the other hand, when a projection range (such as a rendering image or a 2D pseudo image) is to be transformed, it is output according to the photo size (6000 × 4000) of this system. In this case, the feature point coordinates on the screen of the 3D CG software are relatively adjusted to the photo size (6000 × 4000). Regarding the projection range, it is preferable to perform a geometric transformation using the feature points that have undergone such relative transformation. In this case, since the size (6000 x 4000) is the same, no scaling is performed and the image is output at the same size as the original, and does not become coarse pixels (1200 x 800).

[0065] The second method is simple scaling based on the size ratio. Unlike the first method, this method does not involve geometric transformations (rotation, translation, shearing), but simply scales the defect information depicted on the photograph to fit the 3D CG screen (1200 x 800). In this case, feature points are not required. Furthermore, the original photograph can be projected directly onto the 3D CAD model.

[0066] The report creation unit 36 ​​is configured to create a report summarizing the inspection results. The inspection results can include data on the position and dimensions of the defect area drawn by the user on the system within the projection range of the photograph, as well as a 3D CAD model onto which the projection range of the photograph is projected. This allows the inspection results to include not only information about the defect depicted on the 3D CAD model, but also information about areas that cannot be clearly identified as defects, such as the texture around the defect, and related areas around the defect, enabling higher quality inspections. The created report may be stored in the storage unit 11 or transmitted to another device (e.g., a server device that manages reports).

[0067] Next, a description will be given of an inspection support method implemented by the inspection support system 1 having the above configuration. Fig. 7 is a flowchart showing an inspection support method according to one embodiment.

[0068] First, as a preliminary step before the inspection support method is performed, an image of the inspection target 2 is captured using the imaging device 50 (step S1). In step S1, an image of the inspection target 2 can be captured from any position and direction, and the two-dimensional actual image obtained by the imaging device 50 is input as data to the client terminal 6. As described above in this embodiment, the imaging device 50 has a wide-angle lens with a wide angle of view as an imaging lens, so that even if the inspection target 2 is relatively large, a two-dimensional actual image can be captured so that the entire object is included in a single frame.

[0069] In the client terminal 6, the image acquisition unit 30 acquires a 2D actual image as data input from the imaging device 50 (step S2). Then, in the client terminal, the distortion correction unit performs a distortion correction process to correct the distortion contained in the 2D actual image acquired in step S2 (step S3). This reduces the distortion contained in the 2D actual image acquired in step S2, thereby improving inspection accuracy.

[0070] Next, the projection range specifying unit 32 specifies the projection range from the actual 2D image (step S4). As described above with reference to several aspects, the specification of the projection range may be performed manually by an operator, or may be performed at least in part automatically based on a predetermined algorithm (in the embodiment shown in FIG. 7 , as described above, step S5 shows a step in which the actual 2D image is input into a learning model previously constructed using learning data, and out-of-focus areas of the actual 2D image are excluded from the projection range).

[0071] Next, in the client terminal 6, the shape identification unit 32 detects the reference portion of the inspection object 2 in the two-dimensional actual image on which various processes have been performed up to this point, and identifies the shape of the inspection object 2 (step S6).

[0072] Next, the coordinate transformation parameter estimation unit 40 accesses the learning model for the reference portion detected when identifying the shape in step S6 (step S7) and estimates coordinate transformation parameters (step S8). The coordinate transformation parameters are estimated by, for example, PNP, as specifically described above.

[0073] Next, the three-dimensional CAD model modification unit 41 modifies at least one of the position and orientation of the three-dimensional CAD model using the coordinate transformation parameters (for example, translation vectors, rotation matrices, etc.) estimated in step S8 (step S9).

[0074] 8 to 10, the processing by the 3D CAD model modification unit 41 will be specifically described. Fig. 8 is an example of a 2D actual image captured by the imaging device 50, Fig. 9 is a schematic diagram showing the default orientation of the 3D CAD model, and Fig. 10 is a schematic diagram showing the orientation of the 3D CAD model of Fig. 9 after at least one of the position and orientation has been changed by the 3D CAD model modification unit 41.

[0075] As shown in FIG. 8 , the two-dimensional actual image captured by the imaging device 50 captures C1 to C7 of the multiple reference portions C1 to C8 of the inspection object 2. On the other hand, in the default orientation of the three-dimensional CAD model shown in FIG. 9 , only some of the multiple reference portions of the inspection object 2 shown in FIG. 8 can be confirmed (specifically, the reference portions C2 to C8 common to FIG. 8 are visible in FIG. 9 , but the reference portion C1 of FIG. 8 is not visible). As shown in FIG. 10 , by changing at least one of the position or orientation of the three-dimensional CAD model having the default orientation shown in FIG. 9 using coordinate transformation parameters, the three-dimensional CAD model having such a default orientation is brought into an orientation corresponding to the two-dimensional actual image shown in FIG. 8 , thereby including all of the reference portions C1 to C7.

[0076] Next, the image extraction unit 42 of the server 8 extracts a two-dimensional simulated image from the 3D CAD model whose viewpoint information has been changed by the 3D CAD model change unit 41, including portions corresponding to the multiple reference portions of the inspection object 2 detected from the two-dimensional actual image (step S10). In step S10, for example, the two-dimensional simulated image may be extracted by displaying the 3D CAD model whose viewpoint information has been changed on a screen and acquiring a rendering image (screenshot) of the screen. In this case, the rendering image (screenshot) may be acquired by actually displaying the 3D CAD model whose viewpoint information has been changed on a screen, or may be acquired computationally without displaying it on a screen. A trained model may be used for this extraction.

[0077] Next, the defect detection unit 34 of the client terminal 6 instructs the server 8 to detect defects in the inspection object 3 based on the 2D actual image (step S11). The server 8, which has received the detection instruction, accesses a learning model for defect detection prepared in advance (step S12) and executes defect detection using the learning model (step S13).

[0078] The client terminal 6 acquires the detection result of step S13 from the server 8, and determines whether or not there is a defect in the inspection object 2 based on the detection result (step S14). If it is determined that there is no defect in the inspection object 2 (step S14: NO), steps S15 to S19 are skipped, and the report creation unit 36 ​​creates a report indicating that there is no defect (step S20).

[0079] On the other hand, if it is determined that the inspection object 2 has a defect (step S14: YES), the coordinates of the defect included in the 2D actual image (point sequence coordinate data representing the outline of the defect) are converted to match the coordinates of the 2D simulated image, and the projection range (at least a part of the 2D actual image) identified in step S4 is converted to match the coordinates of the 2D simulated image (step S15).Then, the defect depiction unit 44 depicts the coordinate data (point sequence coordinate data) of the defect that has been coordinate-converted in step S15 on a 3D CAD model, and calculates the area of ​​the depicted defect on the 3D CAD model (step S16).

[0080] Next, the image projection unit 43 projects the projection range converted in step S15 onto the 3D CAD model (step S17). The defect area calculated in step S16 and the 3D CAD model onto which the projection range is projected in step S17 are output to the client terminal 6 as a 3D inspection record (step S18). In this way, the 3D inspection record includes not only basic information regarding the defect area, etc., but also the 3D CAD model onto which the projection range is projected. Therefore, not only the defect itself but also information such as the texture around the defect is included, thereby achieving high-quality inspection accuracy.

[0081] The client terminal 6 acquires data relating to the three-dimensional inspection record from the server 8 (step S19), and the report creation unit 36 ​​creates a report based on the three-dimensional inspection record (step S20).

[0082] Each of these steps may be repeated as necessary. For example, in step S4, only a portion of the inspection object 2 included in the 2D actual image may be identified as the projection range. In this case, the area not included in the projection range is omitted from the inspection range. Therefore, by repeatedly capturing multiple 2D actual images using the imaging device 50, multiple projection ranges identified from each 2D actual image are projected onto the 3D CAD model, thereby projecting the projection range over a wide area of ​​the 3D CAD model. In particular, by performing a sufficient number of repetitions, the projection range can be projected over the entire area of ​​the 3D CAD model, thereby achieving good inspection accuracy and inspection range.

[0083] Furthermore, when the above steps are repeatedly performed, the image capture device 50 may capture images of the inspection target 2 from different positions (or angles). Depending on the imaging conditions, a 2D real image captured from a specific position may have unclear areas (e.g., whiteout due to light reflection, out-of-focus, etc.). However, capturing images from multiple positions (or angles) in this manner can reduce the unclear areas in each 2D real image. Therefore, the projection ranges identifiable from each of these 2D real images tend to be different from each other, allowing the unclear areas projected onto the 3D CAD model to be compensated for. As a result, 2D real images can be efficiently projected over a wide area of ​​the 3D CAD model while minimizing the number of repetitions of the above steps.

[0084] In this way, in the above method, at least a portion of the two-dimensional actual image captured by the imaging device 50 is specified as a projection range and projected onto the three-dimensional CAD model depicting the defect. This projection range includes various information necessary for evaluating quality, such as defects and the texture of the surface of the inspection object 2, enabling high-quality inspection of the inspection object 2.

[0085] In the above-described embodiment, the entire inspection object 2 has been treated as a unit. However, depending on the inspection implementation, it may be necessary to define the inspection object 2 as being divided into multiple sections and evaluate the area and length of defects for each section. If such section-by-section inspection is performed manually by a worker, the worker must define the boundary lines of each section of the inspection object 2 and memorize the defined boundary lines before inspecting, which tends to reduce work efficiency and accuracy. In such cases, the inspection can be performed more appropriately using the modified example described below.

[0086] Fig. 11 is a modified example of Fig. 7. In the modified example shown in Fig. 11, the same processes as those in the embodiment shown in Fig. 7 are assigned common step numbers, and duplicated explanations will be omitted unless otherwise specified.

[0087] In this modification, step S21 is additionally performed compared to the embodiment shown in Fig. 7. In step S21, a 3D CAD model in which each part is defined in advance is read as the 3D CAD model onto which the projection range is projected. This eliminates the need for the worker to define or memorize the boundary lines of each part of the inspection object 2, effectively reducing the workload.

[0088] In the previously described embodiment, the defect coordinates (point sequence coordinate data) are depicted on a 3D CAD model in step S16 of FIG. 7 to calculate the defect area. However, in this modified example, instead, the defect is depicted on a 3D CAD model and the overall defect area is calculated (step S16-1). Then, based on information about the boundary lines defined for the 3D CAD model read in step S21, each feature defined on the 3D CAD model is identified, and the defect area for each identified feature is calculated individually (step S16-2). As a result, step S16-1 can evaluate the overall defect area, as in the previously described embodiment. Furthermore, step S16-2 further evaluates the defect area for each feature defined on the 3D CAD model. As a result, the burden on the operator of grasping or memorizing the boundary lines of each feature defined on the 3D CAD model is eliminated, effectively reducing the operator's workload. Furthermore, a decrease in inspection accuracy due to operator errors, such as forgetting or mistaking memorized boundary lines, can be avoided. As described above, according to this modified example, inspection of each part of the inspection object 2 can be carried out automatically, thereby achieving good inspection accuracy while reducing the work efficiency of the workers.

[0089] In the above example, the defect area of ​​each portion identified by a predefined boundary line in the 3D CAD model is evaluated. In another example, a mesh surface (multiple cells) included in each portion of the 3D CAD model is selected, and an identifying name is assigned to an object that is an aggregate of the mesh surfaces. Whether this object is a target for defect projection may be automatically identified based on whether it is included in the range in which the defect is to be drawn (i.e., whether the surface is located on the line indicating the outline of the defect to be drawn). As a result, if the defect is within the range in which the defect is to be drawn, the defect may be drawn only on the target object, and the area and length may be calculated. Furthermore, if there are multiple objects, the defect may be drawn while switching between displaying and processing the aggregate of these objects.

[0090] As described above, according to each of the above embodiments, a corresponding two-dimensional simulated image is extracted from a three-dimensional CAD model based on the shape of the inspection object 2 identified from a two-dimensional actual image captured by the imaging device 50. The projection range identified from the two-dimensional actual image and point sequence coordinate data of detected defects are projected or depicted onto the three-dimensional CAD model by coordinate transformation using the two-dimensional simulated image. The three-dimensional CAD model onto which the projection range is projected in this manner includes various information necessary for evaluating quality, such as the surface texture of the inspection object 2, in addition to the defects depicted in the three-dimensional CAD model. Using a three-dimensional CAD model onto which the projection range and defects are projected or depicted in this manner enables high-quality inspection of the inspection object 2.

[0091] In addition, within the scope of the present disclosure, the components in the above-described embodiments may be replaced with well-known components as appropriate, and the above-described embodiments may be combined as appropriate.

[0092] The contents described in each of the above embodiments can be understood, for example, as follows.

[0093] (1) An inspection support system according to one embodiment includes: a shape identification unit for identifying a shape of an object to be inspected contained in a two-dimensional actual image based on a two-dimensional actual image obtained by capturing an image of the object to be inspected with an imaging device; a two-dimensional simulated image extraction unit for extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from a three-dimensional CAD model of the object to be inspected based on the shape identified by the shape identification unit; a projection range identification unit for identifying a projection range of the two-dimensional actual image; and an image projection unit for projecting the projection range onto the three-dimensional CAD model by fitting the projection range to the two-dimensional simulated image.

[0094] According to the above aspect (1), a corresponding two-dimensional simulated image is extracted from a three-dimensional CAD model based on the shape of the inspection object identified from a two-dimensional actual image captured by an imaging device. The projection range identified from the two-dimensional actual image is projected onto the three-dimensional CAD model by coordinate transformation using the two-dimensional simulated image. The three-dimensional CAD model onto which the projection range is projected in this manner contains various information necessary for evaluating quality, such as defects and the surface texture of the inspection object, enabling high-quality inspection of the inspection object.

[0095] (2) In another aspect, in the aspect (1), the image projection unit projects the projection ranges of the plurality of two-dimensional actual images obtained by capturing images of the inspection object from different positions by the imaging device onto the three-dimensional CAD model, respectively.

[0096] According to the above aspect (2), a plurality of real 2D images are captured by an imaging device from different positions. Projection ranges identified from each of the plurality of real 2D images are projected onto the 3D CAD model. This allows for the real 2D images captured from other positions to compensate for unclear areas (e.g., whiteouts due to light reflections, out-of-focus areas, etc.) in the real 2D images captured from a particular position. As a result, the real 2D images can be suitably projected over a wide area of ​​the 3D CAD model, enabling highly accurate inspections.

[0097] (3) In another aspect, in the aspect (1) or (2) above, a distortion correction unit is further provided for correcting distortion contained in the two-dimensional actual image, and the image projection unit projects the projection range of the two-dimensional actual image whose distortion has been corrected onto the three-dimensional CAD model.

[0098] According to the above aspect (3), the projection range is depicted on the 3D CAD model based on the distortion-corrected 2D actual image, thereby enabling high-quality inspection even when the 2D actual image obtained by the imaging device contains distortion.

[0099] The order of the distortion correction process for the real 2D image and the process of excluding unclear areas (whiteout, out-of-focus, etc.) when specifying the projection range from the real 2D image is not limited to any particular order. That is, the distortion correction process may be performed on the real 2D image first, and the projection range may be specified from the real 2D image whose distortion has been corrected, or the distortion correction process may be performed on the projection range specified from the real 2D image.

[0100] (4) In another aspect, in the aspect (3), the distortion correction unit corrects the distortion based on a result of capturing a test image having a predetermined depiction pattern using the imaging device.

[0101] According to the above aspect (4), a test image having a predetermined pattern (for example, an image of a checkerboard) is captured using an imaging device, and distortion contained in the image is evaluated based on the pattern contained in the obtained image, thereby making it possible to perform correction to reduce the distortion.

[0102] (5) In another aspect, in the aspect (3) or (4), the imaging device has an imaging lens with an angle of view of 60 degrees or more.

[0103] According to the above aspect (5), the imaging device has a so-called wide-angle lens as an imaging lens, which has an angle of view of 60 degrees or more. A two-dimensional real image captured using a wide-angle lens has a wide imaging range and can include a relatively large inspection object. However, the two-dimensional real image captured using a wide-angle lens contains more distortion than an image captured using an imaging lens with a narrow angle of view. Therefore, there is a risk of reduced accuracy when performing an inspection based on the two-dimensional real image captured by the imaging device itself. However, by correcting the distortion contained in the two-dimensional real image using the distortion correction unit, it is possible to perform an inspection with high accuracy based on the two-dimensional real image captured using the wide-angle lens.

[0104] (6) In another aspect, in any one of the aspects (1) to (5) above, the projection range specification unit specifies, as the projection range, a predetermined range in the two-dimensional actual image that includes a focal position of the imaging device.

[0105] According to the above aspect (6), a predetermined range of the 2D actual image including the focal position of the imaging device (a predetermined range focused within the depth of field) is identified as the projection range and projected onto the 3D CAD model (on the other hand, a range of the 2D actual image not identified as the projection range is not depicted in the 3D CAD model). By appropriately identifying the projection range in this way, for example, by excluding unclear areas due to whiteout or out-of-focus caused by light reflection from the projection target onto the 3D CAD model, inspection accuracy can be improved. Note that the identification of the projection range in the 2D actual image may be performed manually by an operator or automatically by executing a predetermined algorithm.

[0106] (7) In another aspect, in the aspect (6) above, the projection range is identified as an area in which the depth deviation from a reference depth corresponding to the focus position is within an acceptable range based on a depth map estimated based on the two-dimensional actual image.

[0107] According to the above aspect (7), a projection range in a 2D actual image to be projected onto a 3D CAD model is identified based on a depth map estimated from the 2D actual image. This projection range is identified as a region of the depth map where a depth deviation from a reference depth corresponding to the focal position of the 2D actual image is within an allowable range. This allows regions that are unclear due to depth deviations outside the allowable range to be excluded from the projection onto the 3D CAD model, thereby making it possible to identify a projection range suitable for accurate inspection.

[0108] (8) In another aspect, in any one of the aspects (1) to (7) above, the inspection system further includes a defect detection unit for detecting defects in the inspection object contained in the two-dimensional actual image, a defect depiction unit for depicting the defects on a three-dimensional CAD model, and an inspection result creation unit for creating an inspection result including information about the three-dimensional CAD model onto which the projection range is projected and the defects depicted on the three-dimensional CAD model.

[0109] According to the above aspect (8), the three-dimensional CAD model onto which the projection range is projected and information on the defects depicted on the three-dimensional CAD model are created as the inspection results.

[0110] (9) In another aspect, in the aspect (8), the inspection results are created for each part predefined in the three-dimensional CAD model.

[0111] According to the above aspect (9), each part of the 3D CAD model is defined in advance, and inspection results are created for each part. This eliminates the need for the operator to memorize the boundary lines of each part of the inspection object, and effectively reduces the workload when obtaining inspection results such as the length and area of ​​defects for each part, without having to depict the boundary lines of each part on the inspection object.

[0112] (10) In another aspect, in any one of the aspects (1) to (9) above, the system includes: a coordinate transformation parameter estimation unit for estimating, based on the shape identified by the shape identification unit and the 3D CAD model of the object to be inspected, coordinate transformation parameters for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to the viewpoint of the imaging device that captured the 2D actual image; and a 3D CAD model modification unit for using the coordinate transformation parameters to change the position and direction of viewpoint information of the 3D CAD model of the object to be inspected, wherein the 2D simulated image extraction unit extracts the 2D simulated image from the 3D CAD model whose viewpoint information has been changed by the 3D CAD model modification unit.

[0113] According to the above aspect (10), based on the shape of the inspection object included in the 2D actual image and the 3D CAD model of the inspection object, coordinate transformation parameters are estimated for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to the viewpoint of the imaging device that captured the 2D actual image. The coordinate transformation parameters include, for example, a translation vector and a rotation matrix, and are parameters for transforming the first coordinate system into a second coordinate system, which is a 2D coordinate system. In other words, the coordinate transformation parameters are parameters for estimating the position and orientation that define the viewpoint information of the imaging device in the first coordinate system based on n reference points (n is an arbitrary natural number) represented by three-dimensional coordinates in the first coordinate system corresponding to the 3D CAD model handled in the 3D CG software and those reference points represented by two-dimensional coordinates in the second coordinate system corresponding to the 2D actual image. Using these coordinate transformation parameters, the position and / or orientation of the viewpoint information of the 3D CAD model is changed to correspond to the inspection object included in the 2D actual image. In this way, by using the coordinate transformation parameters to change the position and direction of the viewpoint information of the 3D CAD model so that it corresponds to the 2D actual image, it is possible to suppress deviations in position and direction between the two without requiring operator operation. Then, by depicting the point sequence coordinate data of the defect, which has been coordinate-transformed using the 2D simulated image extracted from the 3D CAD model whose position and direction of the viewpoint information has been changed in this way, on the 3D CAD model, it becomes possible to measure the defect on the 3D CAD model with high accuracy.

[0114] (11) An inspection support method according to one aspect includes the steps of: identifying a shape of an object to be inspected contained in a two-dimensional actual image based on a two-dimensional actual image obtained by capturing an image of the object to be inspected with an imaging device; estimating, based on the shape and a three-dimensional CAD model of the object to be inspected, coordinate transformation parameters for converting a first coordinate system corresponding to the three-dimensional CAD model into a second coordinate system corresponding to the viewpoint of the imaging device that captured the two-dimensional actual image; changing a position and direction of viewpoint information of the three-dimensional CAD model of the object to be inspected using the coordinate transformation parameters; extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model after the viewpoint information has been changed; identifying a projection range of the two-dimensional actual image; and projecting the projection range onto the three-dimensional CAD model by fitting the projection range to the two-dimensional simulated image.

[0115] According to the above aspect (11), a corresponding two-dimensional simulated image is extracted from a three-dimensional CAD model based on the shape of the inspection object identified from a two-dimensional actual image captured by an imaging device. The projection range identified from the two-dimensional actual image is projected onto the three-dimensional CAD model by coordinate transformation using the two-dimensional simulated image. The three-dimensional CAD model onto which the projection range is projected in this manner contains various information necessary for evaluating quality, such as defects and the surface texture of the inspection object, enabling high-quality inspection of the inspection object.

[0116] (12) An inspection assistance program according to one embodiment causes a computer device to execute the following steps: based on a two-dimensional real image obtained by capturing an image of the object to be inspected using an imaging device, identify a shape of the object to be inspected contained in the two-dimensional real image; based on the shape and a three-dimensional CAD model of the object to be inspected, estimate coordinate transformation parameters for converting a first coordinate system corresponding to the three-dimensional CAD model into a second coordinate system corresponding to the viewpoint of the imaging device that captured the two-dimensional real image; using the coordinate transformation parameters to change the position and direction of viewpoint information of the three-dimensional CAD model of the object to be inspected; extracting a two-dimensional simulated image corresponding to the two-dimensional real image from the three-dimensional CAD model after the viewpoint information has been changed; identifying a projection range of the two-dimensional real image; and projecting the projection range onto the three-dimensional CAD model by fitting the projection range to the two-dimensional simulated image.

[0117] According to the above aspect (12), a corresponding two-dimensional simulated image is extracted from a three-dimensional CAD model based on the shape of the inspection object identified from a two-dimensional actual image captured by an imaging device. The projection range identified from the two-dimensional actual image is projected onto the three-dimensional CAD model by coordinate transformation using the two-dimensional simulated image. The three-dimensional CAD model onto which the projection range is projected in this manner contains various information necessary for evaluating quality, such as defects and the surface texture of the inspection object, enabling high-quality inspection of the inspection object.

[0118] REFERENCE SIGNS LIST 1 Inspection support system 2 Inspection object 4 Communication network 6 Client terminal 8 Server 10 Communication unit 11 Memory unit 12 Output unit 13 Input unit 14 Calculation unit 15 Communication unit 16 Memory unit 18 Calculation unit 30 Image acquisition unit 31 Distortion correction unit 32 Depiction range specification unit 33 Shape identification unit 34 Defect detection unit 36 ​​Report creation unit 40 Coordinate transformation parameter estimation unit 41 Three-dimensional model modification unit 42 Image extraction unit 44 Depiction unit 50 Imaging device

Claims

1. An inspection support system comprising: a shape identification unit for identifying a shape of an inspection object contained in a two-dimensional actual image obtained by imaging the inspection object with an imaging device, based on the two-dimensional actual image; a two-dimensional simulated image extraction unit for extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from a three-dimensional CAD model of the inspection object, based on the shape identified by the shape identification unit; a projection range identification unit for identifying a projection range of the two-dimensional actual image; and an image projection unit for projecting the projection range onto the three-dimensional CAD model by fitting the projection range to the two-dimensional simulated image.

2. The inspection support system of claim 1, wherein the image projection unit projects the projection ranges of multiple two-dimensional actual images obtained by imaging the inspection object from different positions using the imaging device onto the three-dimensional CAD model.

3. An inspection support system as described in claim 1 or 2, further comprising a distortion correction unit for correcting distortion contained in the two-dimensional actual image, and the image projection unit projects the projection range of the two-dimensional actual image with the distortion corrected onto the three-dimensional CAD model.

4. An inspection support system as described in claim 3, wherein the distortion correction unit corrects the distortion based on the result of capturing a test image having a predetermined depiction pattern using the imaging device.

5. The inspection support system according to claim 3, wherein the imaging device has an imaging lens with an angle of view of 60 degrees or more.

6. An inspection support system as described in claim 1 or 2, wherein the projection range determination unit determines a predetermined range of the two-dimensional actual image that includes a focal position of the imaging device as the projection range.

7. The inspection support system of claim 6, wherein the projection range is identified as a region in which the depth deviation from a reference depth corresponding to the focus position is within an acceptable range based on a depth map estimated based on the two-dimensional actual image.

8. An inspection support system as described in claim 1 or 2, further comprising: a defect detection unit for detecting defects in the inspection object contained in the two-dimensional actual image; a defect depiction unit for depicting the defects on a three-dimensional CAD model; and an inspection result creation unit for creating an inspection result including the three-dimensional CAD model onto which the projection range is projected, and information regarding the defects depicted on the three-dimensional CAD model.

9. The inspection support system according to claim 8, wherein the inspection results are generated for each part predefined in the three-dimensional CAD model.

10. An inspection support system as described in claim 1 or 2, comprising: a coordinate transformation parameter estimation unit for estimating coordinate transformation parameters for transforming a first coordinate system corresponding to the three-dimensional CAD model into a second coordinate system corresponding to the viewpoint of the imaging device that captured the two-dimensional actual image based on the shape identified by the shape identification unit and the three-dimensional CAD model of the inspection object; and a three-dimensional CAD model modification unit for changing the position and direction of viewpoint information of the three-dimensional CAD model of the inspection object using the coordinate transformation parameters, wherein the two-dimensional simulated image extraction unit extracts the two-dimensional simulated image from the three-dimensional CAD model whose viewpoint information has been changed by the three-dimensional CAD model modification unit.

11. An inspection support method comprising the steps of: identifying a shape of an inspection object contained in a two-dimensional actual image obtained by imaging the inspection object with an imaging device, based on the two-dimensional actual image; extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from a three-dimensional CAD model of the inspection object based on the shape; identifying a projection range of the two-dimensional actual image; and projecting the projection range onto the three-dimensional CAD model by fitting the projection range to the two-dimensional simulated image.

12. An inspection support program for causing a computer device to execute the steps of: identifying the shape of an inspection object contained in a two-dimensional actual image based on a two-dimensional actual image obtained by capturing an image of the inspection object with an imaging device; extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from a three-dimensional CAD model of the inspection object based on the shape; identifying a projection range of the two-dimensional actual image; and projecting the projection range onto the three-dimensional CAD model by fitting the projection range to the two-dimensional simulated image.

Images