Inspection support system, inspection support method, and inspection support program

Abstract

An inspection assistance system according to the present invention comprises a shape recognition unit, a defect detection unit, a coordinate transformation parameter estimation unit, a 3D CAD model modification unit, a 2D simulated image extraction unit, and a depiction unit. The shape recognition unit recognizes the shape of an inspection target on the basis of a 2D actual image captured by an imaging device. The defect detection unit detects defects, of the inspection target, included in the 2D actual image. On the basis of the recognized shape and the 3D CAD model, the coordinate transformation parameter estimation unit estimates a coordinate transformation parameter for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to the viewpoint of the imaging device that captured the 2D actual image. The 3D CAD model modification unit uses the coordinate transformation parameter to modify the position and direction of viewpoint information of the 3D CAD model. The 2D simulated image extraction unit extracts a 2D simulated image from the 3D CAD model after the viewpoint information has been modified. The depiction unit depicts a defect image on the 3D CAD model by matching the 2D actual image including the defect image to the 2D simulated image.

Description

[Technical Field]

[0001] The present disclosure relates to an inspection support system, an inspection support method, and an inspection support program. This application claims priority based on Japanese Patent Application No. 2022-051343, filed with the Japan Patent Office on March 28, 2022, the contents of which are incorporated herein by reference. [Background technology]

[0002] When inspectors visually inspect objects (including products, machine parts, and intermediate products in the manufacturing process), determine the dimensions of defects, and enter the information in a report, it is time-consuming and accuracy varies depending on the ability of the inspector.As a means of solving this problem, technology has been proposed in recent years that enables the determination of defect dimensions by analyzing two-dimensional actual images obtained by capturing images of the object to be inspected with an imaging device such as a camera.

[0003] For example, Patent Document 1 discloses a method for supporting inspection work by analyzing a two-dimensional actual image obtained by capturing an image of an inspection object for which a three-dimensional CAD model exists in advance, thereby determining the dimensions of defects in the inspection object. In this document, the shape of the inspection object contained in the two-dimensional actual image is identified, and a reference portion contained in the shape is compared with a reference portion contained in a two-dimensional simulated image extracted from a three-dimensional CAD model corresponding to the inspection object, thereby depicting the defects contained in the two-dimensional actual image on the three-dimensional CAD model. By depicting the defects on the three-dimensional CAD model in this way, it becomes possible to determine the dimensions of the defects on the three-dimensional CAD model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-21669 Summary of the Invention [Problem to be solved by the invention]

[0005] In the above-mentioned Patent Document 1, in order to depict defects on a 3D CAD model, coordinate transformation is performed by comparing a reference portion identified from the shape of the inspection object contained in a 2D actual image with a reference portion on the 3D CAD model. To perform such coordinate transformation with high accuracy, the position and orientation of the inspection object contained in the 2D actual image obtained by the imaging device must be aligned with the position and orientation set in the 3D CAD model, resulting in low flexibility. If the position and orientation of the inspection object contained in the 2D actual image differ significantly from the position and orientation set in the 3D CAD model, the position of the defect depicted in the 3D CAD model will be shifted, resulting in reduced accuracy in determining dimensions, etc.

[0006] At least one embodiment of the present disclosure has been made in consideration of the above-mentioned circumstances, and aims to provide an inspection support system, an inspection support method, and an inspection support program that can support the implementation of inspections that can derive the three-dimensional position and dimensions of defects in an inspection object with simple operations. [Means for solving the problem]

[0007] In order to solve the above problem, an inspection support system according to at least one embodiment of the present disclosure includes: a shape identification unit for identifying a shape of the inspection object included in a two-dimensional actual image based on the two-dimensional actual image obtained by capturing an image of the inspection object with an imaging device; a defect detection unit for detecting defects of the inspection object included in the two-dimensional actual image; a coordinate transformation parameter estimation unit for estimating, based on the shape identified by the shape identification unit and a 3D CAD model of the inspection object, coordinate transformation parameters for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to a viewpoint of the imaging device that captured the 2D actual image; a three-dimensional CAD model modification unit for modifying a position and a direction of viewpoint information of a three-dimensional CAD model of the object to be inspected using the coordinate transformation parameters; a two-dimensional simulated image extracting unit for extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model after the information has been changed; a depiction unit for depicting the defect image on the three-dimensional CAD model by fitting the two-dimensional actual image, including a defect image showing the defect detected by the defect detection unit, to the two-dimensional simulated image; Equipped with.

[0008] In order to solve the above problem, an inspection support method according to at least one embodiment of the present disclosure includes: a step of identifying a shape of the inspection object included in a two-dimensional actual image based on the two-dimensional actual image obtained by capturing an image of the inspection object with an imaging device; detecting defects in the inspection object included in the two-dimensional actual image; a step of estimating coordinate transformation parameters for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to a viewpoint of the imaging device that captured the 2D actual image, based on the shape and the 3D CAD model of the inspection object; changing the position and orientation of viewpoint information of the 3D CAD model of the object to be inspected using the coordinate transformation parameters; extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model after the viewpoint information has been changed; depicting the defect image on the three-dimensional CAD model by fitting the two-dimensional actual image, including a defect image showing the defect, to the two-dimensional simulated image; Equipped with.

[0009] In order to solve the above problem, an inspection assistance program according to at least one embodiment of the present disclosure includes: On the computer, a step of identifying a shape of the inspection object included in a two-dimensional actual image based on the two-dimensional actual image obtained by capturing an image of the inspection object with an imaging device; detecting defects in the inspection object included in the two-dimensional actual image; a step of estimating coordinate transformation parameters for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to a viewpoint of the imaging device that captured the 2D actual image, based on the shape and the 3D CAD model of the inspection object; changing the position and orientation of viewpoint information of the 3D CAD model of the object to be inspected using the coordinate transformation parameters; extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model after the viewpoint information has been changed; depicting the defect image on the three-dimensional CAD model by fitting the two-dimensional actual image, including a defect image showing the defect, to the two-dimensional simulated image; Execute the following. [Effects of the Invention]

[0010] According to at least one embodiment of the present disclosure, an inspection support system, an inspection support method, and an inspection support program can be provided that can support the implementation of inspections that can derive the three-dimensional position and dimensions of defects in an inspection object with simple operations. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration of an inspection support system according to an embodiment; [Figure 2] FIG. 2 is a schematic diagram showing the hardware configuration of the client terminal and the server in FIG. [Figure 3] 3 is a conceptual diagram for explaining a method for detecting a reference portion by the shape identification unit of FIG. 1. FIG. [Figure 4] 10 is a diagram conceptually showing the correspondence between the first coordinate system and the second coordinate system based on coordinate transformation parameters estimated by PNP. FIG. [Figure 5] 1 is a flowchart illustrating an inspection support method according to an embodiment. [Figure 6] 1 is an example of a two-dimensional actual image captured by an imaging device. [Figure 7]FIG. 2 is a schematic diagram showing a default posture of a three-dimensional CAD model. [Figure 8] 8 is a schematic diagram showing the posture of the three-dimensional CAD model of FIG. 7 after at least one of the position and the direction has been changed by a three-dimensional CAD model change unit. FIG. [Figure 9] FIG. 10 is a block diagram showing a schematic configuration of an inspection support system according to another embodiment. [Figure 10] FIG. 10 is a schematic diagram of the coordinate transformation parameter correction unit of FIG. 9 configured as a variational autoencoder. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, several embodiments of the present disclosure will be described with reference to the accompanying drawings. However, the dimensions, materials, shapes, relative arrangements, etc. of components described as embodiments or shown in the drawings are merely illustrative examples and are not intended to limit the scope of the present disclosure.

[0013] FIG. 1 is a block diagram showing a schematic configuration of an inspection support system 1 according to one embodiment. The inspection support system 1 is a system for supporting the detection of defects in an inspection object 2 and the creation of a report on the inspection work when inspection work is performed on the inspection object 2. The user of the inspection support system 1 may be a worker who performs the inspection work, a user who uses the inspection object 2, or any other third party. The defects detected by the inspection support system 1 are defects that can be identified from the appearance, and the types thereof include, for example, cracks, dents, scratches, poor coating, oxidation, thinning, dirt, etc.

[0014] The inspection object 2 may be the entire product, or may be a component (e.g., a machine part) that constitutes the product. The inspection object 2 may be a new product, a repaired product, or an existing facility. The inspection object 2 may be, for example, a moving blade, a stationary blade, a ring segment, a combustor, or the like of a gas turbine.

[0015] The inspection support system 1 is configured to include at least one computer device. The inspection support system 1 may be configured as a single device, but in Fig. 1 it is configured to include a client terminal 6 and a server 8, which are communication terminals that can communicate with each other via a communication network 4, and the client terminal 6 and the server 8 work together to realize the functions of the inspection support system 1. The communication network 4 may be a WAN (World Area Network) or a LAN (Local Area Network), and may be either wireless or wired.

[0016] Fig. 2 is a schematic diagram showing the hardware configuration of the client terminal 6 and server 8 in Fig. 1. The client terminal 6 includes a communication unit 10 for communicating with the server 8, a memory unit 11 for storing various data, an output unit 12 for outputting various information, an input unit 13 for accepting user input, and a calculation unit 14 for performing various calculations. The server 8 includes a communication unit 15 for communicating with the client terminal 6, a memory unit 16 for storing various data, and a calculation unit 18 for performing various calculations. The internal configurations of the client terminal 6 and the server 8 are connected to each other via a bus line.

[0017] The communication units 10 and 15 are communication interfaces that include a network interface card (NIC) for wired or wireless communication, and enable communication between the client terminal 6 and the server 8.

[0018] The memory units 11 and 16 are composed of RAM (Random Access Memory), ROM (Read Only Memory), etc., and store programs (e.g., inspection support programs, trained models described below) for executing various control processes performed by the client terminal 6 and the server 8, respectively, and data necessary for the various control processes.

[0019] The various data include 3D CAD models of multiple objects. In this embodiment, the 3D CAD models are stored in the storage unit 16 of the server 8. Since 3D CAD models of multiple objects generally require a large storage capacity, storing the models in the storage unit 16 on the server 8 side can prevent the storage capacity on the client terminal 6 side from being overwhelmed. In this case, the client terminal 6 can be realized as a small terminal or a portable terminal such as a laptop computer, which is effective in improving convenience.

[0020] The multiple objects include the aforementioned inspection object 2. The 3D CAD model is 3D CAD data that shows the object as a mesh image in a 3D virtual space with actual dimensions. The mesh image can be rotated, enlarged, and reduced, and the 3D CAD model is configured so that a 2D simulated image can be extracted from any viewpoint. The storage units 11 and 16 may be configured by a single storage device or by multiple storage devices. The storage units 11 and 16 may also be external storage devices.

[0021] The output unit 12 is configured by an output device such as a display device, a speaker device, etc. The output unit 12 is an output interface for presenting various information to the user.

[0022] The input unit 13 is an input interface for inputting information required to perform various processes from the outside, and is composed of input devices such as operation buttons, a keyboard, a pointing device, a microphone, etc. Such information includes, in addition to instructions from a user, data on a two-dimensional actual image acquired by an imaging device, as will be described later.

[0023] The calculation units 14 and 18 are configured by processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The calculation units 14 and 18 execute programs stored in the storage units 11 and 16 to control the operation of the entire system.

[0024] Next, a specific description will be given of the functional configurations of the client terminal 6 and the server 8 that constitute the inspection support system 1. As shown in Fig. 1, the client terminal 6 includes an image acquisition unit 30, a shape identification unit 32, a defect detection unit 34, and a report creation unit 36. The server 8 also includes a coordinate transformation parameter estimation unit 40, a 3D CAD model modification unit 41, an image extraction unit 42, and a depiction unit 44.

[0025] In this embodiment, the case where these functional configurations are arranged across the client terminal 6 and the server 8 is illustrated, but they may be arranged in either the client terminal 6 or the server 8. For example, by arranging the configuration shown on the server 8 side in FIG. 1 in the client terminal 6, this embodiment may be realized by the client terminal 6 alone without using the server 8. Furthermore, the functional configurations provided in either the client terminal 6 or the server 8 may be appropriately arranged in the other. In this way, the layout of the functional configurations provided in the inspection support system 1 can be changed as appropriate depending on the application.

[0026] The image acquisition unit 30 acquires a two-dimensional real image of the inspection object 2 captured by the imaging device 50. The imaging device 50 is, for example, a camera compatible with visible light, and is configured to acquire a two-dimensional real image, which is an image captured of the inspection object 2. The acquisition of the two-dimensional real image by the imaging device 50 may be performed at the same location as the inspection support system 1, particularly the client terminal 6 to which the two-dimensional real image is input, or may be performed at a different location (remote location). Data related to such two-dimensional real image is acquired by inputting it to the input unit 13 of the client terminal 6.

[0027] The shape identification unit 32 identifies the two-dimensional shape of the inspection object 2 in the two-dimensional actual image acquired by the image acquisition unit 30. The shape identification unit 32 may be configured to identify the shape of the inspection object 2 included in the two-dimensional actual image by detecting a plurality of reference portions of the inspection object 2 in the two-dimensional actual image.

[0028] Here, Fig. 3 is a conceptual diagram for explaining a method for detecting a reference portion by the shape identification unit 32 in Fig. 1. Note that Fig. 3 illustrates a case where the inspection object 2 has a triangular shape when viewed from the front, but the shape is not limited thereto.

[0029] First, the shape identification unit 32 is configured to detect a characteristic part of the inspection target object 2 as a reference part. In one embodiment, the characteristic part is a corner. The characteristic part may be any part that is useful for identifying the shape, and may be, for example, a landmark mark, trademark, keyhole, button, etc.

[0030] In one embodiment, a method for detecting a reference portion uses machine learning (e.g., SSD (Single Shot Multibox Detector) technology). SSD technology is known as a method for detecting objects in an image using deep learning. Specifically, as shown in FIG. 3, the shape identification unit 32 searches a two-dimensional actual image P1 using a default box B0 and detects a corner of the inspection object 2 as a reference portion. The shape of the reference portion is identified based on the RGB values ​​indicating the red, green, and blue components of each pixel, the hue compared to other pixels, and brightness information.

[0031] As a result, default boxes B1, B2, and B3 corresponding to the three corners are detected as the reference parts. Furthermore, by including brightness information in the input, the shape identification unit 32 can detect the reference parts even when, for example, there is some brightness variation due to a portion of the actual 2D image P1 not being blurred. Detection of the reference part based on such brightness information can be performed using, for example, an estimation model constructed by machine learning using images with brightness variations as training data, thereby enabling detection of the reference part from the actual 2D image P1 that includes some brightness variation.

[0032] In this way, the shape identification unit 32 detects the reference part based on RGB values, color tone, and brightness information, thereby reducing detection errors due to brightness fluctuations. Furthermore, when identifying the shape of the reference part using deep learning machine learning, it is possible to automate the operator's work and reduce the work time compared to when the operator identifies the reference part based on subjective judgment.

[0033] In other embodiments, other machine learning techniques such as Region Based Convolutional Neural Networks (R-CNN) and You Only Look Once (YOLO) can be used to detect the reference portion.

[0034] The defect detection unit 34 detects defects in the inspection object 2 contained in the two-dimensional actual image. The defect detection unit 34 may detect defects in the inspection object 2 using a trained model that has been machine-learned to understand the relationship between the two-dimensional actual images of each of a plurality of objects including the inspection object 2 and images of defects that may occur in the plurality of objects. For example, the defect detection unit 34 may be configured to analyze the RGB values ​​of each pixel in the two-dimensional actual image and determine defects by pattern classification of the contrast and color tone.

[0035] The coordinate transformation parameter estimation unit 40 estimates coordinate transformation parameters for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to the 2D actual image, based on the shape identified by the shape identification unit 32 and the 3D CAD model of the inspection target object 2. The coordinate transformation parameters are parameters for transforming coordinate points in the first coordinate system and the second coordinate system, and one of the estimation methods is the Perspective-Points Problem (PNP). In other words, the coordinate transformation parameters are parameters for estimating the position and orientation that define viewpoint information of the imaging device in the first coordinate system, based on n-point (n is any natural number) reference points represented by three-dimensional coordinates in the first coordinate system corresponding to the 3D CAD model handled on the 3D CG software, and those reference points represented by two-dimensional coordinates in the second coordinate system corresponding to the 2D actual image.

[0036] 4 is a diagram conceptually showing the correspondence between the first coordinate system and the second coordinate system based on the coordinate transformation parameters estimated by PNP. In PNP, coordinate transformation parameters (such as a translation vector or a rotation matrix) are estimated as parameters for converting between the three-dimensional coordinates (X1, X2, . . .) of n points in the first coordinate system, which is a world coordinate system corresponding to a three-dimensional CAD model handled on 3D graphics software, and the two-dimensional coordinates (x1, x2, . . .) of n points in the second coordinate system corresponding to the viewpoint of the imaging device 50 that captured the two-dimensional actual image including those points.

[0037] The estimation of the coordinate transformation parameters in the PNP is performed, for example, so that corresponding reference portions in the first coordinate system and the second coordinate system coincide with each other. Specifically, the coordinate transformation parameter estimation unit 40 identifies multiple first reference portions (multiple coordinate points in the second coordinate system) of the inspection object 2 in the two-dimensional actual image based on the shape identified by the shape identification unit 32, for example, by automatic detection using machine learning or manual input by a user, and also pre-registers multiple second reference portions (multiple coordinate points in the first coordinate system) in the three-dimensional CAD model. The second reference portions are registered by pre-registering the second reference portions in a database or the like. Such registration of the second reference portions may be performed, for example, by displaying the three-dimensional CAD model on a screen and having an operator specify the second reference portions on the screen using a cursor, pointer, or the like. Then, the coordinate transformation parameters are estimated so that the first reference portions coincide with the second reference portions.

[0038] The coordinate transformation parameters include external parameters of the image capture device 50. The external parameters are parameters necessary to define the position and orientation (6 degrees of freedom) of the image capture device 50 in the first coordinate system. In other words, the external parameters are parameters for reproducing, in the first coordinate system, the same location as the image capture position of the two-dimensional actual image, rather than a relative position, and are expressed by, for example, a translation vector or a rotation matrix.

[0039] The coordinate transformation parameters may also include internal parameters of the image capture device 50. The internal parameters of the image capture device 50 are parameters related to the main body (lens) of the image capture device 50, such as the focal length f and the optical center. The optical center corresponds to the origin of the second coordinate system, is a parameter specific to the lens, and is expressed as two-dimensional coordinates (Cu, Cv). In this case, the relational expression between the three-dimensional coordinates (Xw, Yw, Zw) of the feature point in the first coordinate system corresponding to the 3D CAD model and the two-dimensional coordinates (u, v) of the feature point in the second coordinate system corresponding to the actual 2D image is expressed as follows: TIFF0007728442000001.tif19170

[0040] 1, the 3D CAD model modification unit 41 uses the coordinate transformation parameters estimated by the coordinate transformation parameter estimation unit 40 to modify the position and direction of the viewpoint information of the 3D CAD model of the inspection target 2. As a result, the viewpoint information of the 3D CAD model shown from the viewpoint of a camera located at a specific position and direction on the drawing software for the 3D CAD model is modified to correspond to the position and direction corresponding to the viewpoint of the imaging device 50 that captured the 2D actual image.

[0041] Based on the identification result by the shape identification unit 32, the image extraction unit 42 refers to the three-dimensional CAD model whose viewpoint information has been changed by the three-dimensional CAD model change unit 41, and extracts a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model.

[0042] The rendering unit 44 adjusts the defect image showing the defect detected by the defect detection unit 34 so that it matches the two-dimensional simulated image, and renders the adjusted defect image on the three-dimensional CAD model. This adjustment may be performed before rendering or during rendering, or may be performed on the already rendered defect image after rendering. The three-dimensional CAD model with the rendered defect image may be displayed on the display device of the output unit 12.

[0043] For example, the depiction unit 44 uses a geometric transformation (plane transformation), such as an affine transformation, to transform the two-dimensional actual image into the coordinate system of the two-dimensional simulated image using a reference portion (multiple coordinate points) of the target image as input values, and projects the image onto the three-dimensional CAD model. At this time, the viewpoint information of the three-dimensional CAD model is changed as described above, so the depiction unit 44 defines the defect according to the transformed coordinates and projects it to depict it on the three-dimensional CAD model.

[0044] For example, the rendering unit 44 may compare the lengths of the sides of the actual two-dimensional image and the simulated two-dimensional image, which are determined based on the positional relationships of multiple reference portions, and enlarge or reduce the defect image based on the similarity ratio between them. The rendering unit 44 may also compare the positional relationships of multiple reference portions of the actual two-dimensional image and the simulated two-dimensional image, and perform adjustment of the aspect ratio of the defect image, translation, linear transformation, etc.

[0045] In this embodiment, the case where the depiction unit 44 transforms the defect image is exemplified, but other examples of transformations that may be used include affine transformation, projective transformation, similarity transformation, inversion transformation, perspective transformation, and the like.

[0046] The report creation unit 36 ​​derives the three-dimensional positions and dimensions of defects in the inspection object 2 from the dimensional data of the three-dimensional CAD model, and creates a report including the results of deriving the positions and dimensions. The created report may be stored in the storage unit 11, or may be transmitted to another device (for example, a server device that manages reports).

[0047] Next, a description will be given of an inspection support method implemented by the inspection support system 1 having the above configuration. Fig. 5 is a flowchart showing an inspection support method according to one embodiment.

[0048] First, as a preliminary step before the inspection support method is carried out, an image of the inspection object 2 is captured using the imaging device 50 (step S1). In step S1, an image of the inspection object 2 can be captured from any position and direction, and the two-dimensional actual image obtained by the imaging device 50 is input to the client terminal 6 as data.

[0049] In the client terminal 6, the image acquisition unit 30 acquires a two-dimensional actual image as data input from the imaging device 50 (step S2). Next, in the client terminal 6, the shape identification unit 32 detects a reference portion of the inspection object 2 in the two-dimensional actual image acquired in step S2, and identifies the shape of the inspection object 2 (step S3).

[0050] Next, the coordinate transformation parameter estimation unit 40 accesses the learning model for the reference part detected when identifying the shape in step S3 (step S4) and estimates the coordinate transformation parameters (step S5). The method for estimating the coordinate transformation parameters is performed by, for example, PNP, as specifically described above.

[0051] Next, the three-dimensional CAD model modification unit 41 modifies at least one of the position and orientation of the three-dimensional CAD model using the coordinate transformation parameters (for example, translation vectors, rotation matrices, etc.) estimated in step S5 (step S6).

[0052] Here, the processing by the three-dimensional CAD model modification unit 41 will be specifically described with reference to Fig. 6 to Fig. 8. Fig. 6 is an example of a two-dimensional actual image captured by the imaging device 50, Fig. 7 is a schematic diagram showing the default posture of the three-dimensional CAD model, and Fig. 8 is a schematic diagram showing the posture of the three-dimensional CAD model of Fig. 7 after at least one of the position and the direction has been changed by the three-dimensional CAD model modification unit 41.

[0053] As shown in Fig. 6, the two-dimensional actual image captured by the imaging device 50 captures C1 to C7 out of the multiple reference portions C1 to C8 of the inspection object 2. On the other hand, in the default posture of the three-dimensional CAD model shown in Fig. 7, only some of the multiple reference portions of the inspection object 2 shown in Fig. 6 can be confirmed (specifically, in Fig. 7, the reference portions C2 to C8 that are common to Fig. 6 are visible, but the reference portion C1 of Fig. 6 is not visible). As shown in Fig. 8, the three-dimensional CAD model having such a default posture is brought into a posture corresponding to the two-dimensional actual image shown in Fig. 6 by changing at least one of the position and orientation of the three-dimensional CAD model having the default posture shown in Fig. 6 using coordinate transformation parameters, so that all of the reference portions C1 to C7 are included.

[0054] Next, the image extraction unit 42 of the server 8 extracts a two-dimensional simulated image including portions corresponding to the multiple reference portions of the inspection object 2 detected from the two-dimensional actual image from the three-dimensional CAD model whose viewpoint information has been changed by the three-dimensional CAD model change unit 41 (step S7). In step S7, for example, the two-dimensional simulated image may be extracted by displaying the three-dimensional CAD model whose viewpoint information has been changed on a screen and taking a screenshot of the screen. In this case, the screenshot may be taken by actually displaying the three-dimensional CAD model whose viewpoint information has been changed on a screen, or may be taken computationally without displaying it on a screen. Rendering or a trained model may be used for this extraction.

[0055] Next, the defect detection unit 34 of the client terminal 6 instructs the server 8 to detect defects in the inspection object 3 based on the two-dimensional actual image acquired in step S2 (step S8). The server 8, which has received the detection instruction, accesses a learning model for defect detection prepared in advance (step S9) and executes defect detection using the learning model (step S10).

[0056] The client terminal 6 acquires the detection result of step S10 from the server 8, and determines whether or not there is a defect in the inspection object 2 based on the detection result (step S11). If it is determined that there is no defect in the inspection object 2 (step S11: NO), steps S12 to S14 are skipped, and the report creation unit 36 ​​creates a report indicating that there is no defect (step S15).

[0057] On the other hand, if it is determined that there is no defect in the inspection object 2 (step S11: NO), the depiction unit 44 of the server 8 adapts the 2D actual image (i.e., the defect image) showing the defect detected in step S10 to fit the 2D simulated image (step S12), and depicts the adjusted defect image on the 3D CAD model (step S13).The client terminal 6 then acquires data (e.g., dimensional data) related to the 3D CAD model in which the defect image is depicted (step S14), and the report creation unit 36 ​​derives the 3D position and dimensions of the defect in the inspection object 2 from the data, and creates a report including the derived results (step S15).

[0058] 9 is a block diagram showing a schematic configuration of an inspection support system 1' according to another embodiment. Compared to the above-described embodiment, the inspection support system 1' further includes a coordinate transformation parameter correction unit 46. The coordinate transformation parameter correction unit 46 corrects the coordinate transformation parameters estimated by the coordinate transformation parameter estimation unit 40 by removing noise using machine learning.

[0059] As described above, the coordinate transformation parameter estimation unit 40 estimates coordinate transformation parameters so that the reference portion identified in the actual 2D image coincides with the reference portion identified on the 3D CAD model. Here, the detection of each reference portion for the coordinate transformation parameter estimation unit 40 is performed, for example, by measurement using image analysis in the server 8, manual specification by an operator, or machine learning using a machine learning model. Therefore, the estimated coordinate transformation parameters may contain errors due to measurement errors in the image analysis, human error during manual input by the operator, or uncertainty in the machine learning model. In this embodiment, the coordinate transformation parameter correction unit 46 is provided, and the coordinate transformation parameters are corrected to reduce such errors. The coordinate transformation parameter correction unit 46 can correct the coordinate transformation parameters by, for example, noise removal using machine learning.

[0060] FIG. 10 is a schematic diagram of the coordinate transformation parameter correction unit 46 of FIG. 9, configured as a variational autoencoder. In FIG. 10, the coordinate transformation parameter correction unit 46 includes an encoder 46a and a decoder 46b, which are a type of neural network. The encoder 46a receives coordinate transformation parameters as input and performs dimensionality reduction. The output from the encoder has its mean and variance identified, and is corrected so that the distribution defined by these becomes a standard normal distribution. The correction results are then restored by the decoder 46b to become corrected coordinate transformation parameters with the original dimensions. In this coordinate transformation parameter correction unit 46, abnormalities contained in the input coordinate transformation parameters before correction are not reproduced and the parameters are corrected to normal values ​​and output.

[0061] In the above embodiment, the coordinate transformation parameter correction unit 46 uses variational autoencoders (VAEs), but other techniques such as generative adversarial networks (GANs), principal component analysis, k-means clustering, and vector quantization (VQ) may also be used.

[0062] In this way, the inspection support system 1' includes the coordinate transformation parameter correction unit 46, which can reduce errors in the coordinate transformation parameters due to measurement errors in image analysis, human errors during manual input by an operator, or uncertainties in the machine learning model. As a result, defect images can be accurately depicted in the 3D CAD model.

[0063] As described above, the shape of the inspection object contained in the 2D actual image used to estimate the coordinate transformation parameters is identified, for example, by image analysis or manual input by an operator, and therefore these involve some errors. According to the above aspect (7), by using noise removal through machine learning to correct the estimated coordinate transformation parameters, it is possible to reduce the influence of such errors and effectively improve the accuracy of coordinate transformation using the coordinate transformation parameters.

[0064] As described above, according to each of the above embodiments, based on the shape of the inspection object included in the 2D actual image and the 3D CAD model of the inspection object, coordinate transformation parameters are estimated for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to the viewpoint of the imaging device that captured the 2D actual image. Using these coordinate transformation parameters, the position and orientation of the viewpoint information of the 3D CAD model are changed so that it corresponds to the inspection object included in the 2D actual image. By changing the position and orientation of the 3D CAD model using the coordinate transformation parameters in this way so that it corresponds to the 2D actual image, it is possible to suppress misalignment between the positions and orientations of the two models without requiring operator operation. Then, by depicting the defect image included in the 2D actual image on the 3D CAD model whose position and orientation have been changed in this way, it is possible to accurately measure the defect on the 3D CAD model.

[0065] In addition, within the scope of the present disclosure, the components in the above-described embodiments may be replaced with well-known components as appropriate, and the above-described embodiments may be combined as appropriate.

[0066] The contents described in each of the above embodiments can be understood, for example, as follows.

[0067] (1) An inspection support system according to one aspect includes: a shape identification unit for identifying a shape of the inspection object included in a two-dimensional actual image based on the two-dimensional actual image obtained by capturing an image of the inspection object with an imaging device; a defect detection unit for detecting defects of the inspection object included in the two-dimensional actual image; a coordinate transformation parameter estimation unit for estimating, based on the shape identified by the shape identification unit and a 3D CAD model of the inspection object, coordinate transformation parameters for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to a viewpoint of the imaging device that captured the 2D actual image; a three-dimensional CAD model modification unit for modifying a position and a direction of viewpoint information of a three-dimensional CAD model of the object to be inspected using the coordinate transformation parameters; a two-dimensional simulated image extracting unit for extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model after the viewpoint information has been changed; a depiction unit for depicting the defect image on the three-dimensional CAD model by fitting the two-dimensional actual image, including a defect image showing the defect detected by the defect detection unit, to the two-dimensional simulated image; Equipped with.

[0068] According to the above aspect (1), based on the shape of the inspection object included in the 2D actual image and the 3D CAD model of the inspection object, coordinate transformation parameters are estimated for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to the viewpoint of the imaging device that captured the 2D actual image. The coordinate transformation parameters include, for example, a translation vector and a rotation matrix, and are parameters for transforming the first coordinate system into the second coordinate system, which is a 2D coordinate system. In other words, the coordinate transformation parameters are parameters for estimating the position and orientation that define the viewpoint information of the imaging device in the first coordinate system based on n (n is any natural number) reference points expressed in three-dimensional coordinates in the first coordinate system corresponding to the 3D CAD model handled in the 3D CG software and those reference points expressed in two-dimensional coordinates in the second coordinate system corresponding to the 2D actual image. Using these coordinate transformation parameters, the position and / or orientation of the viewpoint information of the 3D CAD model is changed to correspond to the inspection object included in the 2D actual image. In this way, by using the coordinate transformation parameters to change the position and direction of the viewpoint information of the 3D CAD model so that it corresponds to the 2D actual image, it is possible to suppress deviations in position and direction between them without requiring operator operation. Then, by depicting the defect image contained in the 2D actual image on the 3D CAD model whose position and direction of the viewpoint information have been changed in this way, it becomes possible to measure defects on the 3D CAD model with high accuracy.

[0069] (2) In another embodiment, in the above embodiment (1), The coordinate transformation parameter estimation unit pre-identifies a plurality of first reference parts of the object to be inspected in the two-dimensional actual image based on the shape identified by the shape identification unit, and estimates the coordinate transformation parameters so that a plurality of second reference parts pre-registered in the three-dimensional CAD model match the plurality of first reference parts.

[0070] According to the above aspect (2), the coordinate transformation parameters are estimated so that a first reference portion, which is pre-specified for the inspection object included in the actual two-dimensional image, matches a second reference portion, which is pre-registered in the three-dimensional CAD model. By using the coordinate transformation parameters estimated in this manner, the position and orientation of the viewpoint information of the three-dimensional CAD model can be aligned with the position and orientation of the inspection object included in the actual two-dimensional image, and deviations in the positions and orientations of defects depicted in the three-dimensional CAD model can be effectively suppressed.

[0071] (3) In another aspect, in the above aspect (1) or (2), The coordinate transformation parameters include external parameters for defining the position and orientation of the image capture device in the first coordinate system.

[0072] According to the above aspect (3), by including external parameters in the coordinate transformation parameters, it is possible to suitably transform the first coordinate system corresponding to the three-dimensional CAD model into the second coordinate system corresponding to the two-dimensional actual image.

[0073] (4) In another embodiment, in the above embodiment (3), The coordinate transformation parameters further include internal parameters related to the imaging device.

[0074] According to the above aspect (4), the coordinate transformation parameters include internal parameters that are specific to the imaging device, such as the focal length, the optical center, etc. As a result, even when the actual two-dimensional image is acquired using different imaging devices, the first coordinate system corresponding to the three-dimensional CAD model can be suitably transformed into the second coordinate system corresponding to the actual two-dimensional image using coordinate transformation parameters that take into account the characteristics specific to each imaging device (such as differences in specifications and individual differences).

[0075] (5) In another embodiment, in any one of the above (1) to (4), The depiction unit adjusts the position and size of the defect image to be depicted by planar transforming the two-dimensional actual image including the defect image based on the result of comparing the two-dimensional actual image with the two-dimensional simulated image.

[0076] According to the above aspect (5), the two-dimensional actual image and the two-dimensional simulated image are compared, and the position and size of the defect image are adjusted based on the differences in their positional relationship, shape, size, etc. In this case, it is possible to simplify the processing and improve the accuracy of the adjustment compared to, for example, a case in which the position and size of the defect image are adjusted by comparing lines indicating the outlines of the inspection object.

[0077] (6) In another embodiment, in any one of the above (1) to (5), The system further includes a report creation unit that derives the three-dimensional position and dimensions of the defect in the object to be inspected based on the defect image projected onto the three-dimensional CAD model from the dimensional data of the three-dimensional CAD model, and creates a report including the results of deriving the position and dimensions.

[0078] According to the above aspect (6), a report including the three-dimensional position and dimensions of the defect is created, thereby reducing the workload of creating the report. Also, by collecting information on the position and dimensions of the defect for the same three-dimensional CAD model for multiple cases, it becomes possible to take statistics (for example, if the defect is a dent, the location where the dent is likely to occur can be identified from the statistical data).

[0079] (7) In another embodiment, in any one of the above (1) to (6), The image processing device further includes a coordinate transformation parameter correction unit for correcting the coordinate transformation parameters by noise removal using machine learning.

[0080] As described above, the shape of the inspection object contained in the 2D actual image used to estimate the coordinate transformation parameters is identified, for example, by image analysis or manual input by an operator, and therefore these involve some errors. According to the above aspect (7), by using noise removal through machine learning to correct the estimated coordinate transformation parameters, it is possible to reduce the influence of such errors and effectively improve the accuracy of coordinate transformation using the coordinate transformation parameters.

[0081] (8) An inspection support method according to one aspect includes: a step of identifying a shape of the inspection object included in a two-dimensional actual image based on the two-dimensional actual image obtained by capturing an image of the inspection object with an imaging device; detecting defects in the inspection object included in the two-dimensional actual image; a step of estimating coordinate transformation parameters for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to a viewpoint of the imaging device that captured the 2D actual image, based on the shape and the 3D CAD model of the inspection object; changing the position and orientation of viewpoint information of the 3D CAD model of the object to be inspected using the coordinate transformation parameters; extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model after the viewpoint information has been changed; depicting the defect image on the three-dimensional CAD model by fitting the two-dimensional actual image, including a defect image showing the defect, to the two-dimensional simulated image; Equipped with.

[0082] According to the above aspect (8), based on the shape of the inspection object included in the 2D actual image and the 3D CAD model of the inspection object, coordinate transformation parameters are estimated for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to the viewpoint of the imaging device that captured the 2D actual image. The coordinate transformation parameters include, for example, a translation vector and a rotation matrix, and are parameters for transforming the first coordinate system into the second coordinate system, which is a 2D coordinate system. In other words, the coordinate transformation parameters are parameters for estimating the position and orientation that define the viewpoint information of the imaging device in the first coordinate system based on n (n is any natural number) reference points expressed in three-dimensional coordinates in the first coordinate system corresponding to the 3D CAD model handled in the 3D CG software and those reference points expressed in two-dimensional coordinates in the second coordinate system corresponding to the 2D actual image. Using such coordinate transformation parameters, at least one of the position and direction of the viewpoint information of the 3D CAD model is changed to correspond to the inspection object included in the 2D actual image. In this way, by using the coordinate transformation parameters to change the position and direction of the viewpoint information of the 3D CAD model so that it corresponds to the 2D actual image, it is possible to suppress deviations in position and direction between them without requiring operator operation. Then, by depicting the defect image contained in the 2D actual image on the 3D CAD model whose position and direction of the viewpoint information have been changed in this way, it becomes possible to measure defects on the 3D CAD model with high accuracy.

[0083] (9) An inspection assistance program according to one aspect includes: On the computer, On the computer, a step of identifying a shape of the inspection object included in a two-dimensional actual image based on the two-dimensional actual image obtained by capturing an image of the inspection object with an imaging device; detecting defects in the inspection object included in the two-dimensional actual image; a step of estimating coordinate transformation parameters for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to a viewpoint of the imaging device that captured the 2D actual image, based on the shape and the 3D CAD model of the inspection object; changing at least one of the position and the direction of viewpoint information of the 3D CAD model of the object to be inspected using the coordinate transformation parameters; extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model after the viewpoint information has been changed; depicting the defect image on the three-dimensional CAD model by fitting the two-dimensional actual image, including a defect image showing the defect, to the two-dimensional simulated image; Execute the following.

[0084] According to the above aspect (9), based on the shape of the inspection object included in the 2D actual image and the 3D CAD model of the inspection object, coordinate transformation parameters are estimated for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to the viewpoint of the imaging device that captured the 2D actual image. The coordinate transformation parameters include, for example, a translation vector and a rotation matrix, and are parameters for transforming the first coordinate system into the second coordinate system, which is a 2D coordinate system. In other words, the coordinate transformation parameters are parameters for estimating the position and orientation that define viewpoint information of the imaging device in the first coordinate system based on n-point (n is any natural number) reference points expressed in three-dimensional coordinates in the first coordinate system corresponding to the 3D CAD model handled in the 3D CG software and those reference points expressed in two-dimensional coordinates in the second coordinate system corresponding to the 2D actual image. Using these coordinate transformation parameters, the position and orientation of the viewpoint information of the 3D CAD model are changed to correspond to the inspection object included in the 2D actual image. In this way, by using the coordinate transformation parameters to change the position and direction of the viewpoint information of the 3D CAD model so that it corresponds to the 2D actual image, it is possible to suppress deviations in position and direction between them without requiring operator operation. Then, by depicting the defect image contained in the 2D actual image on the 3D CAD model whose position and direction of the viewpoint information have been changed in this way, it becomes possible to measure defects on the 3D CAD model with high accuracy. [Explanation of symbols]

[0085] 1. Inspection support system 2. Inspection items 4. Communication Network 6 Client terminal 8 Server 10. Communications Department 11 Storage section 12 Output section 13 Input section 14 Arithmetic section 15 Communications Department 16 Memory section 18 Arithmetic section 30 Image acquisition unit 32 Shape recognition unit 34 Defect detection section 36 Report Writing Department 40 Coordinate transformation parameter estimation unit 41 Model Change Section 42 Image extraction section 44 Description section 50 Imaging device

Claims

1. a shape identification unit for identifying a shape of the inspection object included in a two-dimensional actual image based on the two-dimensional actual image obtained by capturing an image of the inspection object with an imaging device; a defect detection unit for detecting defects of the inspection object included in the two-dimensional actual image; a coordinate transformation parameter estimation unit for estimating, based on the shape identified by the shape identification unit and the 3D CAD model of the inspection object, coordinate transformation parameters for transforming a first coordinate system corresponding to the 3D CAD model into a second coordinate system corresponding to a viewpoint of the imaging device that captured the 2D actual image; a three-dimensional CAD model modification unit for modifying a position and a direction of viewpoint information of the three-dimensional CAD model of the inspection object using the coordinate transformation parameters; a two-dimensional simulated image extracting unit for extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model after the viewpoint information has been changed; a depiction unit for depicting the defect image, which is the two-dimensional actual image showing the defect detected by the defect detection unit, on the three-dimensional CAD model by fitting the defect image to the two-dimensional simulated image; An inspection support system equipped with:

2. 2. The inspection support system according to claim 1, wherein the coordinate transformation parameter estimation unit pre-identifies a plurality of first reference portions of the object to be inspected in the two-dimensional actual image based on the shape identified by the shape identification unit, and estimates the coordinate transformation parameters so that a plurality of second reference portions pre-registered in the three-dimensional CAD model coincide with the plurality of first reference portions.

3. The inspection support system according to claim 1 , wherein the coordinate transformation parameters include external parameters for defining the position and orientation of the imaging device in the first coordinate system.

4. The inspection support system according to claim 3 , wherein the coordinate transformation parameters further include internal parameters related to the imaging device.

5. The inspection support system of claim 1 or 2, wherein the depiction unit adjusts the position and dimensions of the defect image to be depicted by planar transforming the two-dimensional actual image including the defect image based on the result of comparing the two-dimensional actual image with the two-dimensional simulated image.

6. 3. The inspection support system according to claim 1, further comprising a report creation unit that derives the three-dimensional position and dimensions of the defect in the object to be inspected based on the defect image projected onto the three-dimensional CAD model from the dimensional data of the three-dimensional CAD model, and creates a report including the results of deriving the position and dimensions.

7. The inspection support system according to claim 1 or 2, further comprising a coordinate transformation parameter correction unit for correcting the coordinate transformation parameters by noise removal using machine learning.

8. a step of identifying a shape of the inspection object included in a two-dimensional actual image based on the two-dimensional actual image obtained by capturing an image of the inspection object with an imaging device; detecting defects in the inspection object included in the two-dimensional actual image; a step of estimating coordinate transformation parameters for transforming a first coordinate system corresponding to the three-dimensional CAD model into a second coordinate system corresponding to a viewpoint of the imaging device that captured the two-dimensional actual image, based on the shape and the three-dimensional CAD model of the inspection object; changing the position and orientation of viewpoint information of the 3D CAD model of the object to be inspected using the coordinate transformation parameters; extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model after the viewpoint information has been changed; Depicting the defect image on the three-dimensional CAD model by fitting the two-dimensional actual defect image showing the defect to the two-dimensional simulated image; An inspection support method comprising:

9. On the computer, a step of identifying a shape of the inspection object included in a two-dimensional actual image based on the two-dimensional actual image obtained by capturing an image of the inspection object with an imaging device; detecting defects in the inspection object included in the two-dimensional actual image; a step of estimating coordinate transformation parameters for transforming a first coordinate system corresponding to the three-dimensional CAD model into a second coordinate system corresponding to a viewpoint of the imaging device that captured the two-dimensional actual image, based on the shape and the three-dimensional CAD model of the inspection object; changing the position and orientation of viewpoint information of the 3D CAD model of the object to be inspected using the coordinate transformation parameters; extracting a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model after the viewpoint information has been changed; Depicting the defect image on the three-dimensional CAD model by fitting the two-dimensional actual defect image showing the defect to the two-dimensional simulated image; An inspection support program to carry out the above.

Images