Examination support device, examination support method, and program

Abstract

To efficiently and highly accurately acquire a segmentation mask.SOLUTION: A control unit 12 of an inspection support device 1 acquires one reference image obtained by photographing a reference object which is an inspection target object serving as a reference, one mask image obtained by applying a segmentation mask to the reference image, and one or more inspection images obtained by photographing the inspection target object, obtains a transformation matrix for transforming the reference image into the inspection image, creates a transformed image by multiplying the mask image by the transformation matrix, and creates an inspection support image by superimposing the transformed image on the inspection image such that the same pixel coordinates overlap each other.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an inspection support device, an inspection support method, and a program. [Background technology]

[0002] In recent years, a technology has been proposed that uses a segmentation AI model to predict a segmentation mask for an image. Building a segmentation AI model requires the construction of a learning dataset for use in supervised learning. However, building a dataset for use in supervised learning requires high annotation costs, so Non-Patent Document 1 discloses a method for making annotation more efficient. Patent Document 1 also discloses a technology that detects shake in the imaging direction and corrects the composite position of a mask image based on the detected value so that the mask image follows the shake of the image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-60448 [Non-patent literature]

[0004] [Non-Patent Document 1] "Efficient annotation in image segmentation," [online], September 14, 2022, [Retrieved May 30, 2024], Internet<URL:https: / / fintan.jp / page / 4646 / > Summary of the Invention [Problem to be solved by the invention]

[0005] However, conventional segmentation techniques are not 100% accurate, and there is a need for a technique that can predict a segmentation mask efficiently and with high accuracy when inspecting an object to be inspected. Furthermore, the technique described in Patent Document 1 does not change the size of the mask image, so distortion correction cannot be performed.

[0006] In view of the above circumstances, an object of the present disclosure is to provide an inspection support device, an inspection support method, and a program that are capable of acquiring a segmentation mask efficiently and with high accuracy. [Means for solving the problem]

[0007] The gist of the present disclosure for solving the above problems is as follows.

[0008] (1) An inspection support device including a control unit, the control unit acquires one reference image of a reference object that is a reference object to be inspected, one mask image in which a segmentation mask is added to the reference image, and one or more inspection images of the object to be inspected, calculates a transformation matrix that converts the reference image to the inspection image, multiplies the mask image by the transformation matrix to create a transformed image, and overlays the transformed image on the inspection image so that the same pixel coordinates overlap to create an inspection support image.

[0009] (2) The inspection support device according to (1), wherein the mask image is an image for distinguishing areas having different required standards.

[0010] (3) The inspection support device according to (1) or (2), wherein the control unit creates the transformed image by multiplying the mask image by the transformation matrix for each pixel.

[0011] (4) An inspection support method in which an inspection support device executes the steps of: acquiring one reference image of a reference object that is a reference object to be inspected; one mask image in which a segmentation mask is added to the reference image; and one or more inspection images of the object to be inspected; determining a transformation matrix that converts the reference image to the inspection image; multiplying the mask image by the transformation matrix to create a transformed image; and overlaying the transformed image on the inspection image so that the same pixel coordinates overlap to create an inspection support image.

[0012] (5) A program for causing a computer to function as the inspection support device according to any one of (1) to (3). [Effects of the Invention]

[0013] According to the present disclosure, it is possible to obtain a segmentation mask efficiently and with high accuracy. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a diagram illustrating an example of the configuration of an inspection support device according to an embodiment. [Figure 2] 3A to 3C are schematic diagrams showing examples of a reference image, a mask image, and an inspection image acquired by an inspection support apparatus according to an embodiment. [Figure 3] 10A and 10B are diagrams illustrating derivation of a transformation matrix by an inspection support device according to an embodiment. [Figure 4] 10A and 10B are diagrams illustrating the creation of a converted image by an inspection support device according to an embodiment. [Figure 5] 1A and 1B are diagrams illustrating the creation of an inspection support image by an inspection support device according to an embodiment. [Figure 6] 1 is a flowchart illustrating an example of a procedure of an inspection assistance method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, one embodiment will be described in detail with reference to the drawings.

[0016] Fig. 1 shows an example of the configuration of a test support device according to one embodiment. The test support device 1 shown in Fig. 1 includes an input unit 11, a control unit 12, a storage unit 13, and an output unit 14. The test support device 1 may further include a communication I / F such as a LAN (Local Area Network) I / F to enable communication with an external device.

[0017] The input unit 11 is, for example, a physical key, a capacitance key, a pointing device, or a touch screen integrated with a display. The input unit 11 accepts an operation to input data used for the operation of the test assistance device 1. The input unit 11 may be connected to the test assistance device 1 as an external input device instead of being provided in the test assistance device 1. As a connection interface, any interface compatible with standards such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface, registered trademark), or Bluetooth (registered trademark) may be used.

[0018] The control unit 12 may be configured with dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be configured with a processor, or may be configured to include both. The control unit 12 executes processes related to the operation of the test support device 1 while controlling each part of the test support device 1.

[0019] The storage unit 13 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a random access memory (RAM), a read only memory (ROM), or a flash memory. The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). The flash memory is, for example, a solid-state drive (SSD). The magnetic memory is, for example, a hard disk drive (HDD). The storage unit 13 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory.

[0020] The output unit 14 is, for example, a display, a speaker, or a printer, and presents the data created by the control unit 12 to the user. The display is, for example, an LCD (liquid crystal display) or an organic EL (electro luminescent) display. The output unit 14 may be connected to the test support device 1 as an external output device instead of being provided in the test support device 1. As a connection interface, any interface compatible with standards such as USB, HDMI, or Bluetooth can be used.

[0021] The control unit 12 shown in FIG. 1 includes an image acquisition unit 121, a transformation matrix derivation unit 122, a transformed image creation unit 123, and an examination support image creation unit .

[0022] The image acquisition unit 121 acquires one reference image and one mask image in which a segmentation mask is added to the reference image. The reference image and the mask image may be prepared by a user and stored in advance in the storage unit 13 via the input unit 11. The image acquisition unit 121 also acquires one or more inspection images during inspection.

[0023] 2 is a schematic diagram showing an example of a reference image A, a mask image B, and an inspection image C. The reference image A is an image obtained by capturing a reference object, which is an object to be inspected as a reference. The reference object is placed at a predetermined position.

[0024] Mask image B is an image in which a segmentation mask is applied to image a of the reference object in reference image A. The object to be inspected may have different required specifications (inspection standards such as the acceptable range of flaw size) for each region. In such a case, mask image B may be an image for distinguishing between regions with different required specifications. Mask image B may also be an image in which the background of the reference object is masked from reference image A. Mask image B shown in FIG. 2 distinguishes three regions with different required specifications in image a of the reference object in reference image A (image a'), and also masks the background of the reference object. Note that in FIG. 2, the regions with different required specifications are distinguished by the direction and spacing of the diagonal lines for convenience, but any method may be used to distinguish them.

[0025] Inspection image C is an image of the object to be inspected. The object to be inspected may be positioned at any position. Image b of the object to be inspected in inspection image C may be tilted or distorted. Note that reference image A, mask image B, and inspection image C are images of the same format.

[0026] The transformation matrix derivation unit 122 determines a transformation matrix for transforming the reference image into the inspection image based on the feature amounts of the reference image and the inspection image.

[0027] 3 is a diagram illustrating the derivation of a transformation matrix. The transformation matrix derivation unit 122 extracts feature points of reference image A and inspection image C (feature points of image a of the reference object and image b of the inspection target object) and calculates a transformation matrix for aligning the feature points. The transformation matrix can be calculated by any method, such as affine transformation, projective transformation, thin plate spline (TPS) transformation, or optical flow. Note that it is not necessary to calculate a transformation matrix from the background region.

[0028] The transformation matrix derivation unit 122 outputs and stores the derived transformation matrix in the storage unit 13. At this time, the transformation matrix derivation unit 122 may store the transformation matrix in the storage unit 13 in association with an identifier (ID) of the test image.

[0029] The transformed image creating unit 123 creates a transformed image by multiplying the mask image by the transformation matrix derived by the transformation matrix derivation unit 122 for each pixel.

[0030] 4 is a diagram illustrating the creation of a transformed image. Transformed image creation unit 123 creates a transformed image D corresponding to inspection image C by multiplying mask image B corresponding to reference image A by a transformation matrix. By multiplying the transformation matrix on a pixel-by-pixel basis, including the background, distortion correction can be performed. As a result, image b' of the object to be inspected in transformed image D is positioned at the same position and with the same size as image b of the object to be inspected in inspection image C.

[0031] The inspection support image creating unit 124 creates an inspection support image by overlaying the converted image created by the converted image creating unit 123 on the inspection image so that the same pixel coordinates overlap.

[0032] 5 is a diagram illustrating the creation of an inspection support image. The inspection support image creation unit 124 creates inspection support image E by overlaying converted image D on inspection image C so that the same pixel coordinates overlap. The inspection support image creation unit 124 may create inspection support image E by overlaying an image b' of the object to be inspected in converted image D, which has been subjected to semi-transparency processing, onto image b of the object to be inspected in inspection image C. In this case, the user can easily visually confirm whether inspection support image E has been created correctly as intended.

[0033] The control unit 12 inspects the object to be inspected using the inspection support image E. When the image b" of the object to be inspected in the inspection support image E is divided into regions, the control unit 12 grasps the required standards for each region and inspects whether each region satisfies the different required standards. For example, the control unit 12 may adjust the detection sensitivity threshold (parameter) for defect judgment for each region. The control unit 12 may make a pass / fail judgment on a pixel-by-pixel basis, or may determine the degree of abnormality on a pixel-by-pixel basis.

[0034] Next, an examination support method according to an embodiment will be described below with reference to a flowchart of FIG.

[0035] In step S101, one reference image and one mask image are prepared in advance.

[0036] In step S102, the image acquisition unit 121 acquires the nth inspection image. The initial value of n is 1, and if it is determined in step S108 that an uninspected inspection object remains, n is incremented.

[0037] In step S103, the transformation matrix derivation unit 122 calculates a transformation matrix for transforming the reference image into the inspection image based on the feature amounts of the reference image and the inspection image.

[0038] In step S104, the transformation matrix derivation unit 122 stores the transformation matrix calculated in step S103 in the storage unit 13.

[0039] In step S105, the transformed image creation unit 123 creates a transformed image by multiplying each pixel of the mask image corresponding to the reference image by a transformation matrix. When the process of step S105 is completed, the control unit 12 may delete the transformation matrix from the storage unit 13.

[0040] In step S106, the inspection support image creating unit 124 creates an inspection support image by overlaying the converted image on the inspection image so that the same pixel coordinates overlap.

[0041] In step S107, the control unit 12 inspects the object to be inspected using the inspection support image.

[0042] In step S108, if inspection of all inspection target objects has been completed, the process is terminated, and if there are any inspection target objects remaining that have not been inspected, the process returns to step S102.

[0043] In this way, in the present invention, a transformation matrix for converting a reference image into an inspection image is calculated, the transformation matrix is ​​multiplied by the mask image to create a transformed image, and the transformed image is then overlaid on the inspection image so that the same pixel coordinates overlap, creating an inspection support image. While it is conceivable to restore the tilt and distortion of the image of the object to be inspected in the inspection image before applying a segmentation mask, performing such processing would result in a large number of similar images being generated that are not managed in the original inspection image state, making management difficult for inspection purposes. Therefore, in the present invention, an inspection support image is created without correcting the tilt and distortion of the image of the object to be inspected in the inspection image.

[0044] According to the present invention, it is possible to create an inspection support image from which a segmentation mask has been acquired efficiently and with high accuracy. Furthermore, when the required specifications differ for each region of an object to be inspected, it is possible to create an inspection support image in which the regions are distinguished with high accuracy, and to inspect whether each region satisfies the different required specifications. Furthermore, by multiplying the mask image by a transformation matrix for each pixel, it is possible to acquire a segmentation mask with high accuracy even if the object to be inspected in the inspection image is tilted or distorted.

[0045] <Program> A computer capable of executing program instructions can be used to function as the above-described test support device 1. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, a mobile terminal, etc. The program instructions may be program code, code segments, etc. for performing necessary tasks.

[0046] The control unit 12 is a processor such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), or SoC (System on a Chip), and may be configured with multiple processors of the same or different types. The processor performs the above-mentioned processing by reading and executing a program from the storage unit 13. Note that at least a part of the processing content may be realized by hardware.

[0047] The program may be recorded on a computer-readable recording medium. Using such a recording medium, the program can be installed on a computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB memory. Furthermore, the program may be downloaded from an external device via a network.

[0048] The above-described embodiments have been described as typical examples, but it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and alterations are possible without departing from the scope of the claims. For example, multiple building blocks shown in the configuration diagrams of the embodiments can be integrated, or a single building block can be divided. Furthermore, multiple steps shown in the flowcharts of the embodiments can be integrated into one, or a single step can be divided. [Explanation of symbols]

[0049] 1. Inspection support equipment 11 Input section 12 Control Unit 13 Storage section 14 Output section 121 Image acquisition unit 122 Transformation matrix derivation part 123 Transformation Image Creation Unit 124 Examination Support Image Creation Department

Claims

1. An examination support device including a control unit, The control unit Obtaining one reference image obtained by photographing a reference object that is a reference object to be inspected, one mask image obtained by adding a segmentation mask to the reference image, and one or more inspection images obtained by photographing the object to be inspected; determining a transformation matrix for transforming the reference image into the inspection image; multiplying the mask image by the transformation matrix to generate a transformed image; The inspection support device creates an inspection support image by overlaying the converted image on the inspection image so that the same pixel coordinates overlap.

2. The inspection support device according to claim 1 , wherein the mask image is an image for distinguishing areas having different required standards.

3. The inspection support device according to claim 1 , wherein the control unit creates the transformed image by multiplying the mask image by the transformation matrix for each pixel.

4. The inspection support device A step of acquiring one reference image obtained by photographing a reference object that is a reference object to be inspected, one mask image obtained by adding a segmentation mask to the reference image, and one or more inspection images obtained by photographing the object to be inspected; determining a transformation matrix for transforming the reference image into the inspection image; multiplying the mask image by the transformation matrix to generate a transformed image; creating an inspection support image by overlaying the converted image on the inspection image so that the same pixel coordinates overlap; An inspection support method for performing the above.

5. A program for causing a computer to function as the inspection support device according to claim 1 or 2.

Images