Image processing device, image processing method, and image processing program

Abstract

This image processing device comprises at least one processor, wherein the processor acquires a plurality of transmission images, receives a designation of a region of interest with respect to at least one of the plurality of transmission images or extracts said region of interest, and uses the plurality of transmission images to perform super-resolution processing only on the region of interest.

Description

Image processing device, image processing method, and image processing program

[0001] The present disclosure relates to an image processing device, an image processing method, and an image processing program.

[0002] 2. Description of the Related Art Conventionally, inspections for defects in metal or other components involve irradiating the components with radiation such as X-rays and using signal values ​​obtained to confirm the presence or absence of defects inside the components.

[0003] Typically, the resolution of a transmission image obtained by radiography is about 50 to 200 μm / pixel, making it possible to visually recognize minute defects. However, depending on the imaging equipment, imaging environment, etc., it may not be possible to capture images at high resolution. In such cases, it is necessary to achieve higher resolution of the transmission image through image processing. One technology that can be applied for this image processing is super-resolution technology.

[0004] Super-resolution technology is classified into a method of generating one high-resolution image from one low-resolution image, and a method of generating one high-resolution image from multiple low-resolution images. The following technologies are related to the latter method.

[0005] Japanese Patent Laid-Open Publication No. 2005-95328 discloses a super-resolution processing device that aims to put into practical use super-resolution processing of image data obtained by a medical image diagnostic device.

[0006] This super-resolution processing device includes a unit that stores projection data relating to a subject obtained by an X-ray computed tomography device, a unit that specifies a super-resolution processing range on a three-dimensional image relating to the subject, a unit that reconstructs image data from the projection data limited to the specified super-resolution processing range, and a unit that performs super-resolution processing on the reconstructed image data using a point spread function related to the X-ray computed tomography device.

[0007] Furthermore, Republished Patent No. 2020 / 183733 discloses an X-ray imaging device that aims to generate high-resolution images while suppressing the calculation time required for generating the high-resolution images and suppressing the size of the image range treated as a single image from becoming smaller.

[0008] This X-ray imaging device comprises an X-ray source, a detector that detects X-rays irradiated from the X-ray source at a plurality of detection positions that are shifted in parallel by an amount smaller than one pixel from each other, and an image processing unit that generates a plurality of acquired images based on the X-rays detected at each of the plurality of detection positions, and performs super-resolution processing to increase the resolution based on the plurality of acquired images, thereby generating a high-resolution image with a higher resolution than the plurality of acquired images.The image processing unit is configured to apply the super-resolution processing to a first region in the acquired image in which a subject is present, and to increase the number of pixels in a second region other than the first region in the acquired image in accordance with the increase in resolution in the first region that is increased by applying the super-resolution processing, using processing that is simpler than the super-resolution processing.

[0009] However, image processing using super-resolution technology is extremely complicated because there are as many unknowns as there are pixels, the dimensions are very high, and iterative calculations such as steepest descent and conjugate gradient methods are used. In particular, the complexity of transmission images obtained during defect inspection is even greater because the number of pixels is large and the number of unknowns can exceed 50 million.

[0010] The present disclosure provides an image processing device, an image processing method, and an image processing program that can perform super-resolution processing more easily than conventional techniques.

[0011] A first aspect of the present disclosure is an image processing device comprising at least one processor, which acquires a plurality of transmission images, accepts designation of a region of interest for at least one of the plurality of transmission images, or extracts the region of interest, and performs super-resolution processing using the plurality of transmission images, targeting only the region of interest.

[0012] A second aspect of the present disclosure is the first aspect, wherein the plurality of transmission images may be identical in appearance including at least the angle of view.

[0013] A third aspect of the present disclosure is the first or second aspect, wherein the plurality of transmission images are images of a component.

[0014] A fourth aspect of the present disclosure is the third aspect, wherein the plurality of transmission images may be images of defects occurring in the component.

[0015] A fifth aspect of the present disclosure is that in the first or second aspect, the processor may extract the region of interest using a trained model created in advance by machine learning using a transmission image as input information and the region of interest as output information.

[0016] A sixth aspect of the present disclosure is the first or second aspect, wherein the processor may extract the region of interest by image processing on the transmission image.

[0017] A seventh aspect of the present disclosure is that in the first or second aspect, the processor may use the region of interest to create partial images of the multiple transmission images, and perform super-resolution processing using the partial images.

[0018] An eighth aspect of the present disclosure is an image processing method, in which a processor provided in an image processing device acquires a plurality of transmission images, accepts designation of a region of interest for at least one of the plurality of transmission images, or extracts the region of interest, and performs super-resolution processing using the plurality of transmission images on only the region of interest.

[0019] Furthermore, a ninth aspect of the present disclosure is an image processing program that causes a processor provided in an image processing device to acquire a plurality of transmission images, accept specification of a region of interest for at least one of the plurality of transmission images, or extract the region of interest, and perform super-resolution processing using the plurality of transmission images, targeting only the region of interest.

[0020] According to the above aspects, the image processing device, image processing method, and image processing program of the present disclosure can perform super-resolution processing more easily than conventional techniques.

[0021] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing device according to an exemplary embodiment; FIG. 2 is a schematic diagram showing an example of the flow of super-resolution processing using a general super-resolution technique; FIG. 3 is a block diagram showing an example of the functional configuration of an image processing device according to an exemplary embodiment; FIG. 4 is a diagram showing an example of an extraction result of a region of interest according to an exemplary embodiment; FIG. 5 is a schematic diagram showing an example of the flow of super-resolution processing according to an exemplary embodiment; FIG. 6 is a schematic diagram showing an example of the configuration of an image information database according to an exemplary embodiment; FIG. 7 is a flowchart showing an example of image processing according to an exemplary embodiment; FIG. 8 is a diagram showing another example of a method for extracting a region of interest according to an exemplary embodiment; FIG. 9 is a schematic diagram showing another example of the flow of super-resolution processing according to an exemplary embodiment.

[0022] Hereinafter, with reference to the drawings, exemplary embodiments for implementing the technology of the present disclosure will be described in detail. Note that in this exemplary embodiment, a case will be described in which an image processing device and an image processing method according to the technology disclosed herein are applied to an image processing device that performs image processing using super-resolution technology (hereinafter referred to as "super-resolution processing") on a region of interest (hereinafter referred to as "region of interest") included in a transmission image obtained by radiography. Note that the "region of interest" here refers to the range of an object, figure, etc. that is to be processed in image processing, computer vision, etc.

[0023] First, the hardware configuration of an image processing device 10 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the hardware configuration of the image processing device 10 according to this exemplary embodiment.

[0024] 1 , the image processing device 10 includes a CPU (Central Processing Unit) 20, a memory 21 serving as a temporary storage area, and a non-volatile storage unit 22. The image processing device 10 also includes a display 23 such as a liquid crystal display, an input device 24 such as a keyboard and a mouse, and a network I / F (Interface) 25 connected to a network. The image processing device 10 also includes an external I / F 26 to which various external devices are connected. The CPU 20, the memory 21, the storage unit 22, the display 23, the input device 24, the network I / F 25, and the external I / F 26 are connected to a bus 27. The CPU 20 is an example of a processor according to the disclosed technology.

[0025] The storage unit 22 is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 22 serves as a storage medium and stores an image processing program 30. The CPU 20 reads the image processing program 30 from the storage unit 22, loads it into the memory 21, and executes the loaded image processing program 30.

[0026] The storage unit 22 also stores an image information database 32 and a trained model 34. The image information database 32 and the trained model 34 will be described in detail later.

[0027] Here, super-resolution processing using a general super-resolution technique will be described with reference to Fig. 2. Fig. 2 is a schematic diagram showing an example of the flow of super-resolution processing using a general super-resolution technique. Note that, here, a case where a reconstruction-type technique is applied as the super-resolution technique will be described.

[0028] As an example, as shown in FIG. 2, in this super-resolution technology, a group of images y obtained by actual photography is ｔ and a group of low-resolution images z obtained by image degradation simulation from the image x for which super-resolution has been estimated. ｔ The error is fed back to the image x repeatedly until the error converges, thereby obtaining a high-resolution image x.

[0029] In addition, for reconstruction-type super-resolution techniques, methods such as the MAP (Maximum A Posteriori) method, the IBP (Iterative Back Projection) method, and the ML (Maximum-Likelihood) method have been proposed due to differences in the energy functions used to update high-resolution images and differences in the update methods, and any of these methods can be applied to the image processing device 10 according to this exemplary embodiment.

[0030] However, as described above, image processing using super-resolution technology is extremely complicated because the number of unknowns is the same as the number of pixels, the dimension is very high, and iterative calculations such as steepest descent method and conjugate gradient method are used. In particular, transmission images obtained in defect inspection have a large number of pixels, and the number of unknowns may reach 50 million or more, which makes the process even more complicated.

[0031] The image processing device 10 according to this exemplary embodiment performs image processing to solve the above problems. Next, the functional configuration of the image processing device 10 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the functional configuration of the image processing device 10 according to this exemplary embodiment.

[0032] 3, the image processing device 10 includes an acquisition unit 20A, an identification unit 20B, a processing unit 20C, and a registration unit 20D. The CPU 20 executes the image processing program 30 to function as the acquisition unit 20A, the identification unit 20B, the processing unit 20C, and the registration unit 20D.

[0033] The acquisition unit 20A according to this exemplary embodiment acquires a plurality of transmission images. Note that in this exemplary embodiment, the plurality of transmission images are acquired by reading them out from the storage unit 22, but this is not limiting. The plurality of transmission images may also be acquired by reading them out from an external device connected to the external I / F 26.

[0034] In this exemplary embodiment, the multiple transmission images are identical in appearance, including at least the angle of view, but are not limited to this. For example, multiple transmission images that have different angles of view but contain the same region of interest of the same subject may be used as the multiple transmission images.

[0035] In addition, in this exemplary embodiment, images of parts such as pipes, screws, bolts, nuts, etc. are used as the multiple transmission images, but the present invention is not limited to this. For example, images of structures such as equipment that may have defects such as cracks or corrosion, or buildings, etc. may be used as the multiple transmission images.

[0036] Furthermore, in this exemplary embodiment, the multiple transmission images are images of defects that have occurred in the component, but the present invention is not limited to this. For example, the multiple transmission images may be images of changes, discolorations, or the like that have occurred in the component.

[0037] 4, the identification unit 20B according to this exemplary embodiment extracts a region of interest 40 from at least one of the multiple transmission images 50 acquired by the acquisition unit 20A, thereby identifying the region of interest 40. Fig. 4 is a diagram showing an example of an extraction result of the region of interest 40 according to this exemplary embodiment, in which the left diagram shows an example of a transmission image 50 in which a subject 60 has been captured, and the right diagram shows an example of an image (hereinafter referred to as an "identification result image") 52 showing a result of identifying the region of interest 40 (the region of the subject 60 in the example shown in Fig. 4) from the transmission image 50 shown in the left diagram.

[0038] In the right diagram of Fig. 4, the region of interest 40 is shown in white, and the other region, the background region 42, is shown in black, resulting in a specified result image 52. By masking the transparent image 50 with this specified result image 52 (taking a logical product on a pixel-by-pixel basis), it becomes possible to subject only the region of interest 40 in the transparent image 50 to super-resolution processing. As a result, the computational load of the super-resolution processing can be significantly reduced. As shown in the left diagram of Fig. 4, the background region 42 in the transparent image 50 is a black (or nearly black) region, and therefore excluding this region from the super-resolution processing has very little visual impact.

[0039] The identification unit 20B according to this exemplary embodiment extracts the region of interest 40 using a trained model created in advance by machine learning using a transparent image as input information and the region of interest as output information. Examples of trained models that can be applied in this case include segmentation models such as a semantic segmentation model and an instance segmentation model, as well as a generative adversarial network (GAN). The trained model 34 stored in the storage unit 22 is a model that represents the trained model itself used by the identification unit 20B.

[0040] As described above, in this exemplary embodiment, a method using the trained model 34 is applied as a method for extracting the region of interest 40, but the present invention is not limited to this. For example, the region of interest 40 may be extracted by image processing of the transmission image 50. Examples of image processing that can be applied in this manner include a filter process that cuts out the image of the background region 42, and a binarization process that uses a value between the pixel values ​​of the image of the background region 42 and the pixel values ​​of the image of the region of interest 40 as a threshold. In this manner, by extracting the region of interest 40 by image processing, the region of interest 40 can be extracted more easily than when the region of interest 40 is extracted using the trained model 34.

[0041] Furthermore, the processing unit 20C according to this exemplary embodiment performs super-resolution processing using the multiple transmission images, targeting only the region of interest 40 extracted by the identification unit 20B, as shown in Fig. 5 as an example. Fig. 5 is a schematic diagram showing an example of the flow of super-resolution processing according to this exemplary embodiment.

[0042] The registration unit 20D according to this exemplary embodiment then registers the image obtained by the super-resolution processing performed by the processing unit 20C as a processed image in the storage unit 22. In this manner, in this exemplary embodiment, the process using the high-resolution image obtained by the processing performed by the processing unit 20C (hereinafter referred to as "utilization process") is the process of registering the high-resolution image, but this is not limited to this. For example, the process of displaying the high-resolution image on the display 23 may be applied as the utilization process.

[0043] Next, the image information database 32 according to this exemplary embodiment will be described with reference to Fig. 6. Fig. 6 is a schematic diagram showing an example of the configuration of the image information database 32 according to this exemplary embodiment.

[0044] The image information database 32 according to this exemplary embodiment is a database in which information about images to be processed is registered. As shown in Fig. 6, the image information database 32 according to this exemplary embodiment stores information about subject IDs (Identifications), original images, and processed images.

[0045] The subject ID is information that is assigned in advance as a unique ID to each subject 60 in order to individually identify the subject 60 targeted by the image processing device 10. The original image is information that indicates the image information itself of the above-mentioned multiple transmission images 50 obtained by performing radiography at a relatively low resolution on the corresponding subject 60. The processed image is information that indicates the image information itself of the above-mentioned processed image that has been made high-resolution by super-resolution processing using the corresponding multiple pieces of image information.

[0046] Next, the operation of the image processing device 10 according to this exemplary embodiment will be described with reference to FIG. 7. The image processing shown in FIG. 7 is performed by the CPU 20 of the image processing device 10 executing the image processing program 30. The image processing shown in FIG. 7 is performed, for example, when a command to perform the image processing is input by a user of the image processing device 10. Note that, to avoid confusion, the following description will be given of a case in which the image information database 32 has already been constructed, excluding the storage area for the processed image. The following description will also be given of a case in which a subject ID corresponding to the subject to be processed has been specified in advance.

[0047] In step 100 of Figure 7, the CPU 20 reads out image information of multiple original images corresponding to the specified subject ID (corresponding to the image information of ``multiple transparent images'' in this disclosure, hereinafter referred to as ``target image information group'') from the image information database 32.

[0048] In step 102, the CPU 20 extracts the region of interest 40 from the transmission image indicated by one piece of image information from the read-out group of target image information using the trained model 34 as described above, thereby identifying the region of interest 40. This process can obtain an example of the identification result image 52 shown in the right diagram of FIG.

[0049] In step 104, the CPU 20 performs the super-resolution process as described above on only the extracted region of interest 40 in each of the target image information sets. At this time, as described above, the specified result image 52 is used to mask areas other than the region of interest.

[0050] In step 106, the CPU 20 stores (registers) image information indicating the processed image obtained by the above processing in the corresponding storage area of ​​the image information database 32, and then ends this image processing.

[0051] As described above, the image processing device according to this exemplary embodiment acquires multiple transmission images, extracts a region of interest from at least one of the multiple transmission images, and performs super-resolution processing using the multiple transmission images, targeting only the region of interest. Therefore, super-resolution processing can be performed more easily than with conventional techniques.

[0052] Furthermore, according to the image processing device of this exemplary embodiment, the multiple transmission images that are applied are images that are identical in appearance, at least in terms of the angle of view, and therefore, super-resolution processing can be performed more easily than when multiple transmission images that are different in appearance are applied.

[0053] Furthermore, according to the image processing device according to this exemplary embodiment, images of a component are used as the multiple transmission images, and therefore, super-resolution processing can be performed on the transmission images of the component.

[0054] Furthermore, according to the image processing device of this exemplary embodiment, the multiple transmission images are images of defects occurring in the component, so that the component defects can be confirmed in detail.

[0055] Furthermore, according to the image processing device of this exemplary embodiment, the region of interest is extracted using a trained model created in advance by machine learning, with the transmitted image as input information and the region of interest as output information. Therefore, the region of interest can be extracted with higher accuracy than when the region of interest is extracted by image processing.

[0056] In the above exemplary embodiment, the case where the region of interest 40 is identified by extraction has been described, but the present invention is not limited to this. For example, as shown in FIG. 8 , the region of interest 40 may be identified by receiving a designation from a user. FIG. 8 is a diagram showing another example of a method for extracting a region of interest 40 according to this exemplary embodiment. The left diagram in FIG. 8 shows an example of a state in which the user has designated the region of interest 40 with a rectangular frame in the transparent image 50 displayed on the display 23, and the right diagram shows a designated result image 52 in which a background region 42, which is an area other than the region of interest 40 designated in accordance with the designation, is masked.

[0057] In the above exemplary embodiment, a case has been described in which an entire image is generated as a processed result image after super-resolution processing, but the present invention is not limited to this. For example, as shown in Fig. 9, a specified region of interest 40 may be used to create partial images 54 of multiple transmission images, and super-resolution processing may be performed using only these partial images 54 to generate a processed result image corresponding to the region of interest 40. Fig. 9 is a schematic diagram showing another example of the flow of super-resolution processing according to this exemplary embodiment.

[0058] In the above exemplary embodiment, the case where the region of interest is identified in only one of the multiple transmission images has been described, but the present invention is not limited to this, and the region of interest may be identified in multiple transmission images excluding all of the multiple transmission images. In this case, an exemplary embodiment of the super-resolution processing may be a form in which a region including all of the multiple identified regions of interest is applied as the region of interest for all transmission images, and the super-resolution processing is performed.

[0059] Furthermore, in the above exemplary embodiment, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the acquisition unit 20A, the identification unit 20B, the processing unit 20C, and the registration unit 20D. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0060] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0061] Examples of configuring multiple processing units with a single processor include: first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server; second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs); and thus, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0062] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0063] In the above exemplary embodiment, the image processing program 30 is pre-stored (installed) in the storage unit 22 of the image processing device 10, but this is not limiting. The image processing program 30 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The image processing program 30 may also be downloaded from an external device via a network.

[0064] From the above description, the present disclosure described in the following appendix can be understood.

[0065] [Supplementary Note 1] An image processing device comprising at least one processor, wherein the processor acquires a plurality of transmission images, accepts designation of a region of interest for at least one of the plurality of transmission images or extracts the region of interest, and performs super-resolution processing using the plurality of transmission images with respect to only the region of interest. [Supplementary Note 2] The image processing device according to Supplementary Note 1, wherein the plurality of transmission images have the same appearance including at least the angle of view. [Supplementary Note 3] The image processing device according to Supplementary Note 1 or Supplementary Note 2, wherein the plurality of transmission images are images of a component. [Supplementary Note 4] The image processing device according to Supplementary Note 3, wherein the plurality of transmission images are images of a defect that has occurred in the component. [Supplementary Note 5] The image processing device according to any one of Supplements 1 to 4, wherein the processor extracts the region of interest using a trained model created in advance by machine learning using the transmission images as input information and the region of interest as output information. [Supplementary Note 6] The image processing device according to any one of Supplementary Notes 1 to 4, wherein the processor extracts the region of interest by image processing of the transmission images. [Supplementary Note 7] The image processing device according to any one of Supplementary Notes 1 to 6, wherein the processor creates partial images of the plurality of transmission images using the region of interest, and performs the super-resolution processing using the partial images. [Supplementary Note 8] An image processing method, in which a processor included in an image processing device executes the following processing: acquiring a plurality of transmission images, accepting designation of a region of interest for at least one of the plurality of transmission images or extracting the region of interest, and performing super-resolution processing using the plurality of transmission images with the region of interest as the target only. [Supplementary Note 9] An image processing program for causing a processor included in an image processing device to execute the following processing: acquiring a plurality of transmission images, accepting designation of a region of interest for at least one of the plurality of transmission images or extracting the region of interest, and performing super-resolution processing using the plurality of transmission images with the region of interest as the target only.

[0066] The disclosure of Japanese Patent Application No. 2024-049355, filed March 26, 2024, is incorporated herein by reference in its entirety.

[0067] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An image processing device comprising at least one processor, which acquires a plurality of transmission images, accepts designation of a region of interest for at least one of the plurality of transmission images or extracts the region of interest, and performs super-resolution processing using the plurality of transmission images, targeting only the region of interest.

2. The image processing device according to claim 1, wherein the plurality of transmitted images are identical in appearance, including at least the angle of view.

3. The image processing device according to claim 1 or claim 2, wherein the plurality of transmission images are images of a component.

4. The image processing device according to claim 3, wherein the plurality of transmission images are images of defects occurring in the component.

5. The image processing device according to claim 1 or claim 2, wherein the processor extracts the region of interest using a trained model created in advance by machine learning using the transmitted image as input information and the region of interest as output information.

6. The image processing device according to claim 1 or claim 2, wherein the processor extracts the region of interest by image processing of the transmission image.

7. An image processing device according to claim 1 or claim 2, wherein the processor creates partial images of the plurality of transmitted images using the region of interest, and performs the super-resolution processing using the partial images.

8. An image processing method in which a processor included in an image processing device executes the following processing: acquiring a plurality of transmission images; accepting designation of a region of interest for at least one of the plurality of transmission images or extracting the region of interest; and performing super-resolution processing using the plurality of transmission images, targeting only the region of interest.

9. An image processing program for causing a processor included in an image processing device to execute the following process: acquiring a plurality of transmission images; accepting designation of a region of interest for at least one of the plurality of transmission images or extracting the region of interest; and performing super-resolution processing using the plurality of transmission images, targeting only the region of interest.

Images