Image processing apparatus, mapping image alignment method, and program

Abstract

To accurately align mapping images of a material.SOLUTION: An image processing apparatus 20 includes a control unit 21 configured to: filter a first mapping image obtained by mapping measurement on a first region 41 of a material 40 and a second mapping image obtained by mapping measurement on a second region 42 of the material 40, the second region being at least partially overlapping the first region 41; extract a feature amount of the first mapping image and the second mapping image to generate a first feature amount map and a second feature amount map; determine a deformation parameter based on a first degree of difference between the generated first feature amount map and a deformed feature amount map which is obtained by deforming the generated second feature amount map by using the deformation parameter; and apply the determined deformation parameter to the second mapping image to align the first mapping image and the second mapping image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, a mapping image alignment method, and a program. [Background technology]

[0002] In the research and development of materials, especially metal materials, various micro-scale analyses are widely performed using mapping images of materials. Mapping images of materials are, for example, structural images obtained by observing structural morphology with an optical microscope, SEM, or TEM, element maps obtained by measuring element distribution in each structure with SEM-EDS, EPMA, or XRF, or IPF maps, KAM maps, misorientation maps, IQ maps, or phase maps obtained by measuring crystal orientation or crystal structure with EBSD. "SEM" is an abbreviation for scanning electron microscope. "TEM" is an abbreviation for transmission electron microscopy. "EDS" is an abbreviation for energy-dispersive X-ray spectroscopy. "EPMA" is an abbreviation for electron micro probe analyzer. "XRF" is an abbreviation for X-ray fluorescence. "EBSD" is an abbreviation for electron backscatter diffraction. "IPF" is an abbreviation for inverse pole figure. "KAM" is an abbreviation for Kernel average misorientation. "IQ" is an abbreviation for image quality.

[0003] In many research and development projects, multiple analytical methods are combined to study materials from multiple angles. However, due to differences in measurement timing and analytical equipment, there is a misalignment between mapping images of materials, which hinders detailed analysis. In particular, computer-based analysis such as data science (DS) analyzes mapping images of materials on a pixel-by-pixel basis, so a mapping image of a material with the same field of view that is highly precisely aligned with the mapping image of a metal material is required.

[0004] Patent Documents 1 and 2 propose a method of extracting a specific marker or region of interest and performing registration according to their coordinates. Patent Document 3 proposes a method of performing registration using local features. Patent Document 4 proposes a method of performing registration using template matching. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2021-126538 A [Patent Document 2] JP 2013-257282 A [Patent Document 3] JP 2013-073476 A [Patent Document 4] JP 2001-283190 A Summary of the Invention [Problem to be solved by the invention]

[0006] When attempting to apply the methods disclosed in Patent Documents 1 to 4 to metal materials, alignment is difficult because the appearance of the tissue varies greatly depending on the measurement method or tissue structure.

[0007] An object of the present disclosure is to perform alignment of mapping images of materials with high accuracy. [Means for solving the problem]

[0008] (1) An image processing device according to an embodiment of the present disclosure, filtering at least one first mapping image obtained by a mapping measurement of a first region of a material and at least one second mapping image obtained by a mapping measurement of a second region of the material at least partially overlapping with the first region, and extracting features of the at least one first mapping image and the at least one second mapping image, thereby generating at least one first feature map and at least one second feature map, respectively; determining a transformation parameter based on a first dissimilarity between the at least one first feature map and at least one transformed feature map obtained by transforming the at least one second feature map using a transformation parameter; The determined deformation parameters are applied to the at least one second mapping image to align the at least one first mapping image with the at least one second mapping image. It has a control unit.

[0009] (2) An image processing device according to an embodiment of the present disclosure is the image processing device according to (1), The control unit evaluates a first dissimilarity between the at least one first feature map and the at least one transformed feature map using entropy as an index.

[0010] (3) An image processing device according to an embodiment of the present disclosure is the image processing device according to (2), The control unit uses, as the entropy, an entropy obtained when it is assumed that the feature quantity follows a specific probability distribution.

[0011] (4) An image processing device according to an embodiment of the present disclosure is the image processing device according to any one of (1) to (3), The control unit is acquiring a plurality of first mapping images obtained by a mapping measurement of the first region of the material, and a plurality of second mapping images obtained by a mapping measurement of the second region of the material; selecting the at least one first mapping image and the at least one second mapping image from among the first mapping images and the second mapping images based on a second degree of difference between each mapping image included in the acquired first mapping images and each mapping image included in the acquired second mapping images, respectively; The deformation parameters are applied to the second mapping images to align the first mapping images with the second mapping images.

[0012] (5) An image processing device according to an embodiment of the present disclosure is the image processing device according to (4), The control unit evaluates a second degree of dissimilarity between each mapping image included in the plurality of first mapping images and each mapping image included in the acquired plurality of second mapping images using texture as an index.

[0013] (6) An image processing device according to an embodiment of the present disclosure is the image processing device according to any one of (1) to (5), The control unit is filtering the at least one first mapping image and the at least one second mapping image in multiple stages, with the degree of filtering decreasing in each stage; generating the at least one first feature map and the at least one second feature map each time the at least one first mapping image and the at least one second mapping image are filtered; The transformation parameters are updated every time the at least one first feature map and the at least one second feature map are created.

[0014] (7) An image processing device according to an embodiment of the present disclosure is the image processing device according to any one of (1) to (6), Filtering the at least one first mapping image and the at least one second mapping image includes coarse-graining the at least one first mapping image and the at least one second mapping image.

[0015] (8) An image processing device according to an embodiment of the present disclosure is the image processing device according to any one of (1) to (7), Filtering the at least one first mapping image and the at least one second mapping image includes smoothing the at least one first mapping image and the at least one second mapping image.

[0016] (9) An image processing device according to an embodiment of the present disclosure is the image processing device according to any one of (1) to (8), The at least one first mapping image and the at least one second mapping image are, respectively, a mapping image obtained by a mapping measurement using a first instrument and a mapping image obtained by a mapping measurement using a second instrument different from the first instrument.

[0017] (10) An image processing device according to an embodiment of the present disclosure is the image processing device according to any one of (1) to (9), The at least one first mapping image and the at least one second mapping image are, respectively, a mapping image obtained by a mapping measurement at a first timing and a mapping image obtained by a mapping measurement at a second timing different from the first timing.

[0018] (11) An image processing device according to an embodiment of the present disclosure is the image processing device according to any one of (1) to (10), The material includes a metallic material.

[0019] (12) A mapping image alignment method according to an embodiment of the present disclosure, an image processing device filters at least one first mapping image obtained by a mapping measurement of a first region of a material and at least one second mapping image obtained by a mapping measurement of a second region of the material at least partially overlapping with the first region, and extracts features of the at least one first mapping image and the at least one second mapping image, thereby creating at least one first feature map and at least one second feature map, respectively; determining a transformation parameter based on a first dissimilarity between the at least one first feature map created by the image processing device and at least one transformed feature map obtained by transforming the at least one second feature map created by the image processing device using the transformation parameter; applying the determined deformation parameters to the at least one second mapping image by the image processing device, thereby aligning the at least one first mapping image with the at least one second mapping image; Includes.

[0020] (13) A program according to an embodiment of the present disclosure, creating at least one first feature map and at least one second feature map by filtering at least one first mapping image obtained by a mapping measurement of a first region of a material and at least one second mapping image obtained by a mapping measurement of a second region of the material that at least partially overlaps with the first region, and extracting features of the at least one first mapping image and the at least one second mapping image; determining a transformation parameter based on a first dissimilarity between the at least one first feature map and at least one transformed feature map obtained by transforming the at least one second feature map using the transformation parameter; applying the determined deformation parameters to the at least one second mapping image to align the at least one first mapping image with the at least one second mapping image; The method causes a computer to perform operations including: Effect of the Invention

[0021] According to the present disclosure, the alignment of a mapping image of a material can be performed with high precision. [Brief description of the drawings]

[0022] [Figure 1] 1 is a block diagram showing a configuration of an image processing system according to an embodiment of the present disclosure. [Diagram 2] 4 is a flowchart illustrating an operation of an image processing device according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a diagram showing a data flow corresponding to the operations shown in FIG. 2. [Figure 4] 3 is a flowchart showing a specific procedure of the determination step shown in FIG. 2. [Figure 5A] FIG. 13 is a diagram illustrating an example of a feature amount map. [Figure 5B] FIG. 13 is a diagram illustrating an example of a feature amount map. [Figure 5C] FIG. 13 is a diagram illustrating an example of a feature amount map. [Figure 5D] FIG. 13 is a diagram illustrating an example of a feature amount map. [Figure 5E] FIG. 13 is a diagram illustrating an example of a feature amount map. [Figure 6] 11 is a table showing an example of a result of alignment. [Figure 7] 11 is a table showing an example of a result of alignment. [Figure 8A] 11 is a table showing an example of a result of alignment. [Figure 8B] 11 is a table showing an example of a result of alignment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0024] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0025] The configuration of an image processing system 10 according to this embodiment will be described with reference to FIG.

[0026] The image processing system 10 includes an image processing device 20, a first device 31, and a second device 32. The image processing device 20 may be capable of communicating with the first device 31 and the second device 32 directly or via a network such as a LAN or the Internet. "LAN" is an abbreviation for local area network.

[0027] The image processing device 20 is a general-purpose computer such as a PC, a server computer such as a cloud server, a microcomputer installed in a mobile device such as a smartphone or a tablet, or a dedicated computer. "PC" is an abbreviation for personal computer.

[0028] The first device 31 and the second device 32 are devices used for mapping measurement of the material 40. The first device 31 and the second device 32 are devices for, for example, observing the structure morphology, measuring the element distribution for each structure, or measuring the crystal orientation or crystal structure, respectively. The device for observing the structure morphology is, for example, an optical microscope, SEM, or TEM. The device for measuring the element distribution for each structure is, for example, an SEM-EDS, EPMA, or XRF. The device for measuring the crystal orientation or crystal structure is, for example, EBSD. The second device 32 may be the same device as the first device 31, but is a different device from the first device 31 in this embodiment. For example, when the first device 31 is an EPMA, the second device 32 is a device other than the EPMA, such as an EBSD. The second device 32 may be integrated with the first device 31. That is, a single device may function as both the first device 31 and the second device 32. For example, an EPMA as the first device 31 may be supplemented with an EBSD as the second device 32 .

[0029] The configuration of an image processing device 20 according to this embodiment will be described with reference to FIG.

[0030] The image processing device 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, and an output unit 25.

[0031] The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 21 executes processes related to the operation of the image processing device 20 while controlling each part of the image processing device 20.

[0032] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, a ROM, or a flash memory. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The flash memory is, for example, an SSD. "SSD" is an abbreviation for solid-state drive. The magnetic memory is, for example, an HDD. "HDD" is an abbreviation for hard disk drive. The storage unit 22 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used in the operation of the image processing device 20 and data obtained by the operation of the image processing device 20.

[0033] The communication unit 23 includes at least one communication module. The communication module is, for example, a module compatible with a wired LAN communication standard such as Ethernet (registered trademark), a wireless LAN communication standard such as IEEE802.11, or a mobile communication standard such as LTE, 4G standard, or 5G standard. "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 23 may communicate with the first device 31 and the second device 32. The communication unit 23 receives data used in the operation of the image processing device 20 and transmits data obtained by the operation of the image processing device 20.

[0034] The input unit 24 includes at least one input device. The input device is, for example, a physical key, a capacitive key, a pointing device, a touch screen integral with a display, a camera, or a microphone. The input unit 24 accepts an operation to input data used for the operation of the image processing device 20. The input unit 24 may be connected to the image processing device 20 as an external input device instead of being provided in the image processing device 20. As a connection interface, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used. "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface.

[0035] The output unit 25 includes at least one output device. The output device is, for example, a display, a printer, or a speaker. The display is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescent. The output unit 25 outputs data obtained by the operation of the image processing device 20. The output unit 25 may be connected to the image processing device 20 as an external output device instead of being provided in the image processing device 20. As a connection interface, an interface compatible with standards such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0036] The functions of the image processing device 20 are realized by executing a program according to this embodiment in a processor serving as the control unit 21. That is, the functions of the image processing device 20 are realized by software. The program causes a computer to execute the operations of the image processing device 20, thereby causing the computer to function as the image processing device 20. That is, the computer functions as the image processing device 20 by executing the operations of the image processing device 20 in accordance with the program.

[0037] The program may be stored in a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include a flash memory, a magnetic recording device, an optical disk, a magneto-optical recording medium, or a ROM. The program may be distributed, for example, by selling, transferring, or lending a portable medium such as an SD card, DVD, or CD-ROM on which the program is stored. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer. The program may be provided as a program product.

[0038] A computer temporarily stores a program stored in a portable medium or transferred from a server in a main storage device. Then, the computer reads the program stored in the main storage device with a processor and executes processing according to the read program with the processor. The computer may read the program directly from a portable medium and execute processing according to the program. The computer may execute processing according to the received program each time a program is transferred from the server to the computer. Processing may be executed by a so-called ASP-type service that realizes functions only by execution instructions and result acquisition without transferring a program from the server to the computer. "ASP" is an abbreviation for application service provider. Programs include information used for processing by a computer and equivalent to a program. For example, data that is not a direct command to a computer but has the property of defining computer processing falls under " equivalent to a program.

[0039] A part or all of the functions of the image processing device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. In other words, a part or all of the functions of the image processing device 20 may be realized by hardware.

[0040] The present embodiment will be outlined with reference to FIG.

[0041] The control unit 21 of the image processing device 20 filters at least one first mapping image obtained by a mapping measurement of a first region 41 of the material 40 and at least one second mapping image obtained by a mapping measurement of a second region 42 of the material 40 that at least partially overlaps with the first region 41. The control unit 21 of the image processing device 20 extracts features of the at least one filtered first mapping image and at least one filtered second mapping image to create at least one first feature map and at least one second feature map, respectively. The number of first feature maps created may be the same as the number of filtered first mapping images, or may be greater than the number of filtered first mapping images. The number of second feature maps created may also be the same as the number of filtered second mapping images, or may be greater than the number of filtered first mapping images. The control unit 21 of the image processing device 20 determines the deformation parameter θ based on the first dissimilarity between at least one created first feature map and at least one transformed feature map obtained by transforming the created at least one second feature map using the deformation parameter θ. Regarding the first dissimilarity, in order to determine the deformation parameter θ, at first, at least one first feature map may be compared with at least one second feature map before being transformed using the deformation parameter θ. That is, the control unit 21 of the image processing device 20 may initially evaluate the first dissimilarity between at least one first feature map and at least one second feature map, and then evaluate the first dissimilarity between at least one first feature map and at least one transformed feature map so that the first dissimilarity decreases therefrom. The control unit 21 of the image processing device 20 applies the determined deformation parameter θ to at least one second mapping image to align at least one first mapping image and at least one second mapping image. Therefore, according to this embodiment, the alignment of the mapping image of the material 40 can be performed with high accuracy.

[0042] In this embodiment, filtering the at least one first mapping image and the at least one second mapping image includes coarse-graining the at least one first mapping image and the at least one second mapping image. In a variation of this embodiment, filtering the at least one first mapping image and the at least one second mapping image may include smoothing the at least one first mapping image and the at least one second mapping image.

[0043] For example, the at least one first mapping image and the at least one second mapping image are a mapping image obtained by a mapping measurement using a first device 31 and a mapping image obtained by a mapping measurement using a second device 32 different from the first device 31, respectively. Alternatively, the at least one first mapping image and the at least one second mapping image may be a mapping image obtained by a mapping measurement at a first timing and a mapping image obtained by a mapping measurement at a second timing different from the first timing. For example, the first device 31 is commonly used for the mapping measurement at the first timing and the mapping measurement at the second timing. Alternatively, the second device 32 may be commonly used for the mapping measurement at the first timing and the mapping measurement at the second timing.

[0044] In this embodiment, the material 40 includes a metal material. For example, the material 40 includes a steel material. In the development of metal materials, particularly steel materials, various micro-scale analyses are performed to evaluate the relationship between the structure and the properties. For this reason, the method according to this embodiment can be suitably applied.

[0045] Preferably, the control unit 21 of the image processing device 20 evaluates the first dissimilarity between at least one first feature map and at least one transformed feature map using entropy as an index. That is, it is preferable to use entropy for evaluating the first dissimilarity. According to such an example, the robustness of the registration is further improved.

[0046] More preferably, the control unit 21 of the image processing device 20 uses, as the entropy, an entropy obtained when it is assumed that the feature quantity follows a specific probability distribution. In other words, it is preferable to approximately evaluate the entropy by assuming that the feature quantity follows a specific probability distribution. According to such an example, an analytical solution can be used, so that the calculation cost can be reduced compared to the case where the entropy is directly calculated. The entropy is, for example, a conditional entropy or a mutual entropy.

[0047] Preferably, the control unit 21 of the image processing device 20 acquires a plurality of first mapping images obtained by a mapping measurement of the first region 41 of the material 40 and a plurality of second mapping images obtained by a mapping measurement of the second region 42 of the material 40. The control unit 21 of the image processing device 20 selects at least one first mapping image and at least one second mapping image from among the plurality of first mapping images and the plurality of second mapping images, respectively, based on a second dissimilarity between each mapping image included in the acquired plurality of first mapping images and each mapping image included in the acquired plurality of second mapping images. The control unit 21 of the image processing device 20 aligns the plurality of first mapping images and the plurality of second mapping images by applying the deformation parameter θ to the plurality of second mapping images. According to such an example, by selecting an image effective for alignment, alignment can be performed using only images effective for alignment, thereby increasing efficiency and success rate.

[0048] For example, the control unit 21 of the image processing device 20 evaluates the second dissimilarity between each mapping image included in the multiple first mapping images and each mapping image included in the multiple acquired second mapping images using texture as an index.

[0049] The control unit 21 of the image processing device 20 may filter at least one first mapping image and at least one second mapping image only once, but preferably filters at least one first mapping image and at least one second mapping image in multiple stages while decreasing the degree of filtering step by step. For example, the control unit 21 of the image processing device 20 coarse-grains at least one first mapping image and at least one second mapping image in multiple stages while decreasing the degree of coarse-graining step by step. Alternatively, the control unit 21 of the image processing device 20 smoothes at least one first mapping image and at least one second mapping image in multiple stages while decreasing the degree of smoothing step by step. The control unit 21 of the image processing device 20 creates at least one first feature map and at least one second feature map every time it filters at least one first mapping image and at least one second mapping image. The control unit 21 of the image processing device 20 updates the deformation parameter θ every time at least one first feature map and at least one second feature map are created. According to such an example, even if a large number of similar crystal grains exist in the material 40, the alignment of the mapping image of the material 40 can be performed with high accuracy.

[0050] Here, the findings of the inventors will be described by taking as an example a case where the relationship between the distribution of elements and the distribution of crystal orientation in a metal material is analyzed. 1 A ,D 2 A A first mapping image group D consisting of A and the second mapping image D 1 B ,D 2 B A second mapping image group D consisting of BAs shown in FIG. 1, there are many cases where misalignment occurs between the mapping images, that is, where the field of view is shifted. Suppose that alignment is performed by deforming one of the mapping images so that the two mapping images match based on the first dissimilarity between the mapping images calculated in pixel units or the local feature amount for each feature point of the mapping images. In such a case, the appearance of the structural morphology often differs between the mapping image of the element distribution and the mapping image of the crystal orientation distribution, and there is a limit to achieving automatic and high-precision alignment. Therefore, the inventors have conducted extensive research in aligning the obtained mapping images, and have found that it is important to perform coarse graining or smoothing of the obtained mapping images and create a feature amount map to perform alignment.

[0051] The operation of the image processing device 20 according to this embodiment will be described with reference to Figures 2 and 3. The operation described below corresponds to the mapping image alignment method according to this embodiment.

[0052] As shown in FIG. 2, the mapping image alignment method according to this embodiment includes an acquisition step S1, a coarse-graining step S3, a feature amount map creation step S4, a determination step S5, a transformation step S6, and an output step S7.

[0053] The acquisition step S1 is to obtain a first mapping image group D for the material 40 to be evaluated. A and the second mapping image group D B The first mapping image group D A corresponds to the plurality of first mapping images described above. B corresponds to the above-mentioned plurality of second mapping images.

[0054] The coarse graining step S3 is a first mapping image group D A A first mapping image group D of the reference selected from A* and the second mapping image group D B A second mapping image group D to be aligned selected from B* This is a process of coarse-graining the first mapping image group D A*corresponds to at least one of the first mapping images described above. A* Instead of the first mapping image, a single reference may be used. B* corresponds to at least one second mapping image described above. B* Alternatively, a second mapping image of a single registration target may be used. A and the second mapping image group D B Instead of acquiring the first mapping image group D A* and the second mapping image group D to be aligned. B* According to such an example, a first mapping image group D A* and the second mapping image group D to be aligned. B* You no longer have to choose between and .

[0055] The feature map creation step S4 is a coarse-grained reference first mapping image group D A* The first feature map group F A* and a second mapping image group D of the coarse-grained alignment target. B* The second feature map group F B* This is a process of creating a first feature map group F A* corresponds to at least one first feature map described above. A* Instead of the second feature map group F, a single first feature map may be used. B* corresponds to at least one second feature map described above. B* Alternatively, a single second feature map may be used.

[0056] The determination step S5 is a first feature map group F A* and the second feature map group F B* Based on the first dissimilarity obtained from B*This is a process of determining a transformation parameter θ for transforming the second feature map group F after transformation. B* The deformation feature map set F (B*,deform) corresponds to at least one of the deformation feature maps described above. (B*,deform) Alternatively, a single deformation feature map may be used.

[0057] The deformation step S6 is to deform the second mapping image group D using the deformation parameter θ. B This is a process of transforming the above.

[0058] In the output step S7, the second mapping image group D after deformation is B A group of deformation mapping images D (B,deform) This is a process of outputting the signal.

[0059] As shown in FIG. 2, the mapping image alignment method according to this embodiment further includes a selection step S2 after the acquisition step S1 and before the coarse-graining step S3.

[0060] The selection step S2 is a first mapping image group D A and the second mapping image group D B Based on the texture of the image, a first mapping image group D is created. A* and the second mapping image group D to be aligned. B* and the like.

[0061] As a modification of this embodiment, the selection step S2 may be omitted. For example, A The first mapping image group D is based on all of the above. A* The second mapping image group D B All of the second mapping images D are to be aligned. B* may be selected as

[0062] As another modified example of this embodiment, the coarse-graining step S3 and the feature map creation step S4 may be interchanged, and the coarse-graining step S3 may be performed after the feature map creation step S4. For example, instead of performing coarse-graining by average pooling and then performing feature extraction by gradient or Hessian matrix to create a feature map, the order may be reversed, and feature extraction by gradient or Hessian matrix may be performed, and then coarse-graining by average pooling may be performed to create a feature map. That is, the control unit 21 of the image processing device 20 may extract features of at least one first mapping image and at least one second mapping image, and then perform a filtering process to create at least one first feature map and at least one second feature map, respectively. However, from the viewpoint of calculation efficiency, it is desirable to extract features after the filtering process.

[0063] The obtaining step S1 will now be described in detail.

[0064] In the acquisition step S1, the control unit 21 acquires a first mapping image group D obtained for the material 40 to be evaluated. A and the second mapping image group D B The first mapping image group D is obtained from the storage unit 22 or from the first device 31 and the second device 32 via the communication unit 23. A are mapping image groups obtained by the same analytical instrument and without positional deviation. B The first mapping image group D is obtained by the same analytical instrument and is not displaced. A Regarding the first device 31, the second mapping image group D B The second device 32 is the first mapping image group D A and the second mapping image group D B is the first mapping image group D A and the second mapping image group D B As a modification of this embodiment, the first mapping image group D A and the second mapping image group D BIf the analytical instruments used to obtain the first mapping image group D and the second mapping image group D are the same, A and the second mapping image group D B are mapping image groups with positional deviations obtained at different measurement times. First mapping image group D A Two or more first mapping images D k A is data on any dimension. k represents the type of the first mapping image. First mapping image group D A Instead of one first mapping image D k A In such an example, it is not necessary to select the first mapping image. B Two or more second mapping images D k’ B k' represents the type of the second mapping image. The second mapping image group D B Instead of one second mapping image D k’ B may be used. In such an example, there is no need to select a second mapping image.

[0065] Specific examples of mapping images include images based on information on structural morphology, distribution information on constituent elements, or information on crystal orientation or crystal structure. Information on structural morphology can be obtained, for example, by a microscope such as an optical microscope, SEM, or TEM. Distribution information on constituent elements can be obtained, for example, by SEM-EDS, EPMA, or XRF. Information on crystal orientation or crystal structure can be obtained, for example, by EBSD. The mapping image may be obtained by an analytical instrument and then subjected to image processing or some other processing.

[0066] The selection step S2 will now be described in detail.

[0067] In the selection step S2, the control unit 21 selects the first mapping image group D A and the second mapping image group D B Based on the texture of the image, a first mapping image group D is A*and the second mapping image group D to be aligned. B* Select " and " respectively.

[0068] Instead of performing the selection step S2, the first mapping image group D A and the second mapping image group D B The first mapping image group and the second mapping image group, which are estimated to have a high correlation from the above, are used as the first mapping image group D A* and a second mapping image group D to be aligned B* In such an example, the control unit 21 receives an operation to select the first mapping image group and the second mapping image group via the input unit 24.

[0069] By carrying out the selection step S2, the calculation cost can be reduced and the success rate of the alignment can be increased. For example, by using an index of the texture of the image, a first mapping image group D is selected from a combination of mapping image groups having similar textures. A* and the second mapping image group D to be aligned. B* When the first mapping image D k A and the second mapping image D k’ B Even if the combination of A* and the second mapping image group D to be aligned. B* As a result, the efficiency and success rate of registration can be improved. The texture of an image can be evaluated by, for example, a power spectrum obtained by texture analysis or Fourier transform, or image statistics obtained from a histogram.

[0070] In the following, a case will be described in which the texture of an image is evaluated using the second dissimilarity obtained by texture analysis using the GLCM. "GLCM" is an abbreviation for gray-level co-occurrence matrix.

[0071] First, the control unit 21 selects a first mapping image group D A The first mapping image D k A (D 1 A ,D 2 A , ) and the second mapping image group D B The second mapping image D k’ B (D 1 B ,D 2 B , ... k A (P 1 A ,P 2 A , ) and P k’ B (P 1 B ,P 2 B ,···).

[0072] Next, the control unit 21 selects the first mapping image D 1 A ,D 2 A , and the second mapping image D 1 B ,D 2 B , ... k A (P 1 A ,P 2 A , ) and P k’ B (P 1 B ,P 2 B , ···) to calculate the JS divergence. "JS" is an abbreviation for Jensen-Shannon. The control unit 21 defines the calculated JS divergence value as a second dissimilarity between the mapping images and evaluates it.

[0073] The control unit 21 selects a first mapping image group D based on the mapping image group of the combination with the small second dissimilarity degree obtained. A* and a second mapping image group D to be aligned B* The first mapping image group D of the reference is selected as A* The first mapping image D of two or more references k A* The first mapping image group D of the reference A* Instead of the first mapping image D k A* may be selected. B* Also, two or more second mapping images D k’ B* The second mapping image group D to be aligned is included. B* Instead of the first mapping image D k’ B* may be selected.

[0074] The coarse-graining step S3 will now be described in detail.

[0075] In the coarse graining step S3, the control unit 21 selects the first mapping image group D A A first mapping image group D of the reference selected from A* and the second mapping image group D B A second mapping image group D to be aligned selected from B* The first mapping image group D A* is the first mapping image group D A Similarly, the second mapping image group D B* is the second mapping image group D B may be also possible.

[0076] Coarse graining is a process of reducing the resolution of a mapping image and removing fine tissue information. Methods of coarse graining of a mapping image include, for example, average pooling or maximum pooling. As a modified example of this embodiment, smoothing may be performed instead of coarse graining in order to remove minute tissue information. Methods of smoothing of a mapping image include, for example, a method using a moving average filter, a Gaussian filter, or a median filter. From the viewpoint of calculation costs, it is more preferable to perform coarse graining.

[0077] By removing fine morphological information by coarse-graining the mapping image and extracting coarse morphological information, for example, in a mapping image of a metal material where there are many crystal grains with similar shapes, it is possible to prevent alignment between separate crystal grains, resulting in a more robust alignment.

[0078] It is desirable to start with the degree of coarse-graining at a level where fine morphological information is crushed to represent broad-area morphological information, and then gradually update the degree of coarse-graining in one or more steps to a level where features that are sufficiently finer than the accuracy of the previous alignment stage are crushed.

[0079] The feature amount map creation step S4 will now be described in detail.

[0080] In the feature map creation step S4, the control unit 21 creates a coarse-grained reference first mapping image group D A* The first feature map group F A* and a second mapping image group D of the coarse-grained alignment target. B* The second feature map group F B* The first feature map group F A* and the second feature map group F B* The feature map included in each of and is a map created by extracting one type of feature or two or more types of feature. As one of the features, an identity map obtained by an operation of outputting the input as it is may be used.

[0081] For example, an image filter can be used to extract the feature amount. Specific examples of the image filter include a filter that smoothes an image, such as a Gaussian filter or a median filter, a filter that extracts the gradient or edge of an image, such as a Sobel filter, a filter that extracts the second derivative of an image, such as a Laplacian filter or a Hessian matrix filter, or a Gabor filter. Cornerness used in a Harris corner detector may be extracted as the feature amount. It is desirable for the image filter to have translation invariance and rotation invariance.

[0082] The control unit 21 selects a first mapping image group D A* First mapping image D k A* For each feature m, extract the feature m and create a first feature map F (k,m) A* A first feature map group F A* The control unit 21 creates a second mapping image group D B* The second mapping image D k’ B* For each feature m, we extract the feature m and create a second feature map F (k’,m) B* A second feature map group F B* where m represents the type of feature.

[0083] Specifically, the control unit 21 generates a mapping image D k i The first derivative of D (k,x) i ,D (k,y) i and second derivative D (k,xx) i ,D (k,yy) i ,D (k,xy) i Then, all or some of the four types of features are extracted as follows, where i is A* or B*.

[0084] The control unit 21 calculates the gradient F of each coordinate (x, y) which is a feature quantity. (k,grad) i(x,y) is extracted using the following formula:

number

[0085] The control unit 21 calculates the matrix norm F of the Hessian matrix, which is a feature quantity. (k,norm) i is extracted using the following formula:

number

[0086] The control unit 21 calculates the cornerness F of the feature used in the Harris corner detector. (k,Harris) i is extracted using the following formula:

number

[0087] The control unit 21 calculates the Gabor energy F (k,GE) i Let g be the Gabor filter with standard deviation σ, wave number k, direction θ, and phase φ. Gabor Extract using (x, y; σ, k, θ, φ) using the following formula.

number

[0088] When the correlation of the feature map for the material 40 is unknown, it is better to have many types of feature since the appropriate type of feature is unknown. However, if there are too many types of feature, it is not preferable because the calculation efficiency decreases or unnecessary information increases.

[0089] The obtained feature map may be normalized, standardized, or power transformed to make the distribution of the feature map easier to handle, assuming that the units of the features are different or the magnitude of the values ​​is significantly different. The power transformation can play a role in justifying the assumption that the features follow a Gaussian distribution, as described below.

[0090] In the following, a case where the Yeo-Johnson transform, which is an example of the power transform, is used will be described.

[0091] The control unit 21 calculates the feature quantity F of each coordinate (x, y) by using the following formula: (k,m) i (x, y) is calculated as a feature F (k,m) i (here the "^" above the "F" is omitted).

number

[0092] As described above, the obtained features are transformed by normalization, standardization, or power transformation, and the set of transformed features is called a feature map group F i It may also be used as.

[0093] The determination step S5 will now be described in detail.

[0094] In the determination step S5, the control unit 21 determines whether the first feature map group F obtained in the feature map creation step S4 is a A* and the second feature map group F B* Based on the first dissimilarity obtained from B* A transformation parameter θ for transforming is determined.

[0095] Second feature map group F B* As a method for transforming, a global transformation such as an affine transformation or a projective transformation, or a local transformation such as a free-form transformation using a B-spline can be used.

[0096] The control unit 21 outputs the second feature amount map group F after the transformation. B* The deformation feature map set F (B*,deform) and the first feature map group F A* The control unit 21 evaluates the first degree of difference between the second feature amount map group F and the first feature amount map group F such that the first degree of difference is minimized. B* The above transformation is repeated to determine the transformation parameter θ that minimizes the first dissimilarity. The minimal value may be a quasi-optimal solution. The first similarity may be defined as the first dissimilarity with its sign inverted, and the control unit 21 may determine the transformation parameter θ that maximizes the first similarity.

[0097] The first dissimilarity can be evaluated using, for example, SSD, SAD, ZNCC, the amount of information defined in information theory, or entropy. "SSD" is an abbreviation for sum of squared differences. "SAD" is an abbreviation for sum of absolute differences. "ZNCC" is an abbreviation for zero-means normalized cross-correlation.

[0098] For example, the control unit 21 can evaluate the first dissimilarity degree based on the weighted average of SSD according to the following formula.

number

[0099] In the above formula, w kmk’m’ is the first feature map F (k,m) A* (x,y) and the second feature map F (k’,m’) (B*,deform) It is preferable to set the weight between feature amount maps with similar features larger than the other weights, and set the weight between feature amount maps with different features to zero.

[0100] The first dissimilarity obtained using SSD, SAD, or ZNCC can evaluate the first dissimilarity between feature maps in a one-to-one correspondence. Therefore, if the combination of feature maps with strong correlations is known empirically, it is possible to select an appropriate w kmk’m’ By selecting the above, effective alignment is possible. On the other hand, when entropy or the amount of information defined by information theory is used to evaluate the first dissimilarity, accurate alignment is possible even for maps with different detailed shapes or features. In other words, the range of pixel misalignment during alignment can be narrowed. In addition, even if the combination of feature maps with strong correlations is unknown, highly robust alignment is possible because the correlation of any combination of features is naturally taken into account. For this reason, it is preferable to use entropy to evaluate the first dissimilarity. When entropy is used, when the number of feature maps included in the group of feature maps is F A* and F B* Even if the above differs, a univocal definition is possible.

[0101] When multiple types of features are used, the first dissimilarity H(P (A*,B*) |P B* ) is preferably used for evaluation.

number

[0102] If the obtained features are transformed by normalization, standardization, or power transformation, F km A* and F k’m’ B* The transformed feature map F km A* (Here, the "^" above "F" is omitted) and F k’m’ B* (Here the "^" above the "F" is omitted) is used.

[0103] If the above formula is calculated directly, the amount of calculation increases exponentially with the number M of feature maps to be considered. To avoid this, it is possible to approximately evaluate the above formula by using the assumption that the features follow a specific statistical distribution, for example, a Gaussian distribution. As a result, it is possible to significantly reduce the calculation cost of the above formula.

[0104] For example, assume that the features follow a multivariate Gaussian distribution. A* and the deformation feature map group F (B*,deform) The variance-covariance matrices of A*B* and Σ B* When the above equation is given by (A*,B*) and s B* can be approximated by the following equations, respectively.

number

[0105] The control unit 21 updates the transformation parameter θ while B* The above transformation is repeated to determine a transformation parameter θ that minimizes the first dissimilarity. A general optimization method such as the Nelder-Mead method or Bayesian optimization can be applied to determine the transformation parameter θ.

[0106] A specific procedure of the determination step S5 is shown in FIG. 4. In step S51, the control unit 21 performs deformation using the deformation parameter θ. In step S52, the control unit 21 evaluates the first dissimilarity. If the convergence conditions, such as the amount of change in the first dissimilarity, are not satisfied, in step S53, the control unit 21 updates the deformation parameter θ. Then, the control unit 21 performs step S51 again. If the convergence conditions are satisfied but the degree of coarse-graining is equal to or greater than the reference value, in step S54, the control unit 21 reduces the degree of coarse-graining. Then, the control unit 21 performs the coarse-graining step S3 to the determination step S5 again. On the other hand, if the convergence conditions are satisfied and the degree of coarse-graining is less than the reference value, the control unit 21 finalizes the deformation parameter θ.

[0107] As described above, the control unit 21 gradually reduces the degree of coarse-graining, repeats the steps from the coarse-graining step S3 to the determination step S5, and completes updating the deformation parameter θ when the degree of coarse-graining is reduced to a level at which morphological information with the required alignment accuracy is expressed. Thereafter, the obtained deformation parameter θ is used to perform the deformation step S6, thereby enabling high-precision alignment.

[0108] In the deformation step S6, the control unit 21 determines the second mapping image group D using the deformation parameter θ obtained in the determination step S5. B For example, the second mapping image group D B When transforming, the transformation parameter θ is a projective transformation matrix.

[0109] In the output step S7, the control unit 21 outputs the second mapping image group D after deformation obtained in the deformation step S6. B A group of deformation mapping images D (B,deform) via the communication unit 23 or the output unit 25. The control unit 21 outputs the first mapping image group D A may be output together. EXAMPLES

[0110] The first embodiment will be described below.

[0111] Using the mapping image alignment method according to the present disclosure, alignment was performed between a mapping image obtained by EPMA and a mapping image obtained by EBSD for a steel material.

[0112] In the acquisition step S1, a first mapping image D is obtained using an EPMA. 1 A and the second mapping image group D was obtained using EBSD. B The first mapping image D was obtained. 1 A The second mapping image group D was used to obtain the concentration map C. B The IQ map, misorientation map, and IPF map were obtained as the second mapping image D 1 B ,D 2 B ,D 3 B is equivalent to.

[0113] In the selection step S2, the first mapping image D 1 A The first mapping image D is the reference image. k A* The C concentration map is the second mapping image group D B to the second mapping image D to be aligned k’ B* We selected IQ Map as the model.

[0114] In the coarse-graining step S3, the selected mapping image was subjected to coarse-graining by 8x8 average pooling.

[0115] In the feature map creation step S4, the identity map, gradient, norm of the Hessian matrix, Harris cornerness, and Gabor energy are extracted as features for each of the coarse-grained C concentration map and IQ map, and a first feature map group F A* and the second feature map group F B* obtained.

[0116] Figures 5A to 5E show examples of feature maps created from the C concentration map. Figure 5A shows an example of a feature map corresponding to the identity mapping, Figure 5B shows an example of a feature map corresponding to the gradient, Figure 5C shows an example of a feature map corresponding to the norm of the Hessian matrix, Figure 5D shows an example of a feature map corresponding to the Harris cornerness, and Figure 5E shows an example of a feature map corresponding to the Gabor energy. Figures 5A to 5E show that abstracted tissue morphology information is obtained by the feature map creation step S4.

[0117] The feature map was subjected to a power transformation to approximate the feature to a multivariate Gaussian distribution, and the conditional entropy was calculated approximately by assuming the feature to be a multivariate Gaussian distribution using the formula described above. The obtained conditional entropy was used as the first dissimilarity measure.

[0118] In the determination step S5, the feature map group obtained from the IQ map is projectively transformed to obtain a deformed feature map group F (B*,deform) Then, the first feature map group F obtained from the C map is created. A* and the created deformation feature map group F (B*,deform) The projection transformation matrix, which is the transformation parameter θ, was determined so that the first dissimilarity between the

[0119] Next, the process returned to the coarse-graining step S3, and coarse-graining was performed by 4x4 average pooling, and the feature amount map creation step S4 and the determination step S5 were performed.The process further returned to the coarse-graining step S3, and coarse-graining was performed by 2x2 average pooling, and the feature amount map creation step S4 and the determination step S5 were performed.The process returned to the coarse-graining step S3 again, and the feature amount map creation step S4 and the determination step S5 were performed on the mapping image that was not coarse-grained.

[0120] In the deformation step S6, the second mapping image group D is determined by using the deformation parameter θ obtained in the final determination step S5. B The following transformation was performed.

[0121] In the output step S7, the transformed IQ map, orientation difference map, and IPF map were output.

[0122] Similarly, when the combination of the C concentration map and the orientation difference map, and the combination of the C concentration map and the IPF map were selected in the selection step S2, the coarse graining step S3 to the output step S7 were also performed. (B,deform) Mapping image D k’ (B,deform) First mapping image D 1 A The deformation mapping image group D is a superimposed image. (B,deform) is the second mapping image D before deformation. k’ B is the second mapping image D to be aligned. k’ B* The "Evaluation" in Fig. 6 shows that A is the best, followed by B and C in order. It can be seen that in all combinations, the present method was able to perform highly accurate registration.

[0123] First mapping image D 1 A The effectiveness of this method was also examined when a Mn concentration map was used instead of a C concentration map. The corresponding alignment results are shown in Figure 6. Although the C concentration map and the Mn concentration map are mapping images obtained with the same EPMA, there is a misalignment due to the difference in measurement timing. Figure 6 shows that high-precision alignment is possible even for mapping images in which there is no clear tissue correspondence, such as a combination of a Mn concentration map and an IPF map.

[0124] The second embodiment will now be described.

[0125] Using the mapping image alignment method according to the present disclosure, alignment was performed between a mapping image obtained by EPMA and a mapping image obtained by EBSD for a steel material.

[0126] In the acquisition step S1, a first mapping image group D is obtained using an EPMA. A and the second mapping image group D was obtained using EBSD. B First mapping image group D AThe C concentration map and the COMPO image were obtained as the first mapping image D 1 A ,D 2 A The second mapping image group D B The IQ map, IPF map, and KAM map were obtained as the second mapping image D. 1 B ,D 2 B ,D 3 B is equivalent to.

[0127] In the selection step S2, a first mapping image group D is selected based on mapping images having similar textures. A* and a second mapping image group D to be aligned B* Specifically, for all the acquired mapping images, GLCM (P k i ) and the obtained P k i JS divergence is calculated using the first mapping image group D A All mapping images of the first group and the second mapping image group D B The second dissimilarity between all combinations of the COMPO image and the IPF map was evaluated. Then, for all combinations of the reference mapping image and the mapping image to be aligned, the same coarse graining step S3 to output step S7 as in the first embodiment were performed. The results are shown in Fig. 7. As in Fig. 6, the "evaluation" in Fig. 7 indicates that A is the best, followed by B and C in that order. It can be seen from Fig. 7 that high-precision alignment was achieved for the combination of the COMPO image and the IPF map with the lowest second dissimilarity.

[0128] From this, it can be seen that by selecting a combination of mapping images having similar image textures in the selection step S2, it is possible to automatically select an appropriate reference mapping image and a mapping image to be aligned.

[0129] The third embodiment will now be described.

[0130] The mapping image alignment method according to the present disclosure was compared with other methods. The mapping image was a first mapping image D obtained by EPMA for a steel material. 1 A and the second mapping image group D obtained by EBSD. B Or the second mapping image group D obtained by EPMA at a different timing B Specifically, the first mapping image D 1 A The second mapping image group D B The IQ map, the orientation difference map, and the IPF map were obtained as the first mapping image, and the Mn concentration map was obtained as the second mapping image. A* and the second mapping image group D to be aligned. B* As combinations of the C concentration map and the Mn concentration map, the C concentration map and the IQ map, the C concentration map and the orientation difference map, and the C concentration map and the IPF map were selected and compared.

[0131] For comparison, we performed alignment using ORB local features, which is a method using feature points, and projective transformation. "ORB" is an abbreviation for oriented FAST and rotated BRIEF. "FAST" is an abbreviation for features from accelerated segment test. "BRIEF" is an abbreviation for Binary Robust Independent Elementary Features. First, feature points are extracted and feature amounts are calculated using the ORB method, and the first mapping image D k A* and the second mapping image group D to be aligned. B* A combination of feature points with similar feature values ​​was selected. The optimal projection transformation matrix was estimated based on the coordinates of the selected combination, and the second mapping image group D Bwas subjected to projective transformation. The results are shown in Fig. 8A. In Fig. 8A, "MO" refers to the orientation difference map. As in Fig. 6 and Fig. 7, the "evaluation" in Fig. 8A indicates that A is the best, followed by B, C, and so on. From No. 1 in Fig. 8A, it can be seen that it is difficult to align the C concentration map with the IQ map, orientation difference map, or map in which the tissue appears differently, such as the IPF map, using the alignment method based on local features. It can also be seen that it is difficult to align the C concentration map with the Mn concentration map.

[0132] Next, the first dissimilarity or first similarity between the data of the mapping images was evaluated from the pixel values ​​of each pixel using SSD, SAD, ZNCC, conditional entropy, and conditional entropy approximately calculated by assuming a Gaussian distribution, and alignment was performed. The results are shown in Figures 8A and 8B. For No. 2 to No. 4 in Figure 8A, the first mapping image D k A* and the second mapping image group D to be aligned. B* The first feature map group F A* and the second feature map group F B* For No. 5 in FIG. 8A and No. 6 to No. 9 in FIG. 8B, the first feature map group F A* and the second feature map group F B* was defined.

[0133] Regarding the calculation of the first dissimilarity, in No. 2, the first dissimilarity H(P (A*,B*) |P B* ) and the formula s (A*,B*) and s B* The conditional entropy calculated by the formula was used. For No. 3 to No. 5 and No. 9, the conditional entropy H(P (A*,B*) |P B* ) and s (A*,B*) and s B*was approximated by Gaussian integral. In No. 6 to No. 8, the first dissimilarity or first similarity was evaluated by the weighted average of SAD, SSD, and ZNNC, respectively. However, in the calculation of the conditional entropy in No. 2, the value of the mapping image was divided into 8 parts and the probability P (A*,B*) kmk’m’ and P B* km In the calculation of the weighted average of SAD, SSD, and ZNNC in No. 6 to No. 8, the weights were set to w for the feature maps using the same image filters. kmk’m’ = 1(m = m'), and the other weights are w kmk’m’ =0 (m≠m').

[0134] Then, the second feature map group F B* was transformed by projective transformation, and the optimal projective transformation matrix was determined so that the first dissimilarity was minimized or the first similarity was maximized, and alignment was performed. However, in No. 2, No. 4, No. 6 to No. 9, the second feature map group F was created after coarse-graining by 8x8 average pooling. B* The degree of coarse graining was gradually reduced to 4x4, then 2x2, and finally the alignment was performed without coarse graining, which was used as the final result.

[0135] Figures 8A and 8B show the results of overlaying the mapping images of the same field of view estimated using each method. Looking at No. 2 to No. 5 and No. 9, which are the results using the conditional entropy or its approximation, there is no mapping image in No. 3 where coarse graining and feature extraction were not performed. There are mapping images in No. 2, No. 4 and No. 5 where only coarse graining and feature extraction were performed where there are partially successful registrations, but it is difficult to perform registrations for all combinations of the C concentration map and Mn concentration map, the C concentration map and IQ concentration map, and the C concentration map and orientation difference map, and it is clear that the robustness is poor. On the other hand, in No. 9 where both coarse graining and feature extraction were performed, registrations were performed for all combinations of the C concentration map and Mn concentration map, the C concentration map and IQ concentration map, and the C concentration map and orientation difference map. Looking at No. 6 to No. 8, since both coarse graining and feature extraction are performed, it is possible to perform at least a certain degree of alignment in all of the combinations of the C concentration map and the Mn concentration map, the C concentration map and the IQ concentration map, and the C concentration map and the orientation difference map. This shows that it is important to combine both coarse graining and feature extraction. In addition, in No. 9, which uses the approximated conditional entropy, coarse graining, and feature extraction, it is possible to perform high-precision alignment in all of the combinations of the C concentration map and the Mn concentration map, the C concentration map and the IQ concentration map, and the C concentration map and the orientation difference map. Moreover, in No. 9, it is possible to perform high-precision alignment even in the combination of the C concentration map and the IPF map. Therefore, it is possible to use the conditional entropy as the first dissimilarity. In No. 6 to No. 8, it is difficult to perform alignment in the combination of the C concentration map and the IPF map, but by applying the selection step S2, it is possible to obtain the reference first mapping image D A* and the second mapping image group D to be aligned. B* It is possible to avoid using a combination of a C concentration map and an IPF map as a combination of the above.

[0136] The present disclosure is not limited to the above-mentioned embodiment. For example, two or more blocks shown in the block diagram may be integrated, or one block may be divided. Instead of executing two or more steps shown in the flowchart in chronological order as described, they may be executed in parallel or in a different order depending on the processing capacity of the device executing each step, or as necessary. Other modifications are possible without departing from the spirit of the present disclosure. [Explanation of symbols]

[0137] 10 Image Processing System 20 Image processing device 21 Control section 22 Memory section 23 Communications Department 24 Input section 25 Output section 31 1st device 32 2nd device 40 materials 41 First area 42 Second area

Claims

1. filtering at least one first mapping image obtained by a mapping measurement of a first region of a material and at least one second mapping image obtained by a mapping measurement of a second region of the material at least partially overlapping with the first region, and extracting features of the at least one first mapping image and the at least one second mapping image, thereby generating at least one first feature map and at least one second feature map, respectively; determining a transformation parameter based on a first dissimilarity between the at least one first feature map and at least one transformed feature map obtained by transforming the at least one second feature map using a transformation parameter; The determined deformation parameters are applied to the at least one second mapping image to align the at least one first mapping image with the at least one second mapping image. An image processing device having a control unit.

2. The image processing device according to claim 1 , wherein the control unit evaluates a first degree of dissimilarity between the at least one first feature map and the at least one transformed feature map using entropy as an index.

3. The image processing device according to claim 2 , wherein the control unit uses, as the entropy, an entropy obtained when it is assumed that the feature quantity follows a specific probability distribution.

4. The control unit is acquiring a plurality of first mapping images obtained by a mapping measurement of the first region of the material, and a plurality of second mapping images obtained by a mapping measurement of the second region of the material; selecting the at least one first mapping image and the at least one second mapping image from among the first mapping images and the second mapping images based on a second dissimilarity between each mapping image included in the acquired first mapping images and each mapping image included in the acquired second mapping images, The image processing apparatus according to claim 1 , further comprising: applying the deformation parameters to the second mapping images to align the first mapping images with the second mapping images.

5. 5. The image processing device according to claim 4, wherein the control unit evaluates a second degree of difference between each mapping image included in the plurality of first mapping images and each mapping image included in the plurality of acquired second mapping images using texture as an index.

6. The control unit is filtering the at least one first mapping image and the at least one second mapping image in multiple stages, with the degree of filtering decreasing in each stage; generating the at least one first feature map and the at least one second feature map each time the at least one first mapping image and the at least one second mapping image are filtered; The image processing apparatus according to claim 1 , wherein the transformation parameters are updated every time the at least one first feature map and the at least one second feature map are created.

7. 2. The image processing apparatus of claim 1, wherein filtering the at least one first mapping image and the at least one second mapping image comprises coarse-graining the at least one first mapping image and the at least one second mapping image.

8. 2. The image processing apparatus of claim 1, wherein filtering the at least one first mapping image and the at least one second mapping image comprises smoothing the at least one first mapping image and the at least one second mapping image.

9. 2. The image processing device of claim 1, wherein the at least one first mapping image and the at least one second mapping image are, respectively, a mapping image obtained by mapping measurement using a first instrument and a mapping image obtained by mapping measurement using a second instrument different from the first instrument.

10. 2. The image processing device according to claim 1, wherein the at least one first mapping image and the at least one second mapping image are, respectively, a mapping image obtained by a mapping measurement at a first timing and a mapping image obtained by a mapping measurement at a second timing different from the first timing.

11. The image processing device according to claim 1 , wherein the material includes a metallic material.

12. an image processing device filters at least one first mapping image obtained by a mapping measurement of a first region of a material and at least one second mapping image obtained by a mapping measurement of a second region of the material at least partially overlapping with the first region, and extracts features of the at least one first mapping image and the at least one second mapping image, thereby creating at least one first feature map and at least one second feature map, respectively; determining a transformation parameter based on a first dissimilarity between the at least one first feature map and at least one transformed feature map obtained by transforming the at least one second feature map using a transformation parameter; applying the determined deformation parameters to the at least one second mapping image by the image processing device, thereby aligning the at least one first mapping image with the at least one second mapping image; A mapping image registration method comprising:

13. creating at least one first feature map and at least one second feature map by filtering at least one first mapping image obtained by a mapping measurement of a first region of a material and at least one second mapping image obtained by a mapping measurement of a second region of the material that at least partially overlaps with the first region, and extracting features of the at least one first mapping image and the at least one second mapping image; determining a transformation parameter based on a first dissimilarity between the at least one first feature map and at least one transformed feature map obtained by transforming the at least one second feature map using the transformation parameter; applying the determined deformation parameters to the at least one second mapping image to align the at least one first mapping image with the at least one second mapping image; A program that causes a computer to perform operations including:

Images