Image combining method, image combining apparatus, and computer-readable program

Abstract

To combine a plurality of captured images at high speed with saved memory.SOLUTION: An image combining method includes steps of: preparing captured images 61 each having an overlapping region 611 partially overlapping with each other; determining at least one of X and Y directions as a weighting direction for the overlapping regions, on the basis of the sizes in the X and Y directions of the overlapping regions; setting a one-dimensional weighting factor array in which a weighting factor gradually decreases toward the outside of the captured images in the weighting direction in the overlapping regions of the captured images; obtaining a weighted pixel value of each overlapping pixel included in the overlapping region in each captured image by weighting a pixel value of the overlapping pixel by using a weighting factor specified from the weighting factor array; and acquiring a combined image by combining the captured images. A pixel value of each overlapping pixel in the combined image is obtained by using weighted pixel values in two or more captured images including the overlapping pixels.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a technique for combining multiple images. [Background technology]

[0002] When capturing an image of an object with high resolution or over a wide range, a common method is to capture the object range multiple times with overlapping imaging areas, rather than capturing the image in one go, and then combine (synthesize) the resulting images. For example, in Patent Documents 1 and 2, weighting is applied to the overlapping areas between two images according to the distance from each image, thereby preventing the boundaries in the combined image from becoming conspicuous. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Publication No. 59-133665 [Patent Document 2] Japanese Patent Application Publication No. 62-140172 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when a combined image is composed of many images and the combined image is large in size, high-speed processing close to real time may be required. In such a case, it is conceivable to determine the weighting coefficients of the overlapping areas for all images having overlapping areas in advance and combine the images collectively to speed up the processing, but a large-capacity memory is required to store all the weighting coefficients. If the data amount (number of bits) of the weighting coefficients is increased to improve the image quality of the combined image, the memory capacity required will further increase.

[0005] The present invention has been made in consideration of the above problems, and has an object to combine a plurality of captured images quickly and with less memory usage. [Means for solving the problem]

[0006] A first aspect of the present invention is an image combination method for combining a plurality of images, the method comprising the steps of: a) preparing a plurality of captured images, the plurality of captured images being images captured at a plurality of imaging positions on an object, the plurality of captured images being images in which a partial overlapping region is provided in each combination of two images corresponding to adjacent imaging positions; b) defining the arrangement direction of pixels in the plurality of captured images as a first direction and a second direction, and determining, for each of the overlapping regions, at least one of the first direction and the second direction as a weighting direction based on the size of each overlapping region in the first direction and the second direction; and c) determining, in the overlapping region of each of the captured images, a weighting direction for the weighting direction. the weighting coefficient array being determined so that the pixel values ​​of the overlapping pixels in each of the captured images are weighted by a weighting coefficient determined from the weighting coefficient array for each overlapping pixel in each of the captured images; and the weighting coefficient array being determined so that the pixel values ​​of the overlapping pixels in each of the captured images are weighted by a weighting coefficient determined from the weighting coefficient array for each overlapping pixel in each of the captured images.

[0007] A second aspect of the present invention is the image combination method of the first aspect, wherein in step d), when two or more weighting coefficients are identified for an overlapping pixel in each of the captured images, a weighted pixel value of the overlapping pixel is obtained by multiplying the pixel value of the overlapping pixel by the product of the two or more weighting coefficients.

[0008] A third aspect of the present invention is the image combination method of the first aspect (which may be either one of the first and second aspects), wherein in step e), the pixel value of each of the overlapping pixels in the combined image is determined by dividing the sum of the weighted pixel values ​​of each of the overlapping pixels in the two or more captured images by the sum of coefficients by which the pixel values ​​of each of the overlapping pixels are multiplied when determining the weighted pixel value in the two or more captured images.

[0009] A fourth aspect of the present invention is an image merging method according to any one of aspects 1 to 3, wherein in step b), if the entirety of one overlapping area overlaps with another overlapping area and the one overlapping area is equal to or smaller than a predetermined size, the one overlapping area is excluded.

[0010] A fifth aspect of the present invention is an image combining device for combining a plurality of images, the image combining device including a storage unit for storing a plurality of captured images, the captured images being images captured at a plurality of imaging positions on an object, the captured images being images in which a partial overlapping area is provided in each combination of two images for adjacent imaging positions; a weighting direction determination unit for determining, based on the size of each overlapping area in the first direction and the second direction, at least one of the first direction and the second direction as a weighting direction for each of the overlapping areas, with a first direction and a second direction being the array direction of pixels in the plurality of captured images; and a weighting direction determination unit for determining, based on the size of each overlapping area in the first direction and the second direction, at least one of the first direction and the second direction as a weighting direction for each of the overlapping areas, in the overlapping area of ​​each captured image, a weighted pixel value calculation unit that calculates a weighted pixel value of each overlapping pixel in each of the captured images by setting a one-dimensional weighting coefficient array that gradually decreases toward the outside of the captured image, treating pixels included in each overlapping region as overlapping pixels, and weighting a pixel value of each overlapping pixel in each of the captured images using a weighting coefficient specified for each overlapping pixel from the weighting coefficient array; and a combined image acquisition unit that acquires a combined image by combining the multiple captured images in accordance with the multiple imaging positions, wherein the combined image acquisition unit calculates the pixel value of each overlapping pixel in the combined image using the weighted pixel values ​​in two or more captured images including each of the overlapping pixels.

[0011] A sixth aspect of the present invention is a computer-readable program that causes a computer to execute a process of combining a plurality of images, and the execution of the program by the computer includes the steps of: a) preparing a plurality of captured images, each of which is an image captured at a plurality of imaging positions on an object, and in each combination of two images for adjacent imaging positions, a partially overlapping overlapping region is provided; b) defining the arrangement direction of pixels in the plurality of captured images as a first direction and a second direction, and determining, for each of the overlapping regions, at least one of the first direction and the second direction as a weighting direction based on the size of each overlapping region in the first direction and the second direction; and c) weighting the overlapping region of each of the captured images. The method executes the steps of: setting a one-dimensional weighting coefficient array in the overlapping region that gradually decreases toward the outside of each of the captured images in the weighting direction; d) determining a weighted pixel value of each overlapping pixel in each of the captured images by treating pixels included in each of the overlapping regions as overlapping pixels and weighting the pixel value of each overlapping pixel in each of the captured images using a weighting coefficient specified from the weighting coefficient array for each overlapping pixel; and e) obtaining a combined image by combining the multiple captured images in accordance with the multiple imaging positions, wherein in the e) step, the pixel value of each overlapping pixel in the combined image is determined using the weighted pixel values ​​in two or more captured images including each overlapping pixel. Effect of the Invention

[0012] According to the present invention, a plurality of captured images can be stitched together quickly and with less memory usage. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 is a block diagram showing a configuration of an image acquisition device. [Diagram 2] FIG. 1 illustrates a configuration of a computer. [Diagram 3] FIG. 13 is a diagram for explaining a comparative example of processing for combining a plurality of images. [Figure 4] FIG. 13 is a diagram showing a process flow for combining a plurality of images. [Diagram 5] FIG. 13 is a diagram showing a plurality of captured images and an overlapping region. [Figure 6] 11 is a diagram for explaining calculation of weighted pixel values ​​of overlapping pixels; FIG. [Figure 7] FIG. 13 is a diagram showing a combined image generation area. [Figure 8] FIG. 13 is a diagram showing a combined image generation region and a coefficient accumulation region. [Figure 9] FIG. 2 is a diagram showing a plurality of captured images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] FIG. 1 is a block diagram showing a configuration of an image acquisition device 1 according to an embodiment of the present invention. The image acquisition device 1 is a device that acquires an image of an object. The object in this embodiment is, for example, a substrate such as a printed circuit board on which a pattern is formed, or cells held in a container such as a petri dish. The image acquisition device 1 includes a captured image acquisition section 2 and an image combination device 4. As will be described later, the image combination device 4 in this embodiment is realized by a computer.

[0015] The captured image acquisition section 2 includes an imaging section 21 and a moving mechanism 22. The imaging section 21 has an imaging element and the like, and captures an image of an object. The moving mechanism 22 has a motor, a ball screw and the like, and moves the imaging section 21 relatively to the object. The operation of the captured image acquisition section 2 to capture an image of an object will be described later. The image captured by the captured image acquisition section 2 (hereinafter referred to as the "captured image") is output to the image combination device 4. The captured image is typically a rectangular image.

[0016] FIG. 2 is a diagram showing the configuration of the computer 3 that realizes the image combining device 4. The computer 3 has a configuration of a general computer system including a CPU 31, a ROM 32, a RAM 33, a storage device 34, a display 35, an input unit 36, a reading device 37, a communication unit 38, a GPU 39, and a bus 30. The CPU 31 performs various arithmetic processing. The GPU 39 performs various arithmetic processing related to image processing and the like. The ROM 32 stores a basic program. The RAM 33 and the storage device 34 store various information. The display 35 displays various information such as images. The input unit 36 ​​includes a keyboard 36a and a mouse 36b that accept input from an operator. The reading device 37 reads information from a computer-readable recording medium 81 such as an optical disk, a magnetic disk, a magneto-optical disk, or a memory card. The communication unit 38 transmits and receives signals between the captured image acquisition unit 2 and an external device. The bus 30 is a signal circuit that connects the CPU 31, the GPU 39, the ROM 32, the RAM 33, the storage device 34, the display 35, the input unit 36, the reading device 37, and the communication unit 38.

[0017] In the computer 3, the program 811 is read in advance from the recording medium 81 via the reading device 37 and stored in the storage device 34. The program 811 may be computer-readable and may be stored in the storage device 34 via a network. The CPU 31 and the GPU 39 execute arithmetic processing while using the RAM 33 and the storage device 34 in accordance with the program 811. The CPU 31 and the GPU 39 function as a calculation unit in the computer 3. Other configurations functioning as a calculation unit other than the CPU 31 and the GPU 39 may be employed.

[0018] In the image combining device 4, the computer 3 executes arithmetic processing and the like according to the program 811, thereby realizing the functional configuration shown in Fig. 1. That is, the CPU 31, GPU 39, ROM 32, RAM 33, storage device 34 of the computer 3 and their peripheral configuration realize a weighting direction determination unit 41, a weighted pixel value calculation unit 42, a combined image acquisition unit 43, and a memory unit 44. The functions of the weighting direction determination unit 41, the weighted pixel value calculation unit 42, the combined image acquisition unit 43, and the memory unit 44 will be described in detail later. All or part of these functions may be realized by dedicated electric circuits. Furthermore, these functions may be realized by multiple computers.

[0019] Here, a comparative example process for combining a plurality of images will be described with reference to FIG. 3. In the comparative example process, as shown in the left side of FIG. 3, a plurality of imaging positions P on an object are imaged by an imaging unit. On the left side of FIG. 3, an imaging range R of the imaging unit for each imaging position P is shown by a two-dot chain line rectangle. This results in a plurality of captured images 91 shown in the upper center of FIG. 3 being acquired. Although only two captured images 91 are shown in the upper center of FIG. 3, in reality, a plurality of captured images 91 are acquired by capturing images of a plurality of imaging positions P. At this time, in each combination of two captured images 91 for adjacent imaging positions P, a partially overlapping overlapping region 911 is provided.

[0020] In the process of the comparative example, the range of the overlapping region 911 in each captured image 91 is known due to the relative positional relationship of the multiple imaging positions P. A weighting coefficient (weighting coefficient) in the range of 0 to 1 is determined in advance for each pixel included in the overlapping region 911 of the captured image 91 and stored in a memory. As described above, in practice, a large number of captured images 91 are acquired, so a large-capacity memory is prepared to store a two-dimensional array of weighting coefficients for the overlapping regions 911 of all captured images 91. Typically, the weighting coefficient in each overlapping region 911 gradually decreases toward the outside of the captured image 91. In the lower center section of FIG. 3, the magnitude of the weighting coefficient set for each pixel in the overlapping region 911 is represented by grayscale shading, and the higher the density (black), the closer the weighting coefficient is to 0, and the lower the density (white), the closer the weighting coefficient is to 1.

[0021] The pixel values ​​of the pixels included in the overlapping region 911 of each captured image 91 are multiplied by the corresponding weighting coefficient. After that, the multiple captured images 91 are arranged according to the relative positional relationship of the multiple imaging positions P, and a combined image 92 shown on the right side of Fig. 3 is obtained. At this time, in the combined image 92, the sum of the pixel values ​​(pixel values ​​multiplied by the weighting coefficient) of the pixel in the two captured images 91 in which the overlapping region 911 is provided is assigned as the pixel value to each pixel in a region 921 corresponding to the overlapping region 911.

[0022] In the process of the comparative example, even when the combined image 92 is large in size because there are many captured images 91 that compose the combined image 92, it is possible to quickly acquire the combined image 92. On the other hand, as described above, the process of the comparative example requires a large-capacity memory for storing a two-dimensional array of weighting coefficients for the overlapping regions 911 of the many captured images 91. If the data amount (number of bits) of the weighting coefficients is increased to improve the image quality of the combined image 92, the required memory capacity will further increase.

[0023] Next, an image combining process by the image combining device 4 that can combine multiple captured images at high speed and with less memory will be described. Fig. 4 is a diagram showing a process flow for combining multiple images by the image combining device 4. In the image combining device 4, first, multiple captured images are prepared (step S11).

[0024] FIG. 5 is a diagram showing a plurality of captured images 61 and an overlapping region 611. In this embodiment, the moving mechanism 22 of the captured image acquisition unit 2 sequentially arranges the imaging unit 21 above a plurality of imaging positions P on the object, thereby acquiring a plurality of captured images (data) 61 obtained by capturing images of the plurality of imaging positions P respectively. The plurality of captured images 61 have the same size. The left side of FIG. 5 shows a plurality of captured images 61 (each captured image is denoted by a reference symbol 61a) arranged in accordance with the relative positional relationship of the plurality of imaging positions P. In each captured image 61, pixels are arranged in a first direction and a second direction that are orthogonal to each other. In FIG. 5 (and in FIG. 6, FIG. 7, and FIG. 9 described later), the first direction and the second direction that are pixel arrangement directions are shown as the X direction and the Y direction, and in the following description, the first direction and the second direction are referred to as the "X direction" and the "Y direction". Although only four captured images 61 are shown on the left side of FIG. 5, in reality, a large number (for example, several thousand) of captured images 61 are obtained by capturing images of a large number of imaging positions P on the object.

[0025] In the plurality of captured images 61, in each combination of two captured images 61 for two adjacent imaging positions P, a partially overlapping overlapping region 611 (in FIG. 5, some overlapping regions are denoted by symbols 611a and 611b) is provided. Since the relative positional relationship of the plurality of imaging positions P is known, the range of the overlapping region 611 in each captured image 61 is also known. On the left side of FIG. 5, one captured image 61a among the four captured images 61 is denoted by a dot pattern, and two overlapping regions 611a and 611b formed between the captured image 61a and the other two captured images 61 are denoted by parallel oblique lines having different orientations. The plurality of captured images 61 are inputted and stored in the storage unit 44 of the image combining device 4. In this processing example, the captured images are assumed to be grayscale images, but the captured images may be color images.

[0026] In the weighting direction determination unit 41, a weighting direction used for specifying a weighting coefficient is determined for each overlapping region 611 (step S12). In this processing example, the size of the overlapping region 611 in the X direction is compared with a first threshold value, and the size in the Y direction is compared with a second threshold value. Typically, the first threshold value and the second threshold value are the same value. When the size in the X direction is greater than the first threshold value and the size in the Y direction is less than or equal to the second threshold value, only the Y direction is set as the weighting direction. When the size of the overlapping region 611 in the X direction is less than or equal to the first threshold value and the size in the Y direction is greater than the second threshold value, only the X direction is set as the weighting direction.

[0027] In other cases, each of the X direction and the Y direction is set as the weighting direction. Other cases refer to the case where the size of the overlapping region 611 in the X direction is greater than the first threshold value and the size in the Y direction is greater than the second threshold value, and the case where the size of the overlapping region 611 in the X direction is less than or equal to the first threshold value and the size in the Y direction is less than or equal to the second threshold value. As described above, typically, the first threshold value and the second threshold value are the same value. However, for example, when the resolution in the X direction and the resolution in the Y direction are different in the captured image 61, the first threshold value and the second threshold value may be different (the same applies to the processing example described later). Also, the first threshold value may be determined based on the size of the captured image 61, such as M times (0 < M < 1) the size of the captured image 61 in the X direction (the same applies to the second threshold value).

[0028] In the example on the left side of FIG. 5, for the overlapping region 611a, since the size in the X direction is less than or equal to the first threshold value and the size in the Y direction is greater than the second threshold value, only the X direction is set as the weighting direction. For the overlapping region 611b, since the size in the X direction is greater than the first threshold value and the size in the Y direction is greater than the second threshold value, each of the X direction and the Y direction is set as the weighting direction. As described above, in step S12, based on the sizes of the overlapping region 611 in the X direction and the Y direction, at least one of the X direction and the Y direction is determined as the weighting direction for the overlapping region 611.

[0029] Next, the weighted pixel value calculation unit 42 sets a one-dimensional weighting coefficient array related to the weighting direction in the overlapping region 611 of each captured image 61 (step S13). For example, in the overlapping region 611 whose weighting direction is the X direction, the X coordinate of the overlapping region 611 on the (-X) side (left end in FIG. 5) is x, the width of the overlapping region 611 in the X direction is w, and the X coordinate of each pixel position included in the overlapping region 611 is x0, and an array of weighting coefficients C[x0] is obtained by Equation 1.

[0030]

number

[0031] In the weighting coefficient array obtained by Equation 1, the weighting coefficient gradually decreases from 1 to 0 in the X direction from the center of the captured image 61 including the overlapping region 611 toward the outside. In the overlapping region 611, the weighting coefficient for the pixels arranged in the Y direction at each pixel position in the X direction is the same, and the weighting coefficients of the pixels arranged two-dimensionally in the overlapping region 611 are expressed as a one-dimensional weighting coefficient array in the X direction. Therefore, as will be described later, when determining a weighting coefficient for each pixel included in the overlapping region 611, it is possible to easily and quickly determine the weighting coefficient of the pixel without performing complex distance calculations. On the right side of FIG. 5, the magnitude of the weighting coefficient of each pixel in the overlapping region 611a, whose weighting direction is the X direction, is represented by grayscale shading.

[0032] In the weighting coefficient array for the X direction obtained by Equation 1, the weighting coefficients change linearly with respect to the distance from the edge of the overlapping region 611 in the X direction, but the weighting coefficients may change in a curved manner (nonlinearly) as long as they gradually decrease toward the outside of the captured image 61. In one example, a Gaussian function or the like is used. Also, a function such as Equation 1 may be set as the weighting coefficient array. The same applies to the weighting coefficient array for the Y direction described below.

[0033] For the overlap region 611 in which the weighting direction is the Y direction, a weighting coefficient array is obtained in the same manner as in Equation 1. In the weighting coefficient array, the weighting coefficient gradually decreases from 1 to 0 in the Y direction from the center to the outside of the captured image 61 including the overlap region 611. In the overlap region 611, the weighting coefficient for pixels aligned in the X direction at each pixel position in the Y direction is the same, and the weighting coefficients of the pixels two-dimensionally arranged in the overlap region 611 are expressed as a one-dimensional weighting coefficient array in the Y direction.

[0034] In the overlap region 611 in which the weighting directions are the X direction and the Y direction, a one-dimensional weighting coefficient array for the X direction and a one-dimensional weighting coefficient array for the Y direction are set separately. In the overlap region 611b shown on the right side of Fig. 5, the product of a weighting coefficient specified from the weighting coefficient array for the X direction and a weighting coefficient specified from the weighting coefficient array for the Y direction is represented by grayscale shading for each pixel.

[0035] Next, the weighted pixel value calculation unit 42 calculates a weighted pixel value of each pixel included in an overlapping region 611 (hereinafter referred to as "overlapping pixel") in each captured image 61 by weighting the pixel value of the overlapping pixel using a weighting factor specified from the weighting factor array (step S14). Specifically, when an overlapping pixel is included in only one overlapping region 611 and only one weighting factor array in the X direction or Y direction is set for the overlapping region 611 that includes the overlapping pixel, only one weighting factor is specified for the overlapping pixel from the weighting factor array. Then, the pixel value of the overlapping pixel is multiplied by the weighting factor to obtain a weighted pixel value.

[0036] On the other hand, when an overlapping pixel is included in two or more overlapping regions 611, and / or when the X direction and the Y direction are each determined as the weighting direction for the overlapping region 611 including the overlapping pixel, two or more weighting coefficient arrays are set for the overlapping pixel. In this case, a weighting coefficient is specified for the overlapping pixel from each of the two or more weighting coefficient arrays. In the following description, the N weighting coefficients (N is an integer equal to or greater than 1) specified for each overlapping pixel are denoted as weighting coefficient C1, weighting coefficient C2, ..., weighting coefficient CN.

[0037] Fig. 6 is a diagram for explaining calculation of weighted pixel values ​​of overlapping pixels for which two or more weighting factors are specified, and shows a captured image 61a on the left side of Fig. 5. On the upper side of the captured image 61a in Fig. 6, the weighting coefficient array in the X direction in the overlapping region 611a and the weighting coefficient array in the X direction in the overlapping region 611b are represented by grayscale shading, and on the right side of the captured image 61a, the weighting coefficient array in the Y direction in the overlapping region 611b is represented by grayscale shading.

[0038] For example, an overlap pixel 612 indicated by a black dot in Fig. 6 is included in both overlap region 611a and overlap region 611b. Therefore, for the overlap pixel 612, a weighting factor C1 is determined from the weighting factor array related to the X direction in overlap region 611a, a weighting factor C2 is determined from the weighting factor array related to the X direction in overlap region 611b, and a weighting factor C3 is determined from the weighting factor array related to the Y direction in overlap region 611b. Specifically, the weighting factor C1 is 0.5, the weighting factor C2 is 0.3, and the weighting factor C3 is 0.6. The weighted pixel value of the overlap pixel 612 is calculated by multiplying the pixel value V of the overlap pixel 612 by the product of the weighting factors C1 to C3, i.e., (0.5 x 0.3 x 0.6).

[0039] As described above, when two or more weighting coefficients are specified for one overlapping pixel 612 in each captured image 61, a weighted pixel value of the overlapping pixel 612 is obtained by multiplying the pixel value of the overlapping pixel 612 by the product of the two or more weighting coefficients. The weighted pixel value is treated as a new pixel value of the overlapping pixel 612. In the following description, the value by which the pixel value of the overlapping pixel 612 is multiplied in calculating the weighted pixel value, i.e., the product of the weighting coefficients specified for the overlapping pixel 612 (when only one weighting coefficient is specified, that weighting coefficient) is referred to as a "composite weighting coefficient."

[0040] FIG. 7 is a diagram showing a combined image generation area 491. The combined image generation area 491 is a storage area for generating a combined image by combining a plurality of captured images 61, and is set in the internal memory of the CPU 31, the internal memory of the GPU 39, the RAM 33, or the like. When the weighted pixel value of the overlapping pixel 612 in each captured image 61 is calculated, the combined image acquisition unit 43 adds the pixel value of each pixel of the captured image 61 to the combined image generation area 491 (step S15). In detail, in the combined image generation area 491, the arrangement of the plurality of captured images 61 is determined in advance in accordance with the relative positional relationship of the plurality of imaging positions P, and the pixel value of each pixel of the captured image 61 arranged in the combined image generation area 491 is added to the position where the pixel of the captured image 61 overlaps. The initial value of each position in the combined image generation area 491 is 0. 7, the captured image 61a arranged in the combined image generation area 491 is indicated by a solid line, and two captured images 61 forming overlapping areas 611a, 611b with the captured image 61a are indicated by two-dot chain lines. In the overlapping areas 611a, 611b, the dot pattern of the captured image 61a is superimposed with the grayscale shading pattern shown on the right side of Fig. 5. In reality, other overlapping areas 611 also exist in the captured image 61a.

[0041] In adding pixel values ​​at each position in the combined image generation area 491, if a pixel of the captured image 61 overlapping at that position is not an overlapping pixel 612, the pixel value of that pixel is added to that position as is. On the other hand, if the pixel is an overlapping pixel 612, a new pixel value of that pixel, i.e., a weighted pixel value, is added to that position. As described above, the overlapping pixels 612 in each captured image 61 are included in one or more overlapping areas 611, and in the combined image generation area 491, a position where the overlapping pixel 612 of one captured image 61 overlaps also overlaps with one or more other captured images 61. Therefore, the value of that position is the sum of the weighted pixel values ​​of the overlapping pixels 612 in two or more captured images 61 overlapping at that position.

[0042] [Table 1]

[0043] Table 1 focuses on positions where overlapping pixels 612 of three captured images 61 overlap in the combined image generation area 491, and shows, for each captured image 61, the pixel value V, weighting coefficients C1 to C3, synthesis weighting coefficient, and weighted pixel value of the overlapping pixel 612 corresponding to that position. Furthermore, in Table 1, the three captured images 61 are assigned image numbers 1 to 3. The value of that position in the combined image generation area 491 is the sum of the weighted pixel values ​​of that overlapping pixel 612 in the three captured images 61 with image numbers 1 to 3, i.e., 21.

[0044] 8 is a diagram showing a combined image generation area 491 and a coefficient accumulation area 492. The coefficient accumulation area 492 is a storage area for accumulating synthesis weighting coefficients corresponding to each position of the combined image generation area 491, and includes an area of ​​the same size as the area in which the multiple captured images 61 are arranged in the combined image generation area 491. Like the combined image generation area 491, the coefficient accumulation area 492 is set in the internal memory of the CPU 31, the internal memory of the GPU 39, the RAM 33, or the like.

[0045] In the combined image acquisition section 43, when the weighted pixel value of each overlapping pixel 612 of the captured image 61 is added to the combined image generation area 491, the composite weight coefficient used for the overlapping pixel 612 is added to the coefficient accumulation area 492 (step S16). Specifically, in the coefficient accumulation area 492, similar to the combined image generation area 491, the arrangement of the multiple captured images 61 is determined in advance in accordance with the relative positional relationship of the multiple imaging positions P, and the composite weight coefficient used for calculating the weighted pixel value of the overlapping pixel 612 is added to the position where each overlapping pixel 612 of the arranged captured images 61 overlaps in the coefficient accumulation area 492. In the coefficient accumulation area 492, since the overlapping pixel 612 of one captured image 61 overlaps with one or more other captured images 61, the value of the position is the sum of the composite weight coefficients of the overlapping pixels 612 in two or more captured images 61 that overlap with the position. The value of the position in the coefficient accumulation area 492 corresponding to the position of the combined image generation area 491 of interest in Table 1 is the sum of the synthesis weighting coefficients of the overlapping pixels 612 in the three captured images 61 of image numbers 1 to 3, that is, 0.15. Note that in the coefficient accumulation area 492, a value of 1, for example, is assigned to positions corresponding to pixels other than the overlapping pixels 612.

[0046] When the pixel values ​​of all the pixels of the captured images 61 are added to the combined image generation area 491 and the synthesis weighting coefficients for all the overlapping pixels 612 are added to the coefficient accumulation area 492, the combined image acquisition unit 43 changes the value of each position of the combined image generation area 491 to a value obtained by dividing the value by the value of the same position in the coefficient accumulation area 492. For example, at a position of interest in the combined image generation area 491 in Table 1, the sum of the weighted pixel values, 21, is divided by the sum of the synthesis weighting coefficients, 0.15, to change the value of that position to 140. Note that, for positions in the combined image generation area 491 corresponding to pixels other than the overlapping pixels 612, the value in the coefficient accumulation area 492 is 1, so the value of that position remains the same.

[0047] Each position in the combined image generation area 491 corresponds to a pixel of the combined image, and by the above process, a combined image is obtained by combining a plurality of captured images 61 in the combined image generation area 491 (step S17). In the combined image, the pixel value of a pixel corresponding to each overlapping pixel 612 (hereinafter also referred to as an "overlapping pixel") is calculated using the weighted pixel value of the overlapping pixel 612 in two or more captured images 61 that include the overlapping pixel 612.

[0048] Specifically, the pixel value of the overlapping pixel in the combined image is obtained by dividing the sum of the weighted pixel values ​​of the overlapping pixel 612 in the two or more captured images 61 by the sum of the coefficients (i.e., composite weight coefficients) by which the pixel values ​​of the overlapping pixel 612 are multiplied when the weighted pixel values ​​are obtained in the two or more captured images 61. This allows the density in the area of ​​the combined image corresponding to the overlapping area 611 (hereinafter, also referred to as the "overlapping area") to be approximately matched to the density in other areas, making it possible to suppress the occurrence of unintended density changes in the overlapping area. In particular, even if there are overlapping pixels in which two or more overlapping areas 611 overlap in the combined image, that is, there are overlapping pixels in which three or more captured images 61 overlap, the pixel value of the overlapping pixel can be obtained appropriately and easily by the above method. The acquired combined image is used, for example, for inspecting patterns on a substrate or inspecting cells.

[0049] In the above image combination process, a storage area including the entire combined image is secured in the combined image generation area 491 and the coefficient accumulation area 492, but each of the combined image generation area 491 and the coefficient accumulation area 492 may store only values ​​of positions overlapping with the overlapping pixels 612. In this case, for positions in the combined image that do not overlap with the overlapping pixels 612, the pixel values ​​of the pixels in the captured image 61 that correspond to those positions are used as they are.

[0050] As described above, in the image combining method shown in FIG. 4, a plurality of captured images 61 are prepared, which are images captured at a plurality of imaging positions P on an object, and each combination of two images for adjacent imaging positions P has an overlapping region 611 that partially overlaps with each other (step S11). Next, based on the size of each overlapping region 611 in the X direction and the Y direction, at least one of the X direction and the Y direction is determined as a weighting direction for the overlapping region 611 (step S12). In the overlapping region 611 of each captured image 61, a one-dimensional weighting coefficient array that gradually decreases toward the outside of the captured image 61 in the weighting direction is set (step S13). The pixels included in each overlapping region 611 are set as overlapping pixels 612, and the pixel values ​​of the overlapping pixels 612 in each captured image 61 are weighted using weighting coefficients specified for the overlapping pixels 612 from the weighting coefficient array, thereby obtaining weighted pixel values ​​of the overlapping pixels 612 in the captured image 61 (step S14). Thereafter, a combined image is acquired by combining the multiple captured images 61 according to the multiple imaging positions P (steps S15 to S17). At this time, the pixel value of each overlapping pixel in the combined image is calculated using weighted pixel values ​​in two or more captured images 61 including the overlapping pixel.

[0051] In the process of the comparative example described with reference to FIG. 3, a two-dimensional array of weighting coefficients is prepared for overlapping regions of many captured images, and therefore a large-capacity memory is required to store the two-dimensional array of weighting coefficients. If the data amount (number of bits) of the weighting coefficients is increased to improve the image quality of the combined image, the capacity required for the memory will further increase. In contrast, in the image combining method shown in FIG. 4, the weighting direction is determined based on the size of each overlapping region 611 in the X direction and the Y direction, and a one-dimensional weighting coefficient array related to the weighting direction is set for the overlapping region 611. In this image combining method, since it is not necessary to store a two-dimensional array of weighting coefficients for the overlapping regions 611 of many captured images 61, it is possible to combine multiple captured images 61 with less memory and generate a combined image, compared to the process of the comparative example.

[0052] As another comparative example of processing for combining multiple images, it is possible to combine the captured images one by one in order, but in this case, it takes a long time to combine the multiple captured images. Also, due to the influence of rounding errors, pixel values ​​in overlapping areas of the combined images may differ depending on the combining order, that is, calculation errors may occur. In contrast, the image combining method shown in FIG. 4 is capable of processing multiple captured images 61 at once, so that the multiple captured images 61 can be combined at high speed. Also, it is possible to avoid the occurrence of calculation errors due to the combining order.

[0053] In a preferred step S12, in each overlapping region 611, if the size in one of the X and Y directions is greater than a predetermined threshold and the size in the other direction is equal to or smaller than the threshold, only the other direction is set as the weighting direction. This makes it possible to determine an appropriate weighting direction that is approximately in line with the relative direction of the two captured images 61 that form the overlapping region 611, and to generate a combined image with beautiful seams. In addition, by setting only one direction as the weighting direction for some of the overlapping regions 611, it becomes possible to generate a combined image with less memory and at higher speed.

[0054] When one overlap pixel 612 in each captured image 61 is included in two or more overlap regions 611 and / or when the X direction and the Y direction are each determined as the weighting direction for the overlap pixel 612, two or more weighting coefficients are specified for the overlap pixel 612 in step S14. In such a case, it is preferable to obtain a weighted pixel value of the overlap pixel 612 by multiplying the pixel value of the overlap pixel 612 by the product of the two or more weighting coefficients. This makes it possible to easily obtain the weighted pixel value of the overlap pixel 612.

[0055] Next, another example of the image combining process will be described. FIG. 9 is a diagram showing a plurality of captured images 61, and shows a plurality of captured images 61 arranged in accordance with the relative positional relationship of a plurality of imaging positions P. In FIG. 9, one captured image 61b is shown by a thick line, and the other captured images 61 are shown by a thin line. In addition, two overlapping regions 611 formed between the captured image 61b and the captured image 61 arranged on the (+X) side of the captured image 61b, and between the captured image 61 arranged on the (-Y) side of the captured image 61b, respectively, are marked with parallel oblique lines of different directions. Furthermore, an overlapping region 611c formed between the captured image 61b and the captured image 61 arranged on both the (+X) side and the (-Y) side of the captured image 61b (i.e., the captured image 61 at the lower right of the captured image 61b in FIG. 9) is surrounded by a thick dashed line.

[0056] When the multiple captured images 61 and 61b shown in FIG. 9 are prepared (FIG. 4: step S11), the weighting direction determination unit 41 compares the size of each overlapping region 611 in the X direction with a first threshold value and compares the size of each overlapping region 611 in the Y direction with a second threshold value, as in the above processing example, to determine a weighting direction for each overlapping region 611 (step S12). At this time, if the size of the overlapping region 611 in the X direction is greater than the first threshold value and the size of the overlapping region 611 in the Y direction is equal to or less than the second threshold value, only the Y direction is set as the weighting direction (see the overlapping region 611 between the captured image 61b and the captured image 61 on the (-Y) side). If the size of the overlapping region 611 in the X direction is equal to or less than the first threshold value and the size of the Y direction is greater than the second threshold value, only the X direction is set as the weighting direction (see the overlapping region 611 between the captured image 61b and the captured image 61 on the (+X) side).

[0057] Moreover, when the size of the overlapping region 611 in the X direction is equal to or less than the first threshold value and the size in the Y direction is equal to or less than the second threshold value, if the overlapping region 611 entirely overlaps with another overlapping region 611, the weighting direction is not determined and the overlapping region 611 is invalidated. In the example of FIG. 9, the size of the overlapping region 611c in the X direction is equal to or less than the first threshold value, the size in the Y direction is equal to or less than the second threshold value, and further, the overlapping region 611c entirely overlaps with the other two overlapping regions 611 marked with parallel diagonal lines. Therefore, the overlapping region 611c is invalidated and is not treated as an overlapping region in the following processing. In other cases, the X direction and the Y direction are each set as the weighting direction. The processing of steps S13 to S17 is the same as the above processing example, except that only valid overlapping regions 611 are used.

[0058] As described above, in another example of the image combining method, in step S12, if one overlapping region 611 overlaps with another overlapping region 611 in its entirety and the one overlapping region 611 is equal to or smaller than a predetermined size, the one overlapping region 611 is excluded. The one overlapping region 611 has a smaller effect on the combined image than the other overlapping regions 611, but if the combined image is generated while leaving the one overlapping region 611, there is a possibility that an unnatural appearance will occur in the combined image due to a local change in density in the region corresponding to the one overlapping region 611. On the other hand, in the above processing example, by excluding the one overlapping region 611, it is possible to suppress the occurrence of an unnatural appearance in the combined image.

[0059] The image combining method and the image combining device 4 described above can be modified in various ways.

[0060] 4, the weighting direction may be determined based on, for example, the ratio (aspect / width ratio) of the size in the Y direction to the size in the X direction of the overlapping region 611. In this case, for example, when the ratio exceeds a predetermined range, only the X direction is determined as the weighting direction, when the ratio is within a predetermined range, both the X direction and the Y direction are determined as the weighting direction, and when the ratio is below the predetermined range, only the Y direction is determined as the weighting direction. In this way, the weighting direction for each overlapping region 611 may be determined by various methods based on the sizes of the overlapping region 611 in the X direction and the Y direction.

[0061] The image combining method and the combined image obtained by the image combining device 4 may be used for purposes other than inspection of patterns or cells on a substrate.

[0062] The configurations in the above-described embodiment and each of the modified examples may be combined as appropriate as long as they are not mutually inconsistent. [Explanation of symbols]

[0063] 3. Computer 4 Image Combining Device 41 Weighting direction determination unit 42 Weighted pixel value calculation unit 43 Combined image acquisition unit 44 Memory section 61, 61a, 61b Captured images 611,611a~611c Overlapping area 612 overlapping pixels 811 Program P Imaging position S11~S17 steps

Claims

1. 1. A method for combining a plurality of images, comprising the steps of: a) preparing a plurality of captured images, each captured at a plurality of imaging positions on an object, in which an overlapping region is provided in each combination of two images corresponding to adjacent imaging positions; b) defining an arrangement direction of pixels in the plurality of captured images as a first direction and a second direction, and determining, for each of the overlapping regions, at least one of the first direction and the second direction as a weighting direction based on the size of each of the overlapping regions in the first direction and the second direction; c) setting a one-dimensional weighting coefficient array in an overlapping region of each captured image, the weighting coefficient array gradually decreasing toward the outside of each captured image in the weighting direction; d) determining a weighted pixel value of each of the overlapping pixels in each of the captured images by weighting a pixel value of each of the overlapping pixels in each of the captured images using a weighting coefficient specified for each of the overlapping pixels in each of the captured images, the weighting coefficient being determined from the weighting coefficient array; e) acquiring a combined image by combining the plurality of captured images according to the plurality of imaging positions; Equipped with In the step e), a pixel value of each of the overlapping pixels in the combined image is calculated using weighted pixel values ​​in two or more captured images including each of the overlapping pixels.

2. 2. The image combining method according to claim 1, In the image combining method, in step d), when two or more weighting coefficients are specified for one overlapping pixel in each of the captured images, a weighted pixel value of the one overlapping pixel is determined by multiplying the product of the two or more weighting coefficients by the pixel value of the one overlapping pixel.

3. 2. The image combining method according to claim 1, In the step e), the pixel value of each of the overlapping pixels in the combined image is calculated by dividing the sum of the weighted pixel values ​​of each of the overlapping pixels in the two or more captured images by the sum of coefficients by which the pixel values ​​of each of the overlapping pixels are multiplied when calculating the weighted pixel value in the two or more captured images.

4. 4. The image combining method according to claim 1, further comprising the steps of: In the step b), if one overlapping region entirely overlaps with another overlapping region and the size of the one overlapping region is equal to or smaller than a predetermined size, the one overlapping region is excluded.

5. An image combining device for combining a plurality of images, comprising: a storage unit that stores a plurality of captured images, each of which is an image captured at a plurality of imaging positions on an object, and each combination of two images corresponding to adjacent imaging positions has an overlapping region where the images partially overlap; a weighting direction determination unit that defines an arrangement direction of pixels in the plurality of captured images as a first direction and a second direction, and determines at least one of the first direction and the second direction as a weighting direction for each of the overlapping regions based on a size of each of the overlapping regions in the first direction and the second direction; a weighted pixel value calculation unit that sets, in an overlapping region of each captured image, a one-dimensional weighting coefficient array that gradually decreases toward an outside of each captured image in the weighting direction, treats pixels included in each overlapping region as overlapping pixels, and weights pixel values ​​of each overlapping pixel in each captured image using a weighting coefficient specified for each overlapping pixel from the weighting coefficient array, thereby determining a weighted pixel value of each overlapping pixel in each captured image; a combined image acquisition unit that acquires a combined image by combining the plurality of captured images according to the plurality of imaging positions; Equipped with An image combining device in which the combined image acquisition unit determines a pixel value of each of the overlapping pixels in the combined image by using weighted pixel values ​​in two or more captured images including each of the overlapping pixels.

6. A computer-readable program for causing a computer to execute a process for combining a plurality of images, the process comprising the steps of: a) preparing a plurality of captured images, each captured at a plurality of imaging positions on an object, in which an overlapping region is provided in each combination of two images corresponding to adjacent imaging positions; b) defining an arrangement direction of pixels in the plurality of captured images as a first direction and a second direction, and determining, for each of the overlapping regions, at least one of the first direction and the second direction as a weighting direction based on the size of each of the overlapping regions in the first direction and the second direction; c) setting a one-dimensional weighting coefficient array in an overlapping region of each captured image, the weighting coefficient array gradually decreasing toward the outside of each captured image in the weighting direction; d) determining a weighted pixel value of each of the overlapping pixels in each of the captured images by weighting a pixel value of each of the overlapping pixels in each of the captured images using a weighting coefficient specified for each of the overlapping pixels in each of the captured images, the weighting coefficient being determined from the weighting coefficient array; e) acquiring a combined image by combining the plurality of captured images according to the plurality of imaging positions; Run the command, A computer-readable program in which, in step e), a pixel value of each of the overlapping pixels in the combined image is calculated using weighted pixel values ​​in two or more captured images including each of the overlapping pixels.

Images