Image processing device, image processing method, and program

Abstract

To prevent a position associated with an image used in connection processing from being corrected by mistake.SOLUTION: An image processing device corrects a position within a target area associated with a portion image obtained by imaging a part within the target area in an object to be imaged, and generates a connected image corresponding to the target area. The image processing device comprises: determination means which determines an imaging condition so that predetermined information about the object to be imaged is included, in an overlapping area where a portion image of interest among the plurality of portion images obtained from within the target area and other portion images adjacent to the portion image of interest overlap; acquisition means which acquires the plurality of portion images obtained by imaging the portion images based on the determined imaging condition; correction means which corrects the position associated with at least one portion image among the plurality of portion images, based on the overlapping area; and generation means which connects the plurality of portion images in which the positions have been corrected to generate the connected image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to techniques for stitching together multiple images. [Background technology]

[0002] There is a method for generating an image representing a target area with high resolution by linking a plurality of captured images. In order to link the captured images appropriately, position information in the target area associated with the captured images is corrected before linking the captured images. In order to perform position correction appropriately, it is preferable that information suitable for position correction, such as feature points, is included in the area where the two images to be linked overlap (called an overlap area).

[0003] Patent Document 1 describes a method of determining whether there is a correlated image area between partial images to be linked, and linking is performed starting from partial images in which a correlated image exists. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-254236 A Summary of the Invention [Problem to be solved by the invention]

[0005] Position correction associated with the images to be linked may be performed by moving the images to be linked and searching for a position where the overlapping areas of the images to be linked and the images to be linked have a high degree of coincidence. In this case, if the overlapping areas do not contain information suitable for position correction, the position information associated with the images may be erroneously corrected. The method of Patent Document 1 postpones position correction of images that are not suitable for position correction, so if there are multiple images with overlapping areas that do not contain information suitable for position correction, there is a risk that the positions associated with the images may be erroneously corrected.

[0006] The technique disclosed herein aims to prevent erroneous correction of positions associated with images used in a linking process. [Means for solving the problem]

[0007] The image processing device of the present disclosure is an image processing device that corrects a position within a target area associated with a partial image obtained by capturing an image of a part of the target area of ​​an imaging object, and generates a connected image corresponding to the target area, an imaging condition acquisition means for acquiring a maximum magnification and a minimum magnification designated by a user; and an overlap region determination means for determining an overlap region where a partial image of interest overlaps with another partial image adjacent to the partial image of interest among a plurality of partial images obtained from within the target region when imaging is performed at each of a plurality of magnifications included in the range from the minimum magnification to the maximum magnification; a derivation means for deriving an evaluation value of the overlapping area determined by the overlapping area determination means; a determining means for determining a magnification ratio at which the evaluation value derived by the derivation means is the highest as a magnification ratio for obtaining the partial image; , said determined magnification the plurality of partial images obtained by imaging based on the above-mentioned, a correction means for correcting the position associated with at least one of the plurality of partial images based on the overlapping area, and a generation means for generating the concatenated image by concatenating the plurality of partial images whose positions have been corrected. Effect of the Invention

[0008] According to the technology of the present disclosure, it is possible to prevent erroneous correction of positions associated with images used in a linking process. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus. [Diagram 2] FIG. 2 is a block diagram showing the functional configuration of the image processing apparatus. [Diagram 3] FIG. 2 is a diagram showing an example of a target region and an entire image of an object. [Figure 4] FIG. 4 is a diagram for explaining position correction. [Diagram 5] FIG. 4 is a diagram for explaining position correction. [Figure 6] FIG. 13 is a diagram showing an example of a UI screen. [Figure 7] 4 is a flowchart showing a process executed by the image processing apparatus. [Figure 8]4 is a flowchart showing a process executed by the image processing apparatus. [Figure 9] 4 is a flowchart showing a process executed by the image processing apparatus. [Figure 10] 4 is a flowchart showing a process executed by the image processing apparatus. [Figure 11] FIG. 2 is a block diagram showing the functional configuration of the image processing apparatus. [Figure 12] 4 is a flowchart showing a process executed by the image processing apparatus. [Figure 13] FIG. 4 is a diagram showing the arrangement of partial image groups. [Figure 14] FIG. 13 is a diagram for explaining the relationship between an evaluation value and a search range. [Figure 15] 4 is a flowchart showing a process executed by the image processing apparatus. [Figure 16] FIG. 13 is a diagram showing an example of a UI screen. [Figure 17] 10A to 10C are diagrams for explaining examples of overlapping regions determined under different imaging conditions. [Figure 18] 10A to 10C are diagrams for explaining examples of overlapping regions determined under different imaging conditions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments for implementing the technology of the present disclosure will be described with reference to the drawings. The components described in the following embodiments are merely examples and do not limit the scope of the technology of the present disclosure. In addition, all combinations of components described in the following embodiments are not necessarily essential for solving the problems, and various modifications and changes are possible. In each figure, the same parts or elements are given the same numbers, and duplicated descriptions are omitted.

[0011] <Embodiment 1> In this embodiment, an image processing device that connects a plurality of images (partial images) captured by an optical microscope to generate a high-resolution image with a wide field of view will be described. The connected image obtained by connecting the partial images by the image processing device is used, for example, for the purpose of detecting particles with high accuracy and analyzing the uneven distribution state of the particles when analyzing particles.

[0012] The partial images are captured so that there is an overlapping area between adjacent partial images (overlapping area). The relative coordinates of each partial image used when linking the partial images are determined by correcting the initial values ​​of the relative coordinates determined at the time of capturing the partial images. The relative coordinates are corrected by comparing an image of the overlapping area of ​​the partial images when the partial images are arranged based on their relative positions with an image of the overlapping area of ​​the adjacent images, and searching for a position where the images of the overlapping area match highly. In this embodiment, a method of changing the capturing conditions when capturing the partial images is described so that information useful for performing position correction is included in the overlapping area even when the partial images are arranged with the initial values ​​of the relative positions.

[0013] [Hardware configuration] 1 shows an example of the hardware configuration of an image processing device 100 in this embodiment. The image processing device 100 is realized by, for example, a PC, a computer, etc. The image processing device 100 has a CPU 101, a ROM 102, a RAM 103, a general-purpose I / F (interface) 104, a SATA (Serial ATA) I / F 105, and a VC (video card) 106. These are connected to a system bus 107 that enables data to be transmitted and received between them.

[0014] The CPU 101 uses a RAM 103 as a work memory and executes an OS (operating system) and various programs stored in a ROM 102, a HDD (hard disk drive) 170, etc. The CPU 101 also controls each component via a system bus 107.

[0015] The general-purpose I / F 104 is, for example, a serial bus interface such as USB, and connects an input device 130 such as a mouse or a keyboard, and an imaging device 190 via a serial bus 120. The imaging device 190 is, for example, an optical microscope having at least an area sensor, a lens, and a stage on which an object to be imaged is placed.

[0016] The SATA I / F 105 is a serial bus interface, and connects to a general-purpose drive 180 that reads and writes data from and to the HDD 170 and various recording media via the serial bus 160. The CPU 101 uses the HDD 170 or various recording media mounted on the general-purpose drive 180 as a storage location for various data.

[0017] The VC 106 is a video interface, and connects to a display 150. The CPU 101 controls displaying a UI (user interface) screen provided by a program on the display 150, and receives inputs such as user instructions received via the input device 130.

[0018] [Functional configuration of image processing device] 2 is a block diagram showing the functional configuration of the image processing device 100. The image processing device 100 includes an imaging control unit 210, an information processing unit 220, a display control unit 230, an input receiving unit 240, and a data management unit 250.

[0019] The imaging control unit 210 acquires imaging conditions, controls the imaging device 190, and causes the imaging device 190 to image the object under the imaging conditions. Then, an image obtained by the imaging is acquired. The imaging control unit 210 controls the imaging device 190 to change the lens of the imaging device 190, thereby changing the imaging magnification. The imaging control unit 210 also controls the imaging device 190 to move the stage position of the imaging device 190 on which the object is placed, and can acquire an image of a desired area of ​​the object.

[0020] The imaging control unit 210 includes an entire image acquisition unit 211 , a partial image group acquisition unit 212 , and an imaging condition acquisition unit 213 .

[0021] The whole image acquisition unit 211 instructs imaging under imaging conditions such that a target area in the object designated by the user falls within an imaging field of view (one field of view), and acquires the whole image (also referred to as a first image) obtained as a result.

[0022] The partial image group acquisition unit 212 acquires a group of partial images (also called second images) of the entire image. Each partial image is an image obtained by performing divided imaging in which the imaging device 190 sequentially moves the stage and captures images under imaging conditions with a higher magnification than that of the entire image.

[0023] The imaging condition acquisition unit 213 acquires imaging conditions when the imaging device 190 captures an image. The imaging conditions for capturing an entire image are set by the user via a UI screen described later. The imaging conditions for a partial image group are determined by conditions based on the user's settings via the UI screen and conditions determined by an imaging condition determination unit 223 described later.

[0024] Fig. 3 is a diagram for explaining an entire image and a group of partial images. Fig. 3(a) is a diagram showing an example of an object imaged by the imaging device 190, and the object is a sheet of paper on which dots 302 are sparsely formed with toner. The area of ​​the paper on which no dot-like images are formed is called the background. The rectangular area surrounded by a dashed line in Fig. 3(a) represents a target area 301 specified by the user, and the target area 301 is an imaging range for acquiring the entire image.

[0025] Fig. 3(b) is a diagram showing an example of a group of partial images. Each rectangle surrounded by a solid line in a target area 301 in Fig. 3(b) represents a partial image, and shows an example in which each partial image is arranged based on the relative position associated with that partial image.

[0026] In this embodiment, the partial image group will be described as a group of images that, when linked, will become an image that includes the target area. The partial image group does not have to be a group of images that, when linked, will become an image that includes the target area, as shown in FIG. 3(b). It is sufficient that an image that covers at least a predetermined percentage of the target area can be generated by linking the partial image group. In the example of FIG. 3(b), partial images are generated so as to constitute the target area of ​​the entire image using 4×3 partial images, as indicated by the number of rectangles surrounded by solid lines.

[0027] Moreover, the size of one partial image corresponds to the resolution desired by the user. By linking the partial images, the image processing device 100 can generate an image that represents the target area at a higher resolution than the entire image.

[0028] On the other hand, the information processing unit 220 has an overlapping region determining unit 221 , an evaluation value derivation unit 222 , an imaging condition determining unit 223 , a position correcting unit 224 , and an image connecting unit 225 .

[0029] The overlapping area determination unit 221 determines an overlapping area where a partial image of interest in the partial image group overlaps with another partial image adjacent to the partial image of interest based on the imaging conditions of the whole image and the imaging conditions of the partial images. In this embodiment, when imaging is performed to obtain a partial image, imaging is performed so that adjacent partial images have mutually overlapping areas. The area in the partial image represented by diagonal lines in FIG. 3(b) is the overlapping area.

[0030] The evaluation value derivation unit 222 derives an evaluation value of the overlapping region. The evaluation value will be described later.

[0031] The imaging condition determination unit 223 has a function of performing processing for suppressing the influence when characteristic information useful for appropriately performing position correction is not included in the overlapping region. The imaging condition determination unit 223 determines imaging conditions such as the stage movement amount and the number of images captured during imaging for acquiring a partial image group so that the evaluation value derived by the evaluation value derivation unit 222 is high. By determining imaging conditions that result in a high evaluation value, the imaging condition determination unit 223 can acquire a partial image group so that information useful for performing position correction is included in the overlapping region. Details will be described later.

[0032] The position correction unit 224 corrects position information (relative coordinates) that represents a position within the target area associated with each of the partial images. As described above, in this embodiment, the partial images are linked to generate an image that represents the entire target area of ​​the entire image. The corrected relative coordinates are used in the process of linking the partial images.

[0033] The initial values ​​of the relative coordinates associated with the partial images are determined by the stage movement amount specified for capturing the partial images. However, there may be a difference between the specified stage movement amount and the actual stage movement amount due to aging deterioration of the imaging device 190, insufficient adjustment, etc. For this reason, the partial images may not be properly linked with the initial values ​​of the relative coordinates. Therefore, the position correction unit 224 corrects the relative positions associated with the partial images so that the partial images are properly linked.

[0034] The image linking unit 225 links the partial images based on the relative coordinates associated with each partial image to generate a single image representing the target area of ​​the entire image. Note that if the imaging device 190 can capture an image with a resolution desired by the user so that the target area of ​​the entire image fits within one field of view, linking processing is not necessary.

[0035] The display control unit 230 controls displaying a user interface (UI) screen and processing results of each unit on the display 150. The input receiving unit 240 receives instruction input from a user via the input device 130. The data management unit 250 stores and manages data related to processing, such as partial images, in a storage unit such as the ROM 102 or the HDD 170.

[0036] The CPU 101 uses the RAM 103 as a work memory and reads and executes a program stored in the ROM 102 or the HDD 170, thereby causing each functional unit shown in Fig. 2 to function. Note that it is not necessary for all functional units shown in Fig. 2 to be executed by the CPU 101, and the image processing device 100 may be configured so that part or all of the processing is performed by one or more processing circuits other than the CPU 101.

[0037] [Correcting the relative position of partial images] Fig. 4 is a diagram for explaining the positional relationship between two adjacent partial images. An example in which the relative positions of partial images are erroneously corrected will be described with reference to Fig. 4. Fig. 4(a) is an enlarged view of two partial images 404 and 405 from the group of partial images in Fig. 3(b), and the shaded areas indicate the overlapping areas of the respective partial images. Fig. 4(a) shows an example in which the overlapping areas are in a background area without dots 401 and 402.

[0038] Fig. 4(b) is a diagram showing a line profile in which pixel values ​​for each position (pixel) in dashed line portion 403 in Fig. 4(a) are graphed. Curve 406 on the left side of Fig. 4(b) shows the transition of pixel values ​​in partial image 404 on the left side of Fig. 4(a), and curve 407 on the right side of Fig. 4(b) shows the transition of pixel values ​​in partial image 405 on the right side of Fig. 4(a). As shown in Fig. 4(b), even in a background region that should be expressed as the same pixel value, the pixel values ​​at the edge of each partial image tend to be smaller than those at the center due to the effects of peripheral light falloff and the like.

[0039] The position correction of the partial images is performed by moving the partial images, comparing the pixel values ​​of the overlapping areas of the partial images after the movement, and searching for the position of the partial images where the degree of coincidence of the pixel values ​​is greater than other positions. FIG. 4(a) is an example of a partial image that does not require position correction, but FIG. 4(b) shows that, for example, the pixel value of the right end of the left partial image 404 has a large difference 408 with the pixel value of the corresponding position in the adjacent right partial image 405. If the difference in pixel values ​​is large, it is determined that the degree of coincidence is small. Therefore, when the position correction process is performed, the relative positions of the partial images are corrected to positions where the difference in pixel values ​​is small. Also, if noise occurs in the overlapping areas of the partial images, there is a risk that position correction will be performed to align the position of the noise in the same way.

[0040] Fig. 4(c) is a diagram showing the positional relationship between partial images 404 and 405 after the relative position of partial image 405 has been corrected from the state of Fig. 4(a) to a position where the degree of coincidence of pixel values ​​in the overlapping region increases. Fig. 4(d) is a line profile of dashed line portion 409. Curve 410 on the right side of Fig. 4(d) represents the transition of pixel values ​​of partial image 405 after the position correction shown in Fig. 4(c). Curve 406 on the left side of Fig. 4(d) is the same as curve 406 in Fig. 4(b) since the position of partial image 404 has not been changed.

[0041] As shown in Fig. 4(d), the difference 411 between the pixel value at the right end of the left partial image 404 and the pixel value at the corresponding position in the adjacent right partial image 405 is smaller than the difference 408. This means that the degree of agreement of the pixel values ​​in the overlapping area in Fig. 4(c) is greater than in the case of Fig. 4(a). In this way, when an overlapping area exists in an area such as a background that does not have information useful for position correction, position correction may be performed erroneously.

[0042] FIG. 5 is a diagram showing an example of the positional relationship between two adjacent partial images 501 and 502. The hatched areas in FIG. 5(a) represent overlapping areas between the partial images 501 and 502. As shown in FIG. 5(a), the overlapping areas include dots 503 formed with toner as characteristic information useful for position correction. FIG. 5(b) is a diagram showing a line profile that graphically represents pixel values ​​for each position in the dashed line area 504 in FIG. 5(a). A curve 505 on the left side of FIG. 5(b) represents the transition of pixel values ​​of the partial image 501 on the left side of FIG. 5(a), and a curve 506 on the right side of FIG. 5(b) represents the transition of pixel values ​​of the partial image 502 on the right side of FIG. 5(a).

[0043] Curves 505 and 506 in Fig. 5(b) also include the effect of peripheral light falloff in the overlapping region similar to that in Fig. 4(b), but the effect of this on the degree of match is smaller than in the case of pixels in the background, because the overlapping region includes dots 503. Even if the overlapping region includes noise, the effect of noise is smaller because position correction is performed based on information more useful for position correction, such as dots 503, rather than noise.

[0044] For this reason, when the overlapping region includes an image other than the background, such as a dot, it is possible to prevent erroneous position correction as described in Fig. 4. Therefore, by changing the overlapping region so that it includes information useful for position correction, such as a dot, it is possible to perform appropriate position correction. Therefore, in this embodiment, a method for determining imaging conditions for obtaining a partial image group will be described so that information useful for position correction is included in the overlapping region.

[0045] [About the UI screen] 6 is a diagram showing a UI screen 600 that the display control unit 230 displays on the display 150. The user can set the imaging conditions of the imaging device 190 via the UI screen 600. An area 601 in the UI screen 600 is an area that displays a live view screen of the imaging device 190 capturing an image, which is sent from the imaging device 190 via the serial bus 120. The user can set imaging conditions such as a target area, which is the imaging range of the entire image, while referring to the image in the area 601.

[0046] For example, a user can use an input device 130 such as a mouse to specify a target area for obtaining an entire image. The target area of ​​the entire image is a rectangular area determined by the coordinates of two points (start point and end point) indicating the position of an object placed on a stage to be imaged. For example, the user can specify the coordinates of the two points by referring to the live view display.

[0047] When the user clicks on any position within the region 601, the display control unit 230 derives the relative coordinates of the selected pixel in the XY coordinate system of the stage surface and displays them in selected coordinates 602. The selected coordinates 602 are updated every time the user performs a click action. Then, when the user presses a set button 603, the currently displayed selected coordinates 602 are set to the start point of the target region of the entire image (the upper left coordinate when the target region is a rectangle). Similarly, when the user presses a set button 604, the currently displayed selected coordinates 602 are set to the end point of the target region (the lower right coordinate when the target region is a rectangle).

[0048] Alternatively, the user may directly input the coordinates of the start point and the coordinates of the end point, in which case the stage is moved according to the coordinates specified by the user, and a live view of the target object after the stage movement is displayed in area 601. The user inputs the coordinates and checks the imaging range using the live view one by one, and when the stage has been moved to the desired position, the user can set the target area of ​​the entire image by pressing set buttons 603, 604.

[0049] Also, the user can specify the magnification, exposure time, or ISO sensitivity (gain) for capturing the target area of ​​the entire image by selecting a desired value from pull-downs 605 to 607. Alternatively, the maximum magnification that can include the target area set by the user may be derived for the magnification, and the magnification may be set as the magnification for capturing the target area of ​​the entire image. For example, the angle of view of one field of view for each lens that can be selected by the imaging device 190 may be stored in advance, and the lens with the maximum magnification may be selected from among lenses having an angle of view that includes the target area of ​​the entire image, and the lens magnification may be determined as the imaging condition. On the other hand, when the magnification is set by the user, the user can select the number of partial images that represent the target area of ​​the entire image.

[0050] Furthermore, the user can select a desired magnification from a pull-down menu 610 as an imaging condition for acquiring a group of partial images.

[0051] Furthermore, the user can select an initial value of the overlap rate as an imaging condition for acquiring partial images from a pull-down menu 611. The overlap rate is used to determine an overlapping area of ​​partial images. Then, when the user presses an update button 612, the overlap rate is updated so that the overlapping area includes information useful for position correction. The imaging range of each partial image is determined based on the updated overlap rate, the specified magnification, and the coordinates in the target area of ​​the entire image. Then, the display control unit 230 displays the imaging range 613 of the partial image superimposed on the entire image displayed in the area 609. When the user presses a capture button 614, a series of processes from acquisition of a group of partial images to connection processing of the partial images is executed.

[0052] [About image data] The image data obtained by imaging by the imaging device 190 will be described. The image is composed of 1024 x 768 pixels, and each pixel stores a pixel value of three RGB channels. The pixel value is 16 bits ranging from 0 to 65535, and the description will be given assuming the TIFF format. Note that the image format does not need to be limited to this, and may be, for example, a grayscale image in which the distance (height) in the vertical direction on the XY plane is held as the pixel value.

[0053] [Image processing device operation] Fig. 7 is a flowchart showing the processing contents of the image processing device 100 in this embodiment. The details of the processing contents of the image processing device 100 in this embodiment will be described with reference to Fig. 7. The processing contents shown in the flowchart are executed by the CPU 101 after program code stored in the ROM 102 is expanded in the RAM 103. In addition, some or all of the functions of the steps in Fig. 7 may be realized by hardware such as an ASIC or an electronic circuit. The symbol "S" in the explanation of the flowchart represents a step, and the same applies to the explanation of the flowcharts below.

[0054] In S701, the display control unit 230 displays a UI screen 600 for a user to set imaging conditions on the display 150. The user operates the input device 130 to input an instruction to display the UI screen 600, and the input receiving unit 240 receives the input instruction, whereby the display control unit 230 displays the UI screen 600 on the display 150.

[0055] In S702, the imaging condition acquisition unit 213 acquires imaging conditions for the entire image. The imaging conditions for the entire image include coordinate information of the target area, lens magnification (resolution, angle of view), exposure time, ISO sensitivity, etc. The imaging conditions for the entire image to be acquired are the contents set by the user via the UI screen 600.

[0056] In S703, the whole image acquisition unit 211 controls the imaging device 190 and instructs the imaging device 190 to capture an image of the target area of ​​the object based on the imaging conditions acquired in S702. This step is executed in response to the user pressing the capture button 608.

[0057] The imaging device 190 moves the stage position to the center coordinate between the coordinates of the start point and the coordinates of the end point indicating the target area, and captures an image of the target object based on the exposure time, ISO sensitivity, etc. included in the imaging conditions. The whole image acquisition unit 211 acquires an whole image, which is an image obtained as a result of the imaging, from the imaging device 190. The acquired image data is stored in the storage unit by the data management unit 250.

[0058] In S704, imaging conditions for acquiring partial images are determined. This step is executed in response to the user pressing the update button 612. Details will be described later.

[0059] In S705, the partial image group acquisition unit 212 instructs the imaging device 190 to capture an image to obtain a partial image of the target area of ​​the object based on the imaging conditions derived in S704. The steps following this step are executed in response to the user pressing the capture button 614.

[0060] The partial image group is acquired by the imaging device 190 sequentially moving the stage and capturing images. The amount of stage movement (XY coordinates) performed when acquiring each partial image is derived based on the overlap rate determined in S704 and the target area of ​​the entire image acquired in S702. The partial image group acquisition unit 212 acquires the partial image group obtained as a result of the imaging from the imaging device 190. The initial value of the relative position associated with the partial image is determined based on the amount of stage movement during imaging.

[0061] In S706, the position correction unit 224 corrects the relative coordinates associated with the partial image. The correction method is performed by moving the partial image to be processed and searching (matching) a position where the degree of match of the images in the overlapping area between the partial image to be processed and a partial image adjacent to the partial image to be processed is high. The position where the degree of match is high is then set as the post-correction position, and the relative coordinates associated with the partial image to be processed are corrected. This process is performed sequentially for each partial image.

[0062] In the following description, the average squared error of pixel values ​​in the overlapping area is used as a value for evaluating the degree of match. The smaller the average squared error, the greater the degree of match is determined to be. The average squared error is calculated by locating two adjacent partial images in relative coordinates and calculating the square of the difference in pixel value between a pixel in one partial image and the corresponding pixel in the other partial image in the overlapping area of ​​the two partial images. This is calculated for all pixels in the overlapping area, and the calculated squared errors are summed up and divided by the number of pixels in the overlapping area.

[0063] In this embodiment, the position correction unit 224 selects two images, a partial image to be processed (called the N+1 image) and a partial image adjacent to the partial image to be processed (called the N image).The position correction unit 224 then calculates the average value of the square error of the overlapping region between the N image and the N+1 image when the N+1 image is moved from the initial relative coordinate to each coordinate position within the search range.

[0064] Here, in this embodiment, the search range is, for example, 3pxl uniformly. When the search range is 3pxl, the position correction unit 224 moves the N+1 image from the initial coordinate position to ±3px in the x and y axis directions (-3, -2, -1, 0, +1, +2, +3 in each of the X and Y directions) and calculates the average value of the squared error when each is moved. Therefore, when the search range is 3pxl, the position correction unit 224 calculates the average value of the squared error in 7×7 ways. As a result, the position correction unit 224 sets the position where the average value of the squared error is the smallest (the degree of agreement is the largest) as the corrected relative coordinate of the N+1 image. The data management unit 250 updates the relative coordinate associated with the managed N+1 image to the corrected relative coordinate. When the number of partial image groups is N, the position correction is performed N-1 times, and the position correction is performed on at least one partial image in the partial image group.

[0065] In S707, the image connection unit 225 generates data for one image by arranging and connecting each partial image based on the corrected relative coordinates. The image connection unit 225 determines the pixel values ​​of the overlapping area by a weighted linear sum of the pixel values ​​of each image. For example, the weighting coefficient of the pixel values ​​of each image is determined according to the distance from the center position of the image to the target pixel. The sum of the weighting coefficients is set to 1.

[0066] [Partial image capture condition determination process] Fig. 8 is a flowchart for explaining the process of S704. The process of S704 will be explained in detail with reference to Fig. 8. The process of S704 is a process for analyzing the entire image and determining an appropriate overlap rate so that information useful for position correction is included in the overlapping region.

[0067] In S801, the imaging condition acquisition unit 213 acquires parameters such as imaging conditions for partial images. The parameters acquired in this step include at least the resolution of the partial images and an initial value of the overlap rate set by the user via the UI screen 600. The initial value of the overlap rate may not be set by the user, but may be a value stored in advance.

[0068] In S802, the overlapping area determining unit 221 determines an overlapping area of ​​a partial image that occurs when imaging is performed to obtain a partial image with a desired resolution set by the user, the current overlapping rate, and the target area of ​​the entire image.

[0069] The overlap area determination unit 221 determines an area in the entire image that satisfies the x and y coordinates in the following formula 1 as an overlap area. In formula 1, D1 is the resolution D1 of the entire image acquired in S702, and D2 is the resolution D2 of the partial image group acquired in S801. Furthermore, Ix is the horizontal length of the entire image, and Iy is the vertical length of the entire image. O is the overlap rate O, and initially, the initial overlap rate set by the user is substituted.

[0070] D1 / D2×Ix×(1-O+n)≦x≦D1 / D2×Ix×(1+n) D1 / D2×Iy×(1-O+n)≦y≦D1 / D2×Iy×(1+n) Equation (1) where n is an integer greater than or equal to zero.

[0071] The overlapping area determination unit 221 sets the pixel values ​​of pixels whose x and y coordinates satisfy the formula 1, respectively, among pixels in an image of the same image size as the entire image, to 1, and sets the pixel values ​​of pixels that do not satisfy the formula 1 to 0, thereby generating a binary image (mask image) representing the overlapping area. The binary image is managed by the data management unit 250.

[0072] In S803, the evaluation value derivation unit 222 derives an evaluation value for the overlapping region determined in S802. The evaluation value is an index representing the likelihood (accuracy) of whether information useful for position correction is included in the overlapping region, and is used to determine whether useful information for performing the above-mentioned position correction is included.

[0073] In this embodiment, the evaluation value derivation unit 222 generates a histogram of pixel values ​​of the overlapping region in the entire image. The evaluation value derivation unit 222 determines the minimum pixel value Imin and the maximum pixel value Imax from among the pixel values ​​whose frequency is equal to or greater than a predetermined value, and derives the difference between Imax and Imin as the evaluation value.

[0074] If the overlapping region contains characteristic information that is useful for position correction, such as the dots 302 in Fig. 3, the pixel values ​​in the overlapping region will vary greatly. On the other hand, if the overlapping region does not contain information that is useful for position correction, such as dots, the pixel values ​​in the overlapping region will show values ​​close to each other. Therefore, a value that indicates the dispersion of pixel values ​​in the overlapping region, such as the difference between Imax and Imin, can be used as an evaluation value for whether the overlapping region contains information that is useful for position correction.

[0075] In this embodiment, the range in which pixel values ​​appear is used as the evaluation value, but the method of deriving the evaluation value is not limited to the above-mentioned method as long as the variance of pixel values ​​in the overlapping region can be evaluated. For example, the variance or standard deviation of pixel values ​​in the overlapping region may be used as the evaluation value, or the evaluation value may be determined based on the contrast of the images.

[0076] Furthermore, the partial images may be numbered as shown in FIG. 13. In this case, the image to be processed (N+1 image) and the partial image to which the number before the image to be processed (N image) are assigned may be selected from the partial image group, and the position of the N+1 image may be corrected using the overlapping area between the N image and the N+1 image. Then, by selecting N in order starting from 1, the position of the entire partial image group may be corrected. In this case, among the overlapping areas determined in S802, there will be overlapping areas that are not included in the position correction. For this reason, the evaluation value derivation unit 222 may determine the evaluation value excluding the overlapping areas that are not used for the position correction.

[0077] In S804, the evaluation value derivation unit 222 determines whether the evaluation value derived in S803 is greater than a predetermined value. If the evaluation value is greater than the predetermined value (YES in S804), it is considered that the overlapping area contains information useful for performing position correction. Therefore, since there is no need to change the overlapping area, the current overlapping rate O is determined as the imaging condition, and the process of this flowchart ends.

[0078] On the other hand, if the evaluation value is equal to or less than the predetermined value (NO in S804), it is considered that there is a lack of information useful for performing position correction in the overlapping area, so proceed to S805 to change the imaging conditions so that the current overlapping ratio O is changed to increase the overlapping area of ​​the partial images.

[0079] In S805, the imaging condition determination unit 223 adds a predetermined amount α to the current overlap ratio O to update the overlap ratio O. The updated value, O+α, is set as the new overlap ratio O. α is a value determined in advance within the range of 0<α<1.0.

[0080] In S806, the imaging condition determination unit 223 determines whether the updated overlap rate O is greater than a predetermined value ε. If the updated overlap rate O is greater than ε (YES in S806), the process proceeds to S807. The larger the overlap rate O is, the larger the size of the overlap area becomes, and the number of partial images that cover the target area of ​​the entire image increases. For this reason, ε is set as a threshold value, and if the updated overlap rate O is greater than ε, it is determined that the overlap area cannot be determined so as to include useful information for performing position correction processing, and the process of updating the overlap rate O is terminated.

[0081] In S807, the imaging condition determination unit 223 returns the value of the overlap rate O to the initial value and ends the process. If partial images are captured with the initial value of the overlap rate set in this step, the overlapping area of ​​the partial images will lack information useful for position correction. For this reason, in this step, the display control unit 230 may display a warning to inform the user that position correction may not be performed appropriately.

[0082] On the other hand, if the updated overlap rate O is equal to or less than ε (NO in S806), the process returns to S802. The process of S802 to S806 is then repeated until an overlap rate that includes information useful for position correction in the overlap region is derived.

[0083] As described above, according to this embodiment, by determining the overlapping area so that it contains information useful for position correction, it is possible to prevent erroneous position correction of a partial image even if the partial image is affected by noise or peripheral light loss.

[0084] The evaluation value may be determined based on a plurality of factors. For example, the evaluation value derivation unit 222 derives the number of captured partial images covering the target region for each determined overlap rate. Then, the evaluation value derivation unit 222 may obtain a weighted sum of the evaluation value of the overlap region determined based on each overlap rate and the number of captured partial images, and determine the resulting value as a new evaluation value. In this case, the evaluation value is derived such that the greater the number of captured partial images, the lower the evaluation value. Therefore, since the evaluation value can be derived taking into consideration the number of captured images and the imaging time, by determining imaging conditions with a high evaluation value, the number of captured images and the imaging time can be reduced, and imaging conditions can be determined so that the position correction of the partial images can be appropriately performed.

[0085] <Embodiment 2> In the first embodiment, a method for changing the overlap rate so that the overlap area contains information useful for position correction has been described. However, depending on the object of imaging or the target area of ​​the entire image specified by the user, the overlap area may not contain information useful for position correction even if the overlap rate is increased. In addition, when the overlap rate is increased, the number of partial images to cover the target area of ​​the entire image increases, and the imaging time to obtain the partial images may increase.

[0086] Therefore, in this embodiment, a method of determining an exposure time different from that in normal imaging and correcting the position of a partial image using an image obtained by imaging with the determined exposure time will be described. For example, by changing the exposure time to a longer time, the captured image will express a fine texture of the background, and the fine texture of the background can be used as information useful for position correction. In this embodiment, differences from the first embodiment will be mainly described. Parts not specifically mentioned have the same configuration and processing as the first embodiment.

[0087] [Image processing device operation] Fig. 9 is a flowchart showing the processing contents of the image processing device 100 in this embodiment. Hereinafter, the details of the processing contents of the image processing device 100 in this embodiment will be described with reference to Fig. 9. Note that, the processing contents shown in the flowchart are executed by the CPU 101 after program code stored in the ROM 102 is expanded in the RAM 103.

[0088] S901 is the same processing as S701, and the display control unit 230 displays on the display 150 the UI screen 600 for the user to set the imaging conditions of the imaging device 190.

[0089] In S902, the imaging condition acquisition unit 213 acquires imaging conditions for the entire image and imaging conditions for the partial image group. As imaging conditions, the contents set by the user via the UI screen 600 are acquired. The imaging conditions include the imaging target area, lens magnification (resolution, angle of view), ISO sensitivity, overlap rate, exposure time suitable for imaging the entire area, etc. The imaging conditions determined in this step are common conditions that do not depend on the imaging position.

[0090] In S903, the entire image acquisition unit 211 controls the imaging device 190 to capture the target area under the imaging conditions acquired in S902, and acquires an entire image of the target area. Note that in this embodiment, this step may be skipped.

[0091] In S904, the imaging ranges of the partial images of the object are determined based on the imaging conditions acquired in S902, and an imaging range to be processed is selected from the determined imaging range group. Then, the processing of S905 to S906 is performed on the imaging range to be processed. When the processing on the imaging range to be processed is completed, the processing returns to this step, and an imaging range to be processed is selected from the unprocessed imaging ranges.

[0092] In S905, a process for determining an exposure condition when capturing an image of the image capturing range to be processed is performed. The process of this step will be described with reference to the flowchart of FIG.

[0093] In steps S1001 to S1006, a process is performed to determine an exposure time for the entire imaging range to be processed. The exposure time for the entire range is the exposure time used for imaging to obtain partial images to be actually joined.

[0094] In S1001, the imaging condition determination unit 223 stores an initial value in a variable Ev representing an exposure time for the entire area of ​​a partial image, and captures an imaging range to be processed at the exposure time Ev. In this embodiment, the initial value of the variable Ev is described as a value previously set for capturing a partial image, but a value designated by the user via a UI screen may be used.

[0095] In S1002, the imaging condition determination unit 223 creates a histogram for each RGB signal value of the pixel value of the image captured in the previous step, and calculates the frequency of the maximum and minimum values ​​of each signal value. In this embodiment, the maximum value of each RGB signal value is 255, and the minimum value is 0.

[0096] In S1003, if there is a histogram in which the frequency of maximum and minimum values ​​in the histogram of each RGB signal value is greater than a predetermined value, the imaging condition determination unit 223 determines that the current exposure time Ev is not an appropriate exposure time.

[0097] If the current exposure time Ev is not appropriate (YES in S1003), proceed to S1004. In S1004, the imaging condition determination unit 223 updates the exposure time Ev for the entire range to Ev=Ev+β1. Note that, if the frequency of minimum signal values ​​is high in the histogram generated in S1002, β1 is set to a positive value, and if the frequency of maximum signal values ​​is high, β1 is set to a negative value.

[0098] In S1005, the imaging condition determination unit 223 captures the imaging range to be processed with the updated variable Ev. Then, the process returns to S1002, and the histogram is generated again, and it is determined whether the current exposure time Ev is an appropriate exposure time for the entire range.

[0099] If the current exposure time Ev is an appropriate exposure time (NO in S1003), the process proceeds to S1006. In S1006, the imaging condition determination unit 223 determines the current exposure time Ev as the exposure time for the entire imaging range to be processed.

[0100] In the next steps S1007 to S1013, a process for determining an exposure time for the background in the imaging range to be processed is performed. The partial image captured with the exposure time for the background is used for position correction.

[0101] First, the current exposure time Ev is stored as the initial value of a variable Ev' representing the exposure time for the background, and the following steps are started.

[0102] In S1007, the imaging condition determination unit 223 binarizes the image captured in S1001. In this embodiment, the well-known Otsu's binarization method is used, but the binarization method is not limited to this method.

[0103] In S1008, the imaging condition determination unit 223 counts the white pixels and black pixels of the image binarized in S1007, and determines the pixel value with the larger count number (area) as the background pixel. As information indicating the background of the imaging range to be processed, a binary image (mask image) in which a binary value indicating whether or not each coordinate of the image is a background part is stored is generated and held.

[0104] In S1009, the imaging condition determination unit 223 creates a histogram for each RGB signal value of the pixel value of the pixel in the background region determined in S1008 from the image obtained by capturing the imaging range to be processed with the current exposure time Ev'. Then, the frequency of the maximum and minimum values ​​of each signal value is calculated.

[0105] In S1010, the imaging condition determination unit 223 determines that the current exposure time Ev' is not an appropriate exposure time for the background if there is a histogram in which the frequency of maximum and minimum values ​​is greater than a predetermined value in the histograms of each signal value generated in S1009. Here, the predetermined value is a value different from the value used in S1003, and the determination in this step is performed so that the exposure time Ev' for the background is determined to be longer than the exposure time Ev for the entire area.

[0106] If the current exposure time Ev' is not an appropriate exposure time for the background (YES in S1010), proceed to S1011. In S1011, the imaging condition determination unit 223 updates the exposure time Ev' for the background to Ev' = Ev' + β2. Note that if the frequency of minimum signal values ​​is high in the histogram generated in S1009, β2 is a positive value, and if the frequency of maximum signal values ​​is high, β2 is a negative value.

[0107] In S1012, the imaging condition determination unit 223 captures the imaging range to be processed with the updated variable Ev'. Then, the process returns to S1009, and a histogram of background pixels is generated again, and it is then determined whether the current exposure time Ev' is an appropriate exposure time for the background.

[0108] If the current exposure time Ev' is an appropriate exposure time for the background (S1010: NO), the process proceeds to S1013. In S1013, the imaging condition determination unit 223 determines the current exposure time Ev' as the exposure time for the background of the imaging range to be processed.

[0109] Alternatively, steps S1007 to S1012 may be skipped, and in step S1013, the exposure time Ev' for the background may be determined by calculating Ev'=Ev+β2, where β2 is a positive value.

[0110] Returning to Fig. 9, the description of the flowchart in Fig. 9 will be continued. In S906, the partial image group acquisition unit 212 controls the imaging device 190 to capture the imaging range to be processed using the exposure time for the entire area of ​​the partial image determined in S903. Then, the imaging device 190 captures the imaging range to be processed using the exposure time for the background. As a result, in this step, two partial images are acquired for the imaging range to be processed.

[0111] In S907, it is determined whether all partial images covering the target area of ​​the entire image have been acquired. If there is an unprocessed imaging range (NO in S907), the process returns to S904. If there is no unprocessed imaging range (YES in S907), the process proceeds to S908.

[0112] By repeating steps S904 to S907 in this manner, stage movement and imaging are performed sequentially, and partial images obtained by imaging with an exposure time for the entire area and partial images obtained by imaging with an exposure time for the background can be obtained for the entire target area of ​​the overall image.

[0113] In S908, the overlapping area determination unit 221 determines the overlapping area of ​​the partial images based on the resolution and overlapping rate set by the user, the target area of ​​the entire image, and the resolution of the entire image. Then, the overlapping area determination unit 221 generates a binary image (mask image) representing the overlapping area, similar to S802. The binary image is managed by the data management unit 250.

[0114] In S909, the position correction unit 224 corrects the relative coordinates associated with the partial images. In the position correction process of this embodiment, partial images captured with an exposure time for the background are used. The method of position correction is the same as that of S706. That is, from the partial image group acquired in S906 and captured with an exposure time for the background, two partial images adjacent to each other are selected, and the relative coordinates are corrected so that the overlapping regions of the selected partial images have a high degree of coincidence.

[0115] Since the exposure time determined for the entire area is set taking into consideration the influence of noise, etc., the fine texture of the background is difficult to express in the partial image obtained by capturing an image with the exposure time for the entire area. The exposure time for the background derived in this embodiment is determined so that the fine texture of the background part is expressed in the partial image. Therefore, in this embodiment, an image of the fine texture of the background part can be included in the overlapping area as useful information for performing position correction. As a result, even if the overlapping area does not include an image such as a dot, it is possible to perform appropriate position correction.

[0116] In S910, the image linking unit 225 arranges and links each partial image obtained by capturing the image with the exposure time for the entire area based on the corrected relative coordinates to generate data for one image. The linking method is the same as in S707, so the explanation is omitted.

[0117] As described above, according to this embodiment, even if the overlapping area does not include an image such as a dot, it is possible to include useful information for performing position correction in the overlapping area. Therefore, it is possible to improve the accuracy of position correction without increasing the overlapping rate.

[0118] <Embodiment 3> In the method of the first embodiment, even if the overlap rate is increased, the overlap area may not contain information useful for position correction. In addition, when the overlap rate is increased, the number of partial images required to cover the target area of ​​the entire image increases, and the imaging time required to obtain the partial images increases.

[0119] Therefore, in this embodiment, a form in which the search range is changed according to the evaluation value of the overlapping region will be described. In this embodiment, the differences from the first embodiment will be mainly described. The configuration and processing are the same as those of the first embodiment unless otherwise specified.

[0120] [Functional configuration of image processing device] 11 is a block diagram showing the functional configuration of the image processing device 100 of this embodiment. The same functional blocks as those of the image processing device 100 of the first embodiment are given the same reference numbers and the description thereof will be omitted.

[0121] The imaging control unit 1110 has a partial image group acquisition unit 212 and an imaging condition acquisition unit 213. Note that the imaging condition acquisition unit 213 of this embodiment acquires various conditions for acquiring partial images.

[0122] On the other hand, the information processing unit 1120 has an overlapping region determining unit 221 , an evaluation value derivation unit 222 , a position correcting unit 224 , a search range determining unit 1124 , and an image connecting unit 225 .

[0123] The search range determination unit 1124 derives a search range for each overlapping area, which is an area in which a search (matching) is performed to correct the relative coordinates between the partial images. The search range determination unit 1124 determines the search range based on the evaluation value of the overlapping area. The smaller the evaluation value of the overlapping area, the narrower the search range determined for correcting the relative positions of the two partial images including the overlapping area. In this way, the search range determination unit 1124 can perform processing to suppress the influence when useful information for position correction is not included in the overlapping area. Details of the processing by the search range determination unit 1124 will be described later.

[0124] The CPU 101 uses the RAM 103 as a work memory and reads and executes a program stored in the ROM 102 or the HDD 170, thereby causing each functional unit shown in Fig. 11 to function. Note that it is not necessary for all functional units shown in Fig. 11 to be executed by the CPU 101, and the image processing device 100 may be configured so that part or all of the processing is performed by one or more processing circuits other than the CPU 101.

[0125] [Operation of image processing device] Fig. 12 is a flowchart showing the processing contents of the image processing device 100 in this embodiment. The details of the processing contents of the image processing device 100 in this embodiment will be described with reference to Fig. 12(a) and Fig. 12(b). The processing contents shown in the flowchart are executed by the CPU 101 after program code stored in the ROM 102 is expanded in the RAM 103.

[0126] S1201 is the same processing as S701, and the display control unit 230 displays on the display 150 the UI screen 600 for the user to input the imaging conditions of the imaging device 190.

[0127] In S1202, the imaging condition acquisition unit 213 acquires imaging conditions for the partial image group. The imaging conditions acquired are imaging conditions and the like designated by the user via the UI screen 600. The imaging conditions include the target area, lens magnification (resolution, angle of view), ISO sensitivity, overlap rate, exposure time, and the like. The target area is the same as in the first embodiment, and therefore a detailed description thereof will be omitted.

[0128] In S1203, the partial image group acquisition unit 212 controls the imaging device 190 to instruct imaging to obtain partial images of the target area of ​​the object based on the imaging conditions acquired in S1202. The partial image group is acquired by sequentially moving the stage and capturing images by the imaging device 190. The amount of stage movement (XY coordinates) performed when capturing images to acquire each partial image is derived based on the overlap rate and the target area.

[0129] The partial image group acquisition section 212 acquires the partial image group obtained as a result of the imaging from the imaging device 190. The initial value of the relative position associated with the partial image is determined based on the amount of stage movement during imaging.

[0130] In step S1204, the relative positions associated with the respective partial images are corrected, as will be described in detail later.

[0131] In S1205, similarly to S707, the image connection unit 225 arranges and connects each partial image based on the corrected relative coordinates to generate data for one image.

[0132] [Details about position correction processing] Fig. 13 is a diagram for explaining numbering of partial images. Rectangles indicated by dashed lines represent partial images. For example, rectangle 1301 represents the relative position of partial image numbered 17. As shown in Fig. 13, the partial images of this embodiment are numbered as shown in Fig. 13 before position correction is performed. In Fig. 13, a coordinate system 1302 is used in which the origin is at the upper left, the vertical direction is the Y direction, and the horizontal direction is the X direction.

[0133] The numbers assigned to the partial images are set as follows: 1 is assigned to the image located at the smallest position from the origin of the XY coordinates in the initial relative coordinate values ​​based on the amount of stage movement during imaging, and 2 is assigned to the image with the smallest Euclidean distance from the partial image numbered 1. Thereafter, x+1 is assigned to the partial image with the smallest distance from the partial image numbered x and to which no number has been assigned. Next, x is updated to x=x+1, and the same process is repeated thereafter to assign numbers to the partial images as shown in Fig. 13.

[0134] In the partial image position correction process of this embodiment, a partial image assigned the number N is selected from the partial image group. Then, the partial image assigned the next number N+1 (referred to as the N+1 image) is selected as the partial image to be processed. Then, the relative coordinates of the N+1 image are updated based on the overlapping area when the N image and the N+1 image are arranged in relative positions. This process is performed sequentially from partial image 1 in FIG. 13 to image N, thereby correcting the overall position.

[0135] Next, the details of the position correction process of this embodiment will be described using the flowchart in Fig. 12(b). In the following flowchart, the position correction of the N+1 image when the N+1 image is the processing target will be described. In reality, the process of the following flowchart is performed N-1 times.

[0136] In S1211, the search range determination unit 1124 acquires the initial values ​​of the relative coordinates between the N image and the N+1 image. The initial values ​​of the relative coordinates are determined based on the amount of stage movement during imaging, and are managed by the data management unit 250. Therefore, in this step, the initial values ​​of the relative coordinates are acquired from the data management unit 250.

[0137] In S1212, the search range determination unit 1124 determines an overlapping area between the images when the N image and the N+1 image are arranged two-dimensionally based on the acquired relative coordinates. Note that a binary two-dimensional map is generated and stored as data indicating the area determined here. A binary two-dimensional map is also generated for the overlapping area after position correction in the same manner. It is possible to determine whether or not it is an overlapping area by referring to the two-dimensional map.

[0138] If the positions of the N image and the N+1 image are adjacent in the vertical direction in FIG. 13, the areas in the N image and the N+1 image that satisfy the y coordinate of the following equation are determined as overlapping areas.

[0139] D1 / D2×Iy×(1-O+n)≦y≦D1 / D2×Iy×(1+n) If the positions of the N image and the N+1 image are adjacent in the horizontal direction in FIG. 13, the areas in the N image and the N+1 image that satisfy the x coordinate of the following equation are determined as overlapping areas.

[0140] D1 / D2×Ix×(1-O+n)≦x≦D1 / D2×Ix×(1+n) In this embodiment, Iy and Ix are the vertical and horizontal lengths of the target area specified by the user.

[0141] In S1213, the evaluation value derivation unit 222 derives an evaluation value in the overlapping region. In this embodiment, the evaluation value is the relative standard deviation (coefficient of variation) of pixel values ​​in the overlapping region. The evaluation value may be calculated from pixel values ​​in the overlapping region of either the N image or the N+1 image, or the evaluation value may be an average value obtained by calculating relative standard deviations from pixel values ​​in the overlapping regions of both the N image and the N+1 image. Note that, as in the first embodiment, the evaluation value may be determined by determining the minimum pixel value Imin and the maximum pixel value Imax, and the difference between Imax and Imin. The evaluation value may be managed by capturing an entire image and associating an evaluation value with each overlapping region in the entire image.

[0142] In S1214, the search range determination unit 1124 determines a search range for correcting the relative coordinates of the N+1 image. In this embodiment, the search range is determined based on a correspondence table between evaluation values ​​and search ranges. The correspondence table is recorded in the storage unit. Therefore, the search range determination unit 1124 first obtains the correspondence table between evaluation values ​​and search ranges from the storage unit.

[0143] Fig. 14 is a diagram for explaining evaluation values ​​and search ranges. Fig. 14(a) is a diagram showing an example of a correspondence table between evaluation values ​​and search ranges. The correspondence table is a one-dimensional lookup table (one-dimensional LUT) in which the correspondence relationship between evaluation values ​​and search ranges is recorded. The search range is determined by performing an interpolation operation with reference to the correspondence table using the evaluation value derived in S1213.

[0144] The search range determination unit 1124 may determine the search range by converting the evaluation value into the search range based on a predetermined formula other than the correspondence table. For example, the search range determination unit 1124 may derive the search range from the evaluation value based on the sigmoid function indicated by the curve of the graph in FIG. 14(b).

[0145] In S1215, the position correction unit 224 moves the N+1 image from the initial value of the relative coordinates to each coordinate position within the search range determined in S1214, and searches for a position where the degree of match between the images in the overlapping area where the N image and the moved N+1 image overlap is the highest. The search method is the same as the method described in S706 in the first embodiment.

[0146] For example, if the search range is determined to be 0px, the position correction unit 224 does not perform position correction of the N+1 image. Thus, in this embodiment, if the evaluation value of the overlapping area is lower than a predetermined value, correction of the relative position of the N+1 image is not performed. Therefore, if the overlapping area does not contain information useful for position correction, it is possible to prevent erroneous position correction.

[0147] On the other hand, the larger the evaluation value, the wider the search range is, which can improve the accuracy of position correction for partial images in which information useful for position correction is present in the overlapping region.

[0148] In S1216, the data management unit 250 updates the relative coordinates determined in S1215 so that they become the relative position associated with the N+1 image, and stores the updated relative coordinates in the storage unit.

[0149] As described above, in this embodiment, in the correction process of the relative coordinates of multiple partial images, the search range is determined according to the image characteristics in the overlapping area. Therefore, according to this embodiment, even if there is no information useful for position correction in the overlapping area, it is possible to suppress erroneous position correction. In addition, the higher the evaluation value, the wider the search range is determined. Therefore, it is possible to improve the accuracy of position correction of partial images having overlapping areas with high evaluation values.

[0150] In the first embodiment, if the overlapping area cannot include information useful for position correction even if the overlapping rate is increased, the search range may be derived by the method of the present embodiment to perform position correction. This makes it possible to prevent erroneous position correction from being performed even if the overlapping area does not include information useful for position correction.

[0151] <Embodiment 4> In the first and second embodiments, a method for setting the overlap rate or exposure time so that information useful for position correction is included in the overlapping area has been described. In this embodiment, when multiple candidates for lens magnification (resolution) for acquiring partial images are specified by the user, a method for determining the lens magnification from among the multiple candidates so that information useful for position correction is included in the overlapping area will be described. In this embodiment, the differences from the first embodiment will be mainly described. Portions not specifically mentioned have the same configuration and processing as the first embodiment.

[0152] [About image processing device processing] Fig. 15 is a flowchart showing the processing contents of the image processing device 100 in this embodiment. The details of the processing contents of the image processing device 100 in this embodiment will be described with reference to Fig. 15. The processing contents shown in the following flowchart are executed by the CPU 101 after program code stored in the ROM 102 is expanded in the RAM 103.

[0153] In S1501, the display control unit 230 displays on the display 150 a UI screen for the user to input imaging conditions of the imaging device 190. The input receiving unit 240 receives an instruction from the user, and the display control unit 230 displays the UI screen 600 on the display 150.

[0154] 16 is a diagram showing a UI screen 1600 of this embodiment that the display control unit 230 displays on the display 150. The same parts as those in the UI screen 600 of FIG. 6 are given the same numbers. In the UI screen 1600 of this embodiment, the user can specify the allowable range of lens magnification (resolution) when acquiring a partial image. For example, the user can specify the lens magnification that the user is willing to accept via the UI screen 1600 by selecting the maximum magnification and the minimum magnification of the lens magnification (resolution) that the user is willing to accept from pull-down menus 1601 and 1602.

[0155] Since steps S1502 and S1503 are similar to steps S702 and S703, the description thereof will be omitted.

[0156] In S1504, the imaging conditions for the partial images are determined. Details of the process in S1504 will be described with reference to FIG.

[0157] In S1511, the imaging condition acquisition unit 213 acquires parameters such as imaging conditions. The parameters acquired in this step include at least the resolution of the partial image and the initial value of the overlap rate set by the user via the UI screen 1600. The resolution of the partial image is specified as the lens magnification, and the lens magnification within the range permitted by the user is acquired. In this embodiment, the maximum magnification and the minimum magnification of the lens magnification are acquired.

[0158] In S1512, the overlapping area determination unit 221 determines an overlapping area of ​​partial images when partial images of a target area are captured at a resolution desired by the user and an overlapping rate set by the user. In this embodiment, the overlapping area of ​​partial images that occurs when partial images are captured under each imaging condition including a plurality of lens magnifications allowed by the user is determined.

[0159] For example, when the maximum and minimum lens magnifications are specified by the user, the overlapping area is determined for each predetermined magnification between the maximum and minimum magnifications. For example, when the minimum magnification is ×50 and the maximum magnification is ×100, the overlapping area is determined for each of ×50, ×60, ×70, ×80, ×90, and ×100.

[0160] In this step, the overlapping area when each lens magnification is set as an imaging condition is determined based on Equation 1, similarly to S802. Data indicating the overlapping area is held as a binary image (mask image) in which a binary value indicating whether or not each pixel of the entire image is an overlapping area is stored.

[0161] In S1513, the evaluation value derivation unit 222 derives the evaluation value of the overlapping region for each lens magnification determined in S1512. The evaluation value is derived in the same manner as in S803.

[0162] In S1514, the image capturing condition determination unit 223 selects the lens magnification with which the evaluation value derived in S1513 is maximized, and determines the lens magnification as the image capturing condition for acquiring the partial image.

[0163] Returning to Fig. 15(a), the description of the flowchart will continue. In S1505, the partial image group acquisition unit 212 controls the imaging device 190 to cause the imaging device 190 to capture images under the imaging conditions determined in S1504, and acquires a partial image group. As in the first embodiment, the partial image group is acquired by controlling the operation of the imaging device 190, and the imaging device 190 sequentially performing stage movement and imaging.

[0164] The processing in S1506 to S1507 is similar to that in S706 to S707, and therefore the description thereof will be omitted.

[0165] In S1508, the information processing unit 220 generates an evaluation value map image that visualizes the evaluation values ​​in the overlapping region, and the display control unit 230 displays the image. The information processing unit 220 of this embodiment has a function of an evaluation value map generation unit that generates an evaluation value map image.

[0166] A method for generating an evaluation value map image will be described. First, the information processing unit 220 generates a mask image for identifying overlapping regions, similar to S1504. Next, the information processing unit 220 generates a heat map that stores RGB values ​​converted from evaluation values ​​corresponding to each overlapping region. In this embodiment, the color indicating the evaluation value is derived using a linear expression that defines an R value as a fixed value with a maximum value of 255, and defines G and B values ​​as a maximum evaluation value of 255 and a minimum evaluation value of 0.

[0167] By displaying this evaluation value map image after a concatenated image of partial images is generated, the user can check which parts of the concatenated image may be inaccurate. The user can focus on checking parts of the concatenated image with low evaluation values, and if the results are unacceptable, change the imaging conditions and instruct re-imaging. This makes it possible to prevent the generation of an inappropriate concatenated image or the use of an inappropriate concatenated image. The evaluation value map image may also be generated in the above-mentioned embodiments 1 to 3.

[0168] 17A and 17B are diagrams for explaining a method for determining image capturing conditions in this embodiment. Fig. 17A is a diagram showing an object to be captured. Object 1701 is a sheet of paper on which toner particles 1702 are arranged. The rectangle indicated by the dashed line represents object area 1703 of the entire image.

[0169] 17(b), (c), and (d) are diagrams showing the imaging ranges of partial images under imaging conditions that differ only in lens magnification. In other words, they show the imaging ranges of partial images when imaging conditions other than the lens magnification (size of target area, overlap rate, etc.) are common. In Figs. 17(b), (c), and (d), the imaging ranges of each partial image are represented by dashed-dotted rectangles.

[0170] FIG. 17(c) is a diagram showing the imaging range of a partial image when imaging is performed under imaging conditions of high magnification (narrow angle of view) compared to FIG. 17(b). FIG. 17(d) is a diagram showing the imaging range of a partial image when imaging is performed under imaging conditions of low magnification (wide angle of view) compared to FIG. 17(b). At the lens magnifications shown in FIG. 17(c) and FIG. 17(d), the proportion of toner particles useful for position correction included in each overlapping region is high. For this reason, in this embodiment, the lens magnifications of FIG. 17(c) or FIG. 17(d) are determined as the imaging conditions for the partial image.

[0171] As described above, in this embodiment, from among a plurality of imaging conditions presented by a user, an imaging condition in which information useful for position correction is included in the overlapping area is determined. Therefore, according to this embodiment, appropriate position correction processing can be performed.

[0172] [Variations] In this embodiment, a method has been described in which an evaluation value of the overlapping region under each of a plurality of imaging conditions that differ only in lens magnification is calculated as the imaging condition for capturing the partial image, and the lens magnification with the highest evaluation value is set as the imaging condition for capturing the partial image. Alternatively, a plurality of imaging conditions that differ only in the target region of the entire image may be set, and an evaluation value of the overlapping region when captured under the set plurality of imaging conditions may be calculated, and the imaging condition with the highest evaluation value may be set as the imaging condition for capturing the partial image.

[0173] Fig. 18 is a diagram for explaining this modified example. Fig. 18(a) is a diagram showing an object to be imaged. The object 1801 will be described as a sheet of paper on which toner particles 1802 are arranged. The rectangle indicated by the dashed line represents the target area 1803 of the entire image.

[0174] 18(b) and (c) are diagrams showing the imaging range of a partial image under each imaging condition in which only the target region of the entire image is different. In FIG. 18(b) and (c), the imaging range of each partial image is represented by a dashed-dotted rectangle.

[0175] In the example of Fig. 18(b), a start point 1804 that defines the target area of ​​the entire image is the coordinates specified when the user specifies the target area. In order to capture a partial image that covers this target area, the imaging device 190 captures 4 x 3 images in raster scan order, and the partial image group acquisition unit 212 acquires the partial image group. On the other hand, in Fig. 18(c), a start point 1806 that defines the target area is set at a position different from the start point 1804. Fig. 18(c) shows the imaging range of the partial image when the size of the target area is changed to a different size from that in Fig. 18(b) by changing the start point (end point) from that in Fig. 18(b) and other imaging conditions are not changed.

[0176] In Fig. 18(b), the overlapping area determined based on the imaging conditions does not contain any useful information for performing position correction processing. On the other hand, in Fig. 18(c), the overlapping area determined based on the imaging conditions contains a lot of useful information for performing position correction processing. For this reason, the evaluation value of the overlapping area in Fig. 18(c) is higher than that in Fig. 18(b).

[0177] In this way, the user may specify multiple start points (end points) and specify multiple target areas of the entire image that the user allows. In this case, the image processing device 100 may derive an evaluation value for each of the multiple specified target areas and determine the target area with the maximum evaluation value as the imaging condition. By determining the imaging condition in this way, according to this embodiment, it is possible to perform appropriate position correction processing and generate a highly accurate connected image.

[0178] <Other embodiments> In the above-described embodiment, the imaging device 190 has been described as an optical microscope. However, the imaging device 190 may be any imaging device capable of controlling the movement of the relative coordinates with respect to an object, and the imaging device 190 is not limited to an optical microscope.

[0179] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions. [Explanation of symbols]

[0180] 100 Image processing device 212 Partial image group acquisition unit 223 Imaging condition determination unit 224 Position correction section 225 Image Linking Section

Claims

1. 1. An image processing device that corrects a position within a target area associated with a partial image obtained by capturing an image of a part of the target area of ​​an imaging object, and generates a connected image corresponding to the target area, An imaging condition acquisition means for acquiring a maximum magnification and a minimum magnification designated by a user; an overlap region determining means for determining an overlap region where a partial image of interest and another partial image adjacent to the partial image of interest overlap with each other among a plurality of partial images obtained from the target region when the image is captured at each of a plurality of magnifications included in the range from the minimum magnification to the maximum magnification; a derivation means for deriving an evaluation value of the overlapping area determined by the overlapping area determination means; a determining means for determining a magnification ratio at which the evaluation value derived by the derivation means is the highest as a magnification ratio for obtaining the partial image; an acquisition means for acquiring the plurality of partial images obtained by imaging based on the determined magnification; a correction means for correcting the position associated with at least one of the plurality of partial images based on the overlapping region; a generating means for generating the concatenated image by concatenating the plurality of partial images whose positions have been corrected; 13. An image processing device comprising:

2. An image processing device that corrects a position within a target area of ​​an imaged object associated with a partial image obtained by imaging a part of the target area, and generates a concatenated image corresponding to the target area, comprising: a determination means for determining a first exposure time for obtaining the partial images to be joined and a second exposure time for obtaining the partial images for performing the correction; an acquisition means for acquiring, for each imaging range of a partial image within the target area, a first partial image obtained by imaging the imaging range with the first exposure time and a second partial image obtained by imaging the imaging range with the second exposure time; a correction means for correcting the position associated with the corresponding first partial image by using an overlapping area in which a second partial image of interest among the plurality of second partial images obtained within the target area overlaps with another second partial image adjacent to the second partial image of interest; a generating means for generating the combined image by combining the first partial images whose positions have been corrected; 13. An image processing device comprising:

3. An image processing device that corrects a position within a target area of ​​an imaged object associated with a partial image obtained by imaging a part of the target area, and generates a concatenated image corresponding to the target area, comprising: an overlap region determining means for determining an overlap region where a partial image of interest overlaps with another partial image adjacent to the partial image of interest among a plurality of partial images obtained from within the target region when the image is captured under a predetermined imaging condition; a derivation means for deriving an evaluation value of the overlapping area determined by the overlapping area determination means; a determination means for determining an imaging condition based on the evaluation value so that predetermined information related to the imaging object is included in the overlapping area; an acquisition means for acquiring the plurality of partial images obtained by imaging based on the determined imaging conditions; a correction means for correcting the position associated with at least one of the plurality of partial images based on the overlapping region; a generating means for generating the concatenated image by concatenating the plurality of partial images whose positions have been corrected; having The evaluation value is determined based on the number of the plurality of partial images, and the larger the number, the lower the evaluation value is determined.

13. An image processing device comprising:

4. The determining means determines the imaging conditions so that the overlapping area is enlarged compared to the current imaging conditions when the evaluation value is equal to or smaller than a predetermined value.

4. The image processing device according to claim 3.

5. The imaging device further includes an imaging condition acquisition means for acquiring a plurality of imaging conditions based on an instruction from a user, The determining means determines, from among the plurality of imaging conditions, an imaging condition that results in the highest evaluation value as an imaging condition for obtaining the partial image.

4. The image processing device according to claim 3.

6. The imaging condition acquisition means acquires the plurality of imaging conditions each having a different magnification when imaging.

6. The image processing device according to claim 5,

7. The imaging condition acquisition means acquires the plurality of imaging conditions each having a different target area.

6. The image processing device according to claim 5,

8. The method further includes an overall image acquisition means for acquiring an overall image obtained by imaging the target area, The deriving means derives the evaluation value based on an image of a region in the entire image that corresponds to the overlap region.

8. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

9. The evaluation value is a value representing the likelihood that the overlapping region contains predetermined information related to the image capture object.

9. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

10. The predetermined information is an image other than the background.

10. The image processing device according to claim 9,

11. The evaluation value is a value that represents the variance of pixel values ​​in the overlapping region.

11. The image processing device according to claim 9,

12. a map generating means for generating a map image in which an image representing the evaluation value corresponding to the overlapping area is arranged for each of the overlapping areas; and a display control means for displaying the map image.

12. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

13. 1. An image processing method for generating a connected image corresponding to a target area of ​​an image capturing object by correcting a position within the target area associated with a partial image obtained by capturing an image of a part of the target area, the method comprising: an imaging condition acquisition step of acquiring a maximum magnification and a minimum magnification designated by a user; an overlapping area determination step of determining an overlapping area in which a partial image of interest and another partial image adjacent to the partial image of interest overlap among a plurality of partial images obtained from within the target area when the partial images are captured at each of a plurality of magnifications included in the range from the minimum magnification to the maximum magnification; a derivation step of deriving an evaluation value of the overlapping region determined in the overlapping region determination step; a determining step of determining a magnification ratio at which the evaluation value derived in the derivation step is the highest as a magnification ratio for obtaining the partial image; an acquisition step of acquiring the plurality of partial images obtained by imaging based on the determined magnification; correcting the position associated with at least one of the plurality of partial images based on the overlapping region; a generating step of generating the concatenated image by concatenating the plurality of partial images whose positions have been corrected; 13. An image processing method comprising:

14. An image processing method for generating a concatenated image corresponding to a target area of ​​an imaged object by correcting a position within the target area associated with a partial image obtained by imaging a part of the target area, comprising: a determining step of determining a first exposure time for obtaining the partial images to be joined and a second exposure time for obtaining the partial images for performing the correction; an acquisition step of acquiring, for each imaging range of a partial image within the target area, a first partial image obtained by imaging the imaging range with the first exposure time and a second partial image obtained by imaging the imaging range with the second exposure time; a correction step of correcting the position associated with the corresponding first partial image by using an overlapping area in which a second partial image of interest among the plurality of second partial images obtained within the target area overlaps with another second partial image adjacent to the second partial image of interest; a generating step of generating the combined image by combining the first partial images whose positions have been corrected; 13. An image processing method comprising:

15. An image processing method for generating a combined image corresponding to a target area of ​​an imaged object by correcting a position within the target area associated with a partial image obtained by imaging a part of the target area, comprising: an overlapping area determination step of determining an overlapping area in which a partial image of interest and another partial image adjacent to the partial image of interest overlap among a plurality of partial images obtained from within the target area when the image is captured under a predetermined imaging condition; a derivation step of deriving an evaluation value of the overlapping region determined in the overlapping region determination step; a determining step of determining an imaging condition based on the evaluation value so that the overlapping region includes predetermined information related to the imaging object; an acquisition step of acquiring the plurality of partial images obtained by imaging based on the determined imaging conditions; correcting the position associated with at least one of the plurality of partial images based on the overlapping region; a generating step of generating the concatenated image by concatenating the plurality of partial images whose positions have been corrected; having The evaluation value is determined based on the number of the plurality of partial images, and the larger the number, the lower the evaluation value is determined.

13. An image processing method comprising:

16. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 12.

Images