Image processing device

Abstract

An image processing device includes: a receiving unit that receives a designation of at least a portion of stereoscopic microscope image data; and an image generating unit that generates stereoscopic enlarged image data based on the designation. In the above-mentioned image processing device, the image generating unit can sequentially output a first enlarged image obtained based on at least a portion of the stereoscopic enlarged image data, and a second enlarged image obtained based on at least a portion of the stereoscopic enlarged image data whose at least a portion of the data is different from that of the first enlarged image.

Description

[0001] This invention application is a divisional application of an invention application with an international application date of November 24, 2016, an international application number of PCT / JP2016 / 084866, a national application number of 201680091040.9 entering the Chinese national phase, and an invention name of “Image processing device, microscope system, image processing method, and program”. Technical Field

[0002] The present invention relates to an image processing device, a microscope system, an image processing method, and a program. Background Art

[0003] There is a technology for enlarging and displaying a part of a CT image (for example, refer to Patent Document 1).

[0004] Patent Document 1: Japanese Patent Application Publication No. 2015-208539

[0005] However, it is difficult to conduct comparative observations of multiple microscope images. Summary of the invention

[0006] In a first aspect of the present invention, there is provided an image processing device including: a receiving unit that receives designation of position information of at least a portion of a first microscope image; and an image generating unit that outputs, based on the position information, a first enlarged image that enlarges a portion of the first microscope image, and a second enlarged image in a second microscope image that is associated with the first enlarged image.

[0007] In a second aspect of the present invention, there is provided an image processing method, comprising: receiving designation of position information of at least a portion of a first microscope image; and outputting, based on the position information, a first enlarged image obtained by enlarging a portion of the first microscope image, and a second enlarged image associated with the first enlarged image in a second microscope image.

[0008] In a third aspect of the present invention, there is provided a computer program that causes a computer to execute: receiving designation of position information of at least a portion of a first microscope image; and outputting, based on the position information, a first enlarged image obtained by enlarging a portion of the first microscope image, and a second enlarged image associated with the first enlarged image in a second microscope image.

[0009] The above summary of the invention does not list all the features of the present invention. Sub-combinations of these feature groups may also constitute inventions. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram of microscope system 100 .

[0011] Figure 2 An image displayed on the display device 170 is shown.

[0012] Figure 3 An image displayed on the display device 170 is shown.

[0013] Figure 4 2 is a diagram showing the concept of processing in the image processing apparatus 101 .

[0014] Figure 5 2 is a diagram showing the concept of processing in the image processing apparatus 101 .

[0015] Figure 6 2 is a diagram showing the concept of processing in the image processing apparatus 101 .

[0016] Figure 7 2 is a diagram showing the concept of processing in the image processing apparatus 101 .

[0017] Figure 8 An image displayed on the display device 170 is shown.

[0018] Fig. 9 An image displayed on the display device 170 is shown.

[0019] Fig.10 2 is a diagram showing the concept of processing in the image processing apparatus 101 .

[0020] Fig.11 2 is a diagram showing the concept of processing in the image processing apparatus 101 .

[0021] Fig.12 An image displayed on the display device 170 is shown.

[0022] Fig.13 : is a flowchart showing the processing procedure in the image processing apparatus 101 .

[0023] Fig.14 Yes Description Fig.13 FIG. 5 is a diagram of another example of step S102 .

[0024] Fig.15 Yes Description Fig.13 FIG. 4 is a diagram of yet another example of step S102 of FIG.

[0025] Fig.16 is a block diagram of microscope system 105 .

[0026] Fig.17 The diagrams illustrate other combinations of microscope images.

[0027] Fig.18 This is a diagram illustrating still another combination of microscope images.

[0028] Fig.19The state where the first enlarged image 411 and the second enlarged image 412 are displayed on the display device 170 is shown.

[0029] Fig. 20 A diagram for explaining another example of using time-lapse images.

[0030] Fig.21 1 is a block diagram of a microscope system 100 including another image processing device 102 .

[0031] Fig. 22 This is a diagram for explaining a case where the region to be enlarged in the first enlarged image 411 is designated as a display condition.

[0032] Fig.23 It is explained next Fig. 22 Processing diagram.

[0033] Fig.24 : is a flowchart showing the execution order of the processing in the image processing device 102.

[0034] Fig.25 This is a diagram for explaining a process of receiving scans on the first microscope image 401 and the second microscope image 402 .

[0035] Fig.26 This is a diagram for explaining a process of receiving scans on the first microscope image 401 and the second microscope image 402 .

[0036] Fig. 27 is a block diagram of another image processing device 103 .

[0037] Fig.28 It is a diagram for explaining the processing of the image processing device 103.

[0038] Fig.29 A state in which the superimposed image 414 is displayed is shown.

[0039] Fig.30 : is a flowchart showing the operation procedure of the image processing device 103 .

[0040] Fig.31 It is a diagram for explaining still another operation of the image processing device 101 . DETAILED DESCRIPTION

[0041] The present invention will be described below by way of embodiments of the invention. The embodiments described below are not intended to limit the technical solutions of the claims. In addition, the combination of features described in the embodiments is not necessarily all necessary for the solution means of the invention.

[0042] Figure 11 is a block diagram of a microscope system 100 including an image processing device 101. The microscope system 100 includes a microscope 700, an input device 130, the image processing device 101, and a display device 170.

[0043] When an instruction from the user is to be input to the image processing device 101, the input device 130 is operated. As the input device 130, a conventional general-purpose input device such as a pointing device such as a mouse, a keyboard, a touch panel, etc. can be used. The input device 130 is used to input to the image processing device 101, but can also be used as a device provided for operating the first microscope 110 and the second microscope 120.

[0044] The display device 170 is a liquid crystal display panel or the like, and displays the display image output by the image processing device 101 in a state that the user can visually recognize. The display device 170 may also be used as a device provided for the first microscope 110, the second microscope 120, etc. for the purpose of displaying images.

[0045] The image processing device 101 includes an image corresponding unit 140, a receiving unit 150, and an image generating unit 160. The image processing device 101 is composed of a general information processing device, which is controlled by a program for performing image processing described below. The image processing device 101 processes the microscope image data obtained from the microscope 700 and generates an image displayed on the display device 170.

[0046] The microscope 700 includes a first microscope 710 and a second microscope 720 for photographing a sample using different microscope methods. The first microscope 710 and the second microscope 720 share at least a portion of the optical system of the two microscopes (that is, at least a portion of the optical axes of the two microscopes are coaxial). In addition, regarding the first microscope 710 and the second microscope 720, the optical systems of the two microscopes may be independent optical systems.

[0047] The first microscope 710 uses SIM (Structured Illumination Microscopy) to photograph the sample. SIM uses structured illumination with a periodic illumination pattern to illuminate the sample, and generates a super-resolution microscope image showing the fine structure of the sample smaller than the wavelength of the illumination light based on the interference fringes generated by the sample image. The super-resolution microscope image is generated by reconstructing multiple images obtained by illuminating the sample by changing the direction and position of the illumination pattern of the structured illumination through calculation processing. In addition, for ease of explanation, the term "reconstruction" is only expressed as generation and / or photography below.

[0048] On the other hand, the second microscope 720 uses STORM (STochastic Optical Reconstruction Microscopy) to photograph the sample. STORM can reconstruct the fluorescence image (super-resolution microscope image) with a resolution higher than the wavelength of the illumination light by integrating the position information of the fluorescent pigments detected from multiple fluorescence images with high accuracy. In addition, for ease of explanation, the term "reconstruction" is only expressed as generation and / or photography below.

[0049] The microscope 700 uses the first microscope 710 to capture an observation area including at least a portion of the sample, and generates the first microscope image data 401. Furthermore, the microscope 700 uses the second microscope 720 to capture an observation area including at least a portion of the sample corresponding to the observation area captured by the first microscope 710, and generates the second microscope image data 402. In the present embodiment, the observation area of ​​the second microscope 720 corresponding to the observation area of ​​the first microscope 710 is specifically the photographing field of view of the second microscope 720 including at least a portion of the sample area included in the photographing field of view (i.e., the photographed area) of the first microscope 710. Therefore, the image of the sample in the image captured by the second microscope 720 includes at least a portion of the image of the sample in the image captured by the first microscope 710.

[0050] The first microscope image data 401 and the second microscope image data 402 are sent from the microscope 700 to the image processing device 101. Here, the first microscope image data 401 and the second microscope image data 402 are generated in a general format such as JPEG, BMP, etc. In addition, instead of these formats, they can be generated in a dedicated format that the image processing device 101 can perform image processing. In addition, before being displayed on the display device 170, the image captured by the first microscope 710 is processed as data (for example, processed as image data in the image processing device 101), and therefore, it is referred to as the first microscope image data 401 below, but since the information represented by the data is an image, for the convenience of explanation, the same figure mark is used to refer to the first microscope image 401. The second microscope image data 402 is also similar, and for the convenience of explanation, the same figure mark is used to refer to the second microscope image 402.

[0051] Figure 2 It is a diagram showing a display screen of the display device 170 . Figure 2 4 shows the first microscope image 401 and the second microscope image 402 at the time point when they are acquired by the image processing device 101. Figure 2 In the figures that follow, the X-axis and the Y-axis are appropriately labeled, but they are marks for illustrating directions and do not represent specific directions in actual space.

[0052] The first microscope image 401 shown on the left side of the figure includes a sample image 501. The second microscope image 402 shown on the right side of the figure includes a sample image 502 of the same sample as the first microscope image 710.

[0053] The first microscope image 401 and the second microscope image 402 are generated based on different microscope methods. Therefore, the magnification (sometimes referred to as image size), direction, and position of the two sample images 501 and 502 in the two microscope images are different. Figure 2 In the example of , the sample image 502 in the second microscope image 402 is smaller in size on the screen of the display device 170 than the sample image 501 in the first microscope image 401 .

[0054] In addition, while the longitudinal direction of the sample image 501 in the first microscope image 401 is substantially parallel to the X-axis, the sample image 502 in the second microscope image 402 is tilted by about 45° from the X-axis to the Y-axis (i.e., tilted to the upper right). Furthermore, while the sample image 501 is located near the center of the first microscope image 401, the sample image 502 is located near the left end of the second microscope image 402.

[0055] As described above, the size, inclination, and position of the sample images 501, 502 in each image are different in the first microscope image 401 and the second microscope image 402. Therefore, even if the first microscope image 401 and the second microscope image 402 are displayed side by side on the display device 170, it is difficult for the user to compare and observe the two images.

[0056] Figures 3 to 8: is a diagram for explaining the processing of the image correspondence unit 140. The image correspondence unit 140 performs processing for establishing a correspondence between the first microscope image 401 and the second microscope image 402, for example, as described below. The processing for establishing a correspondence can be expressed in another way as processing for establishing a correspondence between the two-dimensional coordinate positions of the pixels constituting the first microscope image 401, that is, the pixel positions, and the two-dimensional coordinate positions of the pixels constituting the second microscope image 402, that is, the pixel positions. The processing for establishing a correspondence can also be expressed in another way as processing for establishing a correspondence relationship such that when a pixel at a certain position of the first microscope image 401 is determined, a pixel at a certain position of the second microscope image 402 is uniquely determined for at least a part of the pixels of the first microscope image 401 and the second microscope image 402. For example, the processing includes processing for determining a pixel in the second microscope image 402 whose position on the microscope image is the same as the pixel position in the first microscope image 401. The pixel position is a position within the first microscope image 401 and the second microscope image 402, and therefore, it can also be called an image position. Furthermore, the association process can be expressed in another way as a process of associating the first microscope image data 401 with the second microscope image data 402 .

[0057] Figure 3 Shown in Figure 2 After the state is changed, the image is displayed on the display device 170. The receiving unit 150 receives the designation of a position in the first microscope image 401 from the user via the input device 130. The image corresponding unit 140 displays a mark 511 at the designated position. Figure 3 In the example of , a position within the sample image 501 is specified.

[0058] The receiving unit 150 also receives from the user a designation of a position in the second microscope image 402 that is considered to correspond to the position designated in the first microscope image 401. In this case, the user designates a position in the second microscope image 402 that is considered to be the same as the position of the sample designated in the first microscope image 401. The image corresponding unit 140 displays a mark 512 at the designated position. Figure 3 In the example of , a position within the sample image 502 is specified.

[0059] Similarly, the user receives inputs for designating a position different from the mark 511 in the first microscope image 401 and designating a position in the second microscope image 402 that is considered to correspond to the position designated in the first microscope image 401 (i.e., the position different from the mark 511) (in other words, the position different from the mark 512). Based on the input to the input device 130, the receiving unit 150 identifies the position in the first microscope image 401 and displays the mark 521, and identifies the position in the second microscope image 402 and displays the mark 522. When designating positions in the first microscope image 401 and the second microscope image 402, for example, the user clicks the positions in the first microscope image 401 and the second microscope image 402 with a mouse, thereby designating position information of the positions, and the receiving unit 150 acquires the position information.

[0060] Figures 4 to 7 Yes Description Figure 3 This is a conceptual diagram of the processing performed by the image matching unit 140 after the position is specified as described in FIG. Figures 4 to 8 , the first microscope image 401 and the second microscope image 402 are shown as "images", but this is for the purpose of explaining the image data processed inside the image processing device 101, and such "images" are not generated or displayed on the display device 170. In addition, the image processing device 101 processes the first microscope image 401 as data, so it is also referred to as the first microscope image data below, but because the information represented by the data is an image, the same reference numerals are sometimes used and illustrated as images. The same is true for the second microscope image 402.

[0061] Figure 4 1 is a diagram for explaining the processing of image data for aligning the directions of the sample image 501 in the first microscope image 401 and the sample image 502 in the second microscope image 402. First, the image corresponding unit 140 generates data of a virtual straight line 601 connecting the markers 511 and 521 in the first microscope image 401. Similarly, the image corresponding unit 140 generates data of a virtual straight line 602 connecting the markers 512 and 522 in the second microscope image 402.

[0062] Figure 5 This is to explain the use of the image matching unit 140 Figure 4 In order to make the inclination of the straight line 601 of the first microscope image 401 and the straight line 602 of the second microscope image 402 consistent, the image corresponding unit 140 processes the second microscope image data 402 so that Figure 5 As shown by the arrow 611 in the middle, the second microscope image 402 is completely Figure 5 The dashed box shows Figure 4The status of Figure 5 Rotate clockwise to Figure 5 direction shown.

[0063] Then, the image corresponding unit 140 calculates the distance D1 between the markers 511 and 521 of the first microscope image 401. Next, the image corresponding unit 140 calculates the distance D2 between the markers 512 and 522 of the second microscope image 402.

[0064] Figure 6 This is to explain the use of the image matching unit 140 Figure 5 The distance D2 and D1 have a magnitude relation of D2<D1. Therefore, when the sample image 502 (i.e., the second microscope image 402) is magnified by D1 / D2 times, Figure 6 As shown, the size (magnification) matches that of the sample image 501 (ie, the first microscope image 401 ). Therefore, the image matching unit 140 processes the second microscope image data 402 so that the second microscope image 402 is magnified by D1 / D2 times.

[0065] Figure 7 is a diagram for explaining the next stage of the processing performed by the image matching unit 140. The image matching unit 140 processes the second microscope image data 402 so that the position of the mark 512 of the second microscope image 402 coincides with the position of the mark 511 of the first microscope image 401. Here, aligning the position of the mark 512 with the position of the mark 511 means adjusting the position (coordinates) of the mark 512 in the XY coordinate system of the second microscope image 402 to be at the same position (coordinates) as the position (coordinates) of the mark 511 in the XY coordinate system of the first microscope image 401.

[0066] Figure 8 The first microscope image 401 and the second microscope image 402 whose sizes, inclinations and positions are aligned by the image matching unit 140 are shown arranged on the screen of the display device 170. Figure 8 In the example shown, the image generation unit 160 only displays the image processed into the image on the display device 170. Figure 7 As shown in FIG. 1 , the sample image 501 in the first microscope image 401 and the sample image 502 in the second microscope image 402 are made to have the same magnification, inclination, and position as the first microscope image 401. The sample image 501 in the first microscope image 401 and the sample image 502 in the second microscope image 402 have the same magnification, inclination, and position as the first microscope image 401. Therefore, the user can easily compare and observe the first microscope image 401 and the second microscope image 402.

[0067] In the above example, the image corresponding unit 140 processes the second microscope image data 402 by rotating, reducing, and moving the second microscope image 402, thereby making the size, direction, and position of the sample image 502 of the second microscope image 402 consistent with the size, direction, and position of the sample image 501 of the first microscope image 401. Alternatively, the image corresponding unit 140 may process the first microscope image data 401 by rotating, enlarging, and moving the first microscope image 401, thereby making the size, direction, and position of the sample image 501 of the first microscope image 401 consistent with the size, direction, and position of the sample image 502 of the second microscope image 402.

[0068] Fig. 9 The processing of the image generating unit 160 in FIG. 1 shows an image displayed on the display device 170. Fig. 9 In, as mentioned above Figure 8 As shown, the sample images of the first microscope image 401 and the second microscope image 402 are displayed in a state where the size, direction and position are consistent. The receiving unit 150 receives designation of position information of at least a portion of the first microscope image 401 using the input device 130 .

[0069] In this case, the image generation unit 160 receives input from the user through the input device 130, such as movement of a mouse. Fig. 9 As shown in FIG. 4 , a cursor 530 is displayed on the first microscope image 401. Furthermore, the receiving unit 150 receives the position of a point designated by the user by pressing the mouse button as position information (for example, the coordinates of the point). Fig. 9 In the example, the user specifies a point by pressing the mouse button at the position displayed by the cursor 530. The position of the accepted point is displayed on the screen of the display device 170 by an X-shaped mark 531. Alternatively, instead of pressing the mouse button, the acceptance unit 150 may accept the designation of a point by pressing a return key or touching the display device 170 when the display device 170 is a touch panel.

[0070] Fig.10is a diagram for explaining the processing of the image generating unit 160. The image generating unit 160 acquires the position information of the point in the first microscope image 401 received by the receiving unit 150 (for example, the coordinates of the point) from the receiving unit 150. Then, the image generating unit 160 determines the first enlarged area 631 corresponding to the position of the point. More specifically, based on the acquired position information of the point in the first microscope image 401, the image generating unit 160 determines a predetermined area including the point received by the receiving unit 150 as the area to be enlarged, that is, the first enlarged area 631. In the example shown in the figure, the first enlarged area 631 is an area surrounded by a circle with a predetermined radius 621 centered on the position of the mark 531.

[0071] In addition, the receiving unit 150 may accept the designation of the position of a region in the first microscope image 401 as the position information instead of accepting the designation of the position of a point in the first microscope image 401 as the position information by the input device 130. In this case, for example, the user drags and moves the mouse at a certain position in the first microscope image 401 to display a rectangle with the amount of movement as the diagonal, and then releases the drag, so that the receiving unit 150 accepts the rectangle. Then, the image generating unit 160 acquires information related to the position of the designated rectangle (for example, the coordinates of the four vertices of the rectangle) from the receiving unit 150, and determines a predetermined region including the designated rectangle (for example, a region with the diagonal of the designated rectangle as the diameter) as the first enlarged region 631 based on the information. In addition, the receiving unit 150 may accept the coordinates of the first microscope image 401 (first microscope image data 401) directly input by a keyboard or the like as the input device 130 as the position information. In this case, the coordinates of one point may be received by specifying a point on the first microscope image 401, or the coordinates of a plurality of points (for example, the coordinates of four vertices in the case of a rectangular area) may be received by specifying an area on the first microscope image 401. The image generation unit 160 acquires the acquired coordinates and specifies the first enlarged area 631 including the point or area corresponding to the coordinates. Furthermore, the acceptance unit 150 may also accept the designation of the position of the first enlarged area 631 itself as the position information. In this case, the mouse may be dragged and moved at a certain position in the first microscope image 401 to display a circle with the movement amount as the diameter or radius, and then the drag may be released, thereby accepting the circle as the first enlarged area 631. Furthermore, the image generation unit 160 acquires information related to the position of the designated circle (first enlarged area 631) (for example, the center coordinates and radius of the circle), and specifies the designated circle as the first enlarged area 631. In any of the above cases, the designation of the area is not limited to a rectangle or a circle, and may be designated in other shapes.

[0072] Moreover, if Fig.10As shown in FIG. 1 , the image generation unit 160 generates a first enlarged image 411 by enlarging the first enlarged region 631 in the first microscope image 401. The enlargement magnification of the first enlarged image 411 may be a magnification preset in the image processing device 101, or may be inquired of the user by the image processing device 101 after the position information is specified. It can be said that the first enlarged image 411 is an image displayed larger than the size of the first enlarged region 631 of the first microscope image 401 when displayed on the display device 170. In addition, when the first enlarged image 411 is generated, a part of the pixels forming the image does not exist in the first microscope image 401 and the image is missing, the image generation unit 160 may generate pixels for complementation to fill the missing pixels. In addition, the first enlarged region 631 is at least a part of the first microscope image 401, and therefore, the first enlarged image 411 can be said to be an image by enlarging at least a part of the first microscope image 401.

[0073] Fig.11 Yes Description Fig.10 The image generating unit 160 determines a point in the second microscope image 402 corresponding to the point in the first microscope image 401 whose designation is accepted by the accepting unit 150. Specifically, the image generating unit 160 determines a point in the second microscope image 402 whose position is identical to the position of the point in the first microscope image 401 whose designation is accepted. In other words, the image generating unit 160 determines a point in the second microscope image 402 whose position on the microscope image is identical to the position of the point in the first microscope image 401 whose designation is accepted. In other words, the image generating unit 160 determines a pixel in the second microscope image 402 whose position is identical to the pixel corresponding to the point in the first microscope image 401 whose designation is accepted.

[0074] Next, the image generator 160 determines a circular region with the point determined in the second microscope image 402 as the center, thereby determining a second enlarged region 632. The second enlarged region 632 has the same radius 622 as the first enlarged region 631. Furthermore, the image generator 160 enlarges the image of the second enlarged region 632 at the same magnification as the magnification of the first enlarged region 631, and generates a second enlarged image 412. In addition, the second enlarged region 632 determined by the image generator 160 may not be set with the point in the second microscope image 402 (the point determined by the image generator 160) at the center, and the position where the second enlarged region 632 is set may be spaced apart from the center of the point in the second microscope image 402 (the point determined by the image generator 160) within a range that does not affect the observation of the second enlarged image 412.

[0075] The second enlarged image 412 is an image obtained by enlarging a region including a point of the second microscope image 402 corresponding to a point designated to be enlarged in the first microscope image 401, and in this respect, it can be said that it is associated with the first enlarged image 411. However, the second enlarged image 412 generated by the image generating unit 160 in association with the first enlarged image 411 is not limited to the positional correspondence of each point, as long as there is a certain correlation. In addition, the second enlarged region 632 is at least a part of the second microscope image 402, and therefore, the second enlarged image 412 can be said to be an image obtained by enlarging at least a part of the second microscope image 402.

[0076] Fig.12 This is an example of displaying the first enlarged image 411 and the second enlarged image 412 on the display device 170. The image generation unit 160 displays the first enlarged image 411 and the first microscope image 401 on the display device 170 in an overlapping manner. The center of the first enlarged image 411 coincides with the position of the point in the first microscope image 401 where the designation is received by the receiving unit 150. Therefore, the first enlarged image 411 displays the periphery of the designated point in the first microscope image 401 as if observed by a magnifying glass, so that the user can observe an image of the region of interest to the user (i.e., the region including the designated position in the first microscope image 401) while overlooking the entire first microscope image 401, thereby enabling intuitive observation of the microscope image.

[0077] Furthermore, the image generator 160 displays the second enlarged image 412 in an overlapping manner with the second microscope image 402. The center of the second enlarged image 412 coincides with the position of the point identified by the image generator 160 in the second microscope image 402. Therefore, the second enlarged image 412 displays the periphery of the point identified in the second microscope image 402 as if it were magnified and observed with a magnifying glass, so that the user can perform intuitive observation of the microscope image while overlooking the entire second microscope image 402.

[0078] Moreover, the first microscope image 401 (sample image) and the second microscope image 402 (sample image) are displayed in advance with their sizes (magnification), directions, and positions being consistent. In addition, the first enlarged image 411 and the second enlarged image 412 are enlarged at the same magnification relative to the displayed first microscope image 401 and the second microscope image 402, respectively. Thus, even if the first microscope image 401 (sample image) and the second microscope image 402 (sample image) are captured at different magnifications and / or directions, it is possible to easily perform a comparative observation of the entire images of the first microscope image 401 and the second microscope image 402. Furthermore, it is possible to easily perform a comparative observation of the area of ​​interest to the user in the first microscope image 401 and the second microscope image 402.

[0079] Furthermore, when the user specifies the position of a point to be enlarged in the first microscope image 401, not only the first enlarged image 411 of the area including the position is displayed on the display device 170, but also the second enlarged image 412 of the area including the corresponding point is displayed on the display device 170. Therefore, the user can save the trouble of specifying the position of the point to be enlarged in the first microscope image 401 and the second microscope image 402, respectively, and can easily compare and observe the first enlarged image 411 and the second enlarged image 412.

[0080] In addition, when a part of the first enlarged image 411 exceeds the first microscope image 401 and overlaps with the second microscope image 402, the image generator 160 may hide the part of the second microscope image 402 overlapping with the first enlarged image 411 and display the first enlarged image 411. In addition, when a part of the second enlarged image 412 exceeds the second microscope image 402 and overlaps with the first microscope image 401, the image generator 160 may hide the part of the first microscope image 401 overlapping with the second enlarged image 412 and display the second enlarged image 412.

[0081] Furthermore, when a portion of the first enlarged region 631 and the second enlarged region 632 exceeds the first microscope image 401 and the second microscope image 402 , the image generator 160 may fill the regions corresponding to the exceeded enlarged regions in the first enlarged image 411 and the second enlarged image 412 with a solid color image.

[0082] Fig.13 2 is a flowchart showing a processing procedure in the image processing device 101. The image processing device 101 first acquires the first microscope image 401 (first microscope image data) and the second microscope image 402 (second microscope image data) from the microscope 700 (S101).

[0083] Next, the image corresponding unit 140 uses Figures 3 to 7 According to the method shown in FIG. 1 , the first microscope image 401 and the second microscope image 402 are matched with each other (S102). Based on the matching, the image matching unit 140 makes the size, direction and position of the sample images of the first microscope image 401 and the second microscope image 402 consistent, and Figure 8 The display device 170 is displayed as shown (S103).

[0084] Next, the image processing device 101 monitors the input from the input device 130 using the receiving unit 150, and waits for the input of the position information in the first microscope image 401 (S104: No). When the position information is input from the user via the input device 130 (S104: Yes), the image generating unit 160 determines the first enlarged area 631 of the first microscope image 401 (S105). The image generating unit 160 generates the first enlarged image 411 that enlarges the first enlarged area 631 in the first microscope image 401 (S109).

[0085] Next, the image generation unit 160 determines the second magnified area 632 of the second microscope image 402 (S106). Here, the size, direction and position of the sample images of the first microscope image 401 and the second microscope image 402 are the same, so if the second microscope image 401 and the second microscope image 402 are used, Fig.11 As described above, the position of the point in the second microscope image 402 that is the same as the position of the point in the first microscope image 401 input in step S104 is determined as the position of the center of the second enlarged region 632. Next, the image generating unit 160 generates the second enlarged image 412 that enlarges the second enlarged region 632 in the second microscope image 402 (S107). Then, as shown in FIG. Fig.12 As described above, the image generator 160 displays the first microscope image 401 and the second microscope image 402 and the first enlarged image 411 and the second enlarged image 412 side by side on the display device 170 ( S108 ).

[0086] In addition, a communication unit capable of acquiring a microscope image from outside the microscope system 100 may be provided in the image processing device 101. The communication unit may acquire the first microscope image 401 or the second microscope image 402 from an external database or the like via the Internet, a dedicated line, or the like, and use the acquired information in the image processing device 101.

[0087] exist Fig.13In step S102, instead of accepting the designation of the position used to establish the correspondence between the first microscope image 401 and the second microscope image 402 from the user, the image correspondence unit 140 can automatically extract the images of the plurality of markers included in the first microscope image 401 and the second microscope image 402 and use the position. In this case, the markers are introduced into the sample or the periphery of the sample before capturing the first microscope image 401 and the second microscope image 402. The introduced markers can use existing markers such as gold particles and / or fluorescent markers.

[0088] In this case, the image corresponding unit 140 adopts Figures 3 to 7 The method similar to the method described in , calculates the rotation amount, size ratio, and movement amount of the second microscope image 402 so that the positions of the images of the plurality of markers in the first microscope image 401 (position coordinates in the first microscope image 401) and the positions of the images of the plurality of markers in the second microscope image 402 (position coordinates in the second microscope image 402) are consistent. Thus, the image correspondence unit 140 can automatically correspond the first microscope image 401 and the second microscope image 402 without the user specifying the position for establishing correspondence between the first microscope image 401 and the second microscope image 402 in step S102, thereby reducing the time and effort for comparative observation of the first microscope image 401 and the second microscope image 402. In addition, the positions of the markers introduced into the sample and / or around the sample do not change when the first microscope image 401 and the second microscope image 402 are captured, so that accurate correspondence can be performed by using the images of the markers for correspondence. Alternatively, the user may specify a plurality of markers in the first microscope image 401 and a plurality of markers in the second microscope image 402 .

[0089] In addition, in step S102, the image correspondence unit 140 may also establish correspondence between the microscope images using morphological features shown in both the first microscope image 401 and the second microscope image 402. For example, the protrusion structure on the cell surface, the shape of the mitochondria in the cell, the trace of the cover glass, etc. may be used as markers. In addition, the image correspondence unit 140 may automatically extract the images of these markers to perform correspondence between the microscope images, or the image correspondence unit 140 may accept the user's designation of the positions of the images of these markers to perform correspondence between the microscope images.

[0090] Fig.14 Yes Description Fig.13 FIG. 1 is another example of step S102. Fig.14In FIG. 4 , the first microscope image 401 is an image obtained by SIM, and the second microscope image 402 is an image obtained by electron microscopy. Here, in both the microscope image of SIM and the microscope image of electron microscopy, sample images 451 and 452 are shown as they are. Fig.13 In step S102 , the image correspondence unit 140 performs pattern matching on the first microscope image 401 obtained by SIM and the second microscope image 402 obtained by electron microscopy, thereby establishing correspondence between the first microscope image 401 and the second microscope image 402 .

[0091] In this example, Fig.13 In step S102, instead of accepting the designation of multiple positions in the two microscope images from the user, the image matching unit 140 performs pattern matching between the first microscope image 401 and the second microscope image 402. In this case, the image matching unit 140 calculates the rotation amount, magnification, and position displacement of the second microscope image 402 relative to the first microscope image 401, which maximizes the similarity between the microscope images. Figure 8 In this way, the second microscope image 402 (second microscope image data) is image processed based on these calculated values, and the first microscope image 401 is associated with the second microscope image 402. Therefore, it is possible to easily associate microscope images with each other through pattern matching, thereby reducing the time spent on comparative observation of microscope images with each other. Here, pattern matching uses a known method such as a normalized correlation method or a geometric shape pattern matching method. In addition, pattern matching can also be called template matching.

[0092] In addition, the image matching unit 140 may not use the entire image in each microscope image for pattern matching. For example, the receiving unit 150 may receive from the user a designation of a partial area of ​​the first microscope image 401 as an area for pattern matching, and the image matching unit 140 may pattern match the designated area in the first microscope image 401 with the second microscope image 402. Alternatively, the receiving unit 150 may receive from the user a designation of a partial area of ​​the second microscope image 402, and the image matching unit 140 may pattern match the designated area in the first microscope image 401 with the designated area in the second microscope image 402. In the microscope image, by reducing the area of ​​the image used for pattern matching, the possibility that foreign matter (garbage, etc.) and / or texture (spots, etc.) that causes a decrease in the similarity of pattern matching is included in one image used for pattern matching (the area used for pattern matching) can be reduced, thereby improving the accuracy of the correspondence between the microscope images, and accurately comparing and observing the microscope images with each other.

[0093] Fig.15 Yes Description Fig.14 FIG. 1 is a diagram of a modified example of pattern matching. In this example, instead of Fig.14 The image corresponding unit 140 performs pattern matching between the first enlarged image 461 and the second microscope image 402 , as described above.

[0094] In this case, the image corresponding unit 140 Fig.13 In step S102, the image matching unit 140 receives input of a position to be enlarged for pattern matching from the user via the input device 130. Fig.15 As shown in FIG. 4 , pattern matching is performed using the pattern included in the first enlarged image 461 at the position. The pattern matching method is as follows: Fig.14 The method is the same as described in .

[0095] By establishing the correspondence between the first microscope image 401 and the second microscope image 402 based on the pattern included in the first magnified image 461, since the area of ​​the first magnified image 461 is smaller than that of the first microscope image 401, as described above, the possibility of garbage or the like being included in the first magnified image 461 is reduced, and thus the possibility of false detection in pattern matching can be reduced. In addition, in the above example, the pattern included in the first magnified image 461 is searched for the second microscope image 402. However, it is also possible to accept the designation of a position to be magnified in the second microscope image 402 and perform pattern matching on the pattern included in the magnified image (which is an image in which the area including the position is magnified) for the first microscope image 401.

[0096] In addition, as Fig.13 In another method of step S102, the image correspondence unit 140 may also obtain the setting conditions of the microscope 700 when the first microscope image 401 is taken and the setting conditions of the microscope 700 when the second microscope image 402 is taken, and establish correspondence between the first microscope image 401 and the second microscope image 402 based on them. Here, the setting conditions include the imaging magnification and / or the stage coordinates of the microscope. The image correspondence unit 140 may use both the imaging magnification and the stage coordinates to correspond the microscope images to each other, or may use either one to correspond.

[0097] The imaging magnification is the magnification of the image of the sample formed on the imaging surface of the imaging device (CCD and / or CMOS, etc.) of the microscope when the sample on the stage is photographed using each microscope. The image correspondence unit 140 can establish a correspondence between the magnification of the sample image 501 in the first microscope image 401 and the magnification of the sample image 502 in the second microscope image 402, based on the imaging magnification when the first microscope image 401 is photographed and the imaging magnification when the second microscope image 402 is photographed. Thus, for example, the size of the sample image 501 in the first microscope image 401 and the size of the sample image 502 in the second microscope image 402 when displayed on the display device 170 can be made consistent. Therefore, it is easy to compare and observe the two.

[0098] In addition, the stage coordinates are the coordinates of the stage on which the sample is placed, which are obtained when the sample is photographed using each microscope. The image correspondence unit 140 can establish correspondence between the position of the sample image 501 in the first microscope image 401 and the position of the sample image 502 in the second microscope image 402, based on the stage coordinates when the first microscope image 401 is photographed and the stage coordinates when the second microscope image 402 is photographed. Thus, for example, the position of the sample image 501 in the first microscope image 401 and the position of the sample image 502 in the second microscope image 402 can be made consistent when displayed on the display device 170. Therefore, it is easy to compare and observe the two.

[0099] In addition, Fig.13 In step S103, the size, direction and position of the image are displayed in accordance with the correspondence between the first microscope image 401 and the second microscope image 402. However, the step of displaying the size, direction and position of the image in accordance with the first microscope image 401 and the second microscope image 402 may be omitted. In this case, in step S102, Figures 3 to 7 The method of establishing correspondence between microscope images described in Fig.14 and Fig.15 In either of the method of establishing correspondence between microscope images described in the foregoing description and the method of establishing correspondence using the setting conditions of the microscope when the above-mentioned microscope images are captured, the image generating unit 160 determines the second magnified area in step S106 in accordance with the determination of the first magnified area in step S105, based on the size, direction and position of each image determined when establishing correspondence between the first microscope image 401 and the second microscope image 402.

[0100] Fig.16 is a block diagram of a microscope system 105 using other microscope images, Fig.171 is a diagram for explaining a microscope image used in the microscope system 105. In this example, microscope images obtained by photographing a sample at different focal planes are used. In addition, the microscope 700 of the microscope system 105 is single, which is different from the microscope system 100. In the microscope system 105, the same reference numerals are attached to the same components as those of the microscope system 100, and the description thereof is omitted.

[0101] In microscopy methods such as STED and confocal laser scanning microscopy (CLSM), different heights of the sample are used as focal planes, and images are taken at each focal plane, thereby generating a group of microscope images that are continuous in the height direction (in other words, have three-dimensional position information). Fig.13 In step S101 , the image matching unit 140 acquires a group of microscope images captured at a plurality of focal planes from a single microscope 700 .

[0102] In this case, the image corresponding unit 140 generates stereoscopic microscope image data (stereoscopic microscope image) having three-dimensional brightness distribution information by synthesizing the microscope images (brightness distribution) of the microscope image group. The receiving unit 150 receives the designation of two plane microscope image data (plane microscope images) cut out in mutually parallel planes from the stereoscopic microscope image data (stereoscopic microscope image) as the first microscope image 401 and the second microscope image 402. In this case, for example, the receiving unit 150 receives the designation of the first microscope image 401 and the second microscope image 402 by the user using a mouse in a drawn image based on the stereoscopic microscope image data displayed on the display device 170.

[0103] The image corresponding unit 140 is still Fig.13 In step S102, the first microscope image 401 is associated with the second microscope image 402. In this case, the image association unit 140 may use Figures 3 to 7 The method for establishing correspondence between microscope images described in Fig.14 and Fig.15 The method for establishing correspondence between microscopes described in the above and the method for establishing correspondence using the setting information of the microscope when taking microscope images. Thus, for example, even in the case where the sample moves due to the time difference generated when taking pictures of different focal planes, the correspondence can be properly established. Therefore, even if the two-dimensional microscope images are cut out from different planes, the size, direction and position of the sample images of the two microscope images are consistent, so they can be easily compared and observed.

[0104] Then proceed Fig.13Specifically, the image corresponding unit 140 displays the first microscope image 401 and the second microscope image 402 on the display device 170 ( Fig.13 When the position information from the user is input via the input device 130 ( Fig.13 S104: Yes), the image generation unit 160 determines the first enlarged region 631 ( Fig.13 ), generating a first enlarged image 411 ( Fig.13 S109).

[0105] Next, the image generation unit 160 determines the second enlarged region 632 ( Figure 3 The second enlarged image 412 (S106) is generated by enlarging the second enlarged area 632 in the second microscope image 402. Fig.13 S107). Further, if using Fig.12 As described above, the image generating unit 160 displays the first microscope image 401 and the second microscope image 402 and the first enlarged image 411 and the second enlarged image 412 in a row on the display device 170 ( Fig.13 S108).

[0106] In this embodiment, stereoscopic microscope image data is generated from a group of microscope images acquired from a single microscope 700. Instead of this method, microscope images may be acquired from a plurality of microscopes, and the plurality of microscope images may be combined to generate stereoscopic microscope image data.

[0107] In addition, as the first microscope image 401 and the second microscope image 402, the directions of the respective planes of the stereo microscope image can be freely specified by the user via the input device 130. For example, as the first microscope image 401 and the second microscope image 402, planes parallel to each other in any direction can be cut out from the stereo microscope image data (stereo microscope image). In addition, as long as it does not hinder comparative observation, the first microscope image 401 and the second microscope image 402 can also be cut out from planes that are not parallel to each other in the stereo microscope image data (stereo microscope image).

[0108] In addition, in microscopy methods such as STED and confocal laser scanning microscopy (CLSM), the same objective lens is often used for detecting the fluorescence intensity on each focal plane. When a planar microscope image generated based on a microscope image group on multiple focal planes using the same objective lens is used as the first microscope image 401 and the second microscope image 402, Fig.13 In step S102, Figures 3 to 7 In the method for establishing correspondence between the first microscope image 401 and the second microscope image 402 shown in FIG. 1 , the image correspondence unit 140 may omit the processing of making the sizes of the images consistent. In addition, sometimes, without moving the sample, the first microscope image 401 and the second microscope image 402 have the same direction, size, and position. In this case, the processing may be omitted. Fig.13 In step S102, Figures 3 to 7 The steps shown are to establish correspondence between the first microscope image 401 and the second microscope image 402.

[0109] Fig.18 It's an example Fig.16 FIG. 1 is a diagram of a combination of other microscope images used in the microscope system 105 . Fig.18 The microscope image group 450 includes a plurality of microscope images (so-called image group acquired by time-lapse photography) obtained by photographing the observation area (photographing field of view of the microscope) at arbitrary elapsed time intervals. Fig.13 In step S101 , the image matching unit 140 acquires a group of microscope images captured by a single microscope 700 with a time-lapse. Fig.18 In FIG. 1 , image groups of different time ranges among the microscope image group 450 are displayed in two columns.

[0110] exist Fig.13 In step S102, the image matching unit 140 receives the designation of the first microscope image 401 and the second microscope image 402 acquired from different time ranges in the microscope image group 450 via the input device 130. That is, in this example, microscope images captured at different times are designated as the first microscope image 401 and the second microscope image 402. The image matching unit 140 further selects the microscope image 401 and the second microscope image 402 according to the use Figures 3 to 7 The method described above establishes a correspondence between the first microscope image 401 and the second microscope image 402. Thus, even in the case where the sample moves due to the time difference between the images, the correspondence can be appropriately established. Figures 3 to 7 The method of the points described in Fig.14 and Fig.15 A method of pattern matching described in , and a method of setting conditions of a microscope when using a microscope image.

[0111] like Fig.19 As shown, the image corresponding unit 140 is Fig.13 In step S103, the first microscope image 401 and the second microscope image 402 are displayed on the display device 170. The receiving unit 150 Fig.13In step S104 , the designation of position information to be enlarged in the first microscope image 401 is received via the input device 130 .

[0112] Then, proceed Fig.13 Specifically, when the position information is input from the user via the input device 130, the image generation unit 160 determines the first magnified area 631 ( Fig.13 ), generating a first enlarged image 411 ( Fig.13 S109).

[0113] In addition, the image generation unit 160 determines the size, direction, and position of each image when the first microscope image 401 and the second microscope image 402 are associated with each other, such as using Fig.10 and Fig.11 As described above, the position of the point in the second microscope image 402 corresponding to the position of the point in the first microscope image 401 specified by the user is determined ( Fig.13 The image generating unit 160 further generates a second enlarged image 412 ( S106 ) by enlarging the region including the identified point of the second microscope image 402 . Fig.13 S107).

[0114] Moreover, if you use Fig.12 As described above, the image generating unit 160 displays the first microscope image 401 and the second microscope image 402 and the first enlarged image 411 and the second enlarged image 412 in parallel on the display device 170 ( Fig.13 S108).

[0115] The image generator 160 causes the first magnified image 411 to overlap with the first microscope image 401, and causes the second magnified image 412 to overlap with the second microscope image 402, and displays them on the display device 170 ( Fig.13 Thus, the first magnified image 411 and the second magnified image 412, which are partial magnified images of the first microscope image 401 and the second microscope image 402 selected from the microscope image group 450 generated by time-lapse photography and shot at different times, can be displayed on the display device 170 for comparative observation. Therefore, comparative observation of time-lapse images can be easily performed.

[0116] Furthermore, the image generation unit 160 displays the first microscope image 401 and the second microscope image 402 on the display device 170 over time. Here, the first magnified image 411 is switched corresponding to the first microscope image 401 displayed sequentially. In addition, the second magnified image 412 is also switched along with the second microscope image 402 displayed sequentially. Thus, the temporal changes of the phenomena occurring in the sample can be efficiently compared and observed.

[0117] In addition, the image generation unit 160 receives switching of the first enlarged image 411 and the second enlarged image 412 displayed on the display device 170 from the user through the operation of the slide bar 551. Specifically, the image generation unit 160 Fig.18 The length of the time-lapse shooting represented by the image group arranged in two columns (if the time lengths represented by the two columns of image groups are the same, it is that time, and if the time lengths are different, it is one of them) as shown in Fig.19 The image generation unit 160 displays the time corresponding to the position of the slider 551 on the horizontal bar 552 as shown in FIG. 5. The horizontal bar 552 shows the movable area of ​​the slider 551 received from the user. When the operation of the slider 551 is received via the input device 130, the image generation unit 160 determines the time corresponding to the position according to the position of the slider 551 on the horizontal bar 552. The image generation unit 160 determines the first microscope image 401 and the second microscope image 402 corresponding to the determined time from each image group, and displays them together with the determined first enlarged image 411 of the first microscope image 401 and the second enlarged image 412 of the second microscope image 402. Thus, the user can observe the time changes of the first enlarged image 411 and the second enlarged image 412 by operating the slider 551, and the efficiency of comparative observation can be further improved.

[0118] In addition, in the present embodiment, a microscope image captured by time-lapse photography using a single microscope is used. However, instead of this, microscope images may be acquired from a plurality of microscopes and these plurality of microscope images may be arranged in the order in which they were captured to serve as a microscope image captured by time-lapse photography.

[0119] In addition, in the above example, as the microscope image group 450, the first microscope image 401 and the second microscope image 402 are obtained from images of a sample observed in sequence. However, the first microscope image 401 and the second microscope image 402 may not be included in a series of microscope image groups captured over time. For example, for the same sample, the first microscope image 401 may be selected from a microscope image group captured over time after reagent A is added to the sample, and then the second microscope image 402 may be selected from a microscope image group captured over time after reagent B is added to the sample. In addition, the first microscope image 401 and the second microscope image 402 may be selected from each microscope image group obtained by adding the same reagent to different samples and capturing each sample over time.

[0120] in addition, Fig.19 The slider 551 can be used in the following situations, that is, if the Fig.17 As described above, the user specifies the first microscope image 401 and the second microscope image 402 from the images obtained by cutting out a plurality of planes in the stereo microscope image data, or the image generation unit 160 displays these microscope images on the display device 170. In this case, the receiving unit 150 stores the position of the slider 551 in correspondence with the position of the cutting plane of the stereo microscope image data. When the receiving unit 150 receives the input of the position of the slider 551 from the user via the input device 130, the receiving unit 150 determines the position of the cutting plane of the stereo microscope image data based on the received position of the slider 551 and outputs it to the image generation unit 160, whereby the image generation unit 160 displays the plane microscope image at the plane on the display device 170.

[0121] Fig. 20 is a diagram for explaining another example of using time-lapse images. Fig.18 and Fig.19 In the time-lapse image described above, there is a case where the position and / or area that the user is focusing on moves. Therefore, in this example, it is detected to which position the position that was designated for zooming in the first microscope image 401 has moved in the first microscope image 401 and the second microscope image 402 that were taken later. In this case, it is possible to use Fig.14 and Fig.15 The pattern matching described in .

[0122] In this case, Fig.13 In step S101, the image association unit 140 acquires a group of microscope images captured with a time-lapse image using a single microscope 700. The image association unit 140 receives, via the input device 130, designations of the first microscope image 401 and the second microscope image 402 acquired from different time ranges in the group of microscope images.

[0123] Furthermore, in this embodiment, the image processing device 101 replaces Fig.13 The steps S102 to S107 of the present invention are operated as follows. The receiving unit 150 receives the designation of the position of the first microscope image 401 that the user wishes to focus on from the input device 130, and the image corresponding unit 140 determines the region 453 including the position. The image corresponding unit 140 uses the region 453 to perform pattern matching with a plurality of first microscope images 401 at a later time than the first microscope image 401 currently displayed. Thus, the image corresponding unit 140 enlarges the regions detected as having a high degree of similarity with the region 453 in the plurality of first microscope images 401 at a later time than the first microscope image 401, thereby generating a plurality of first enlarged images 411.

[0124] The image corresponding unit 140 also uses the region 453 to perform pattern matching with the second microscope image 402 currently displayed and a plurality of second microscope images 402 at subsequent times, and establishes correspondence between the first microscope image 401 and the second microscope image 402. Thus, the image corresponding unit 140 enlarges the region 454 detected as having a high degree of similarity with the region 453 in each of the plurality of second microscope images 402, thereby generating a plurality of second enlarged images 412.

[0125] like Fig. 20 As shown, the image generator 160 displays the first enlarged image 411 and the second enlarged image on the display device 170 over time. Thus, even if the position and / or area of ​​interest to the user moves in the time-lapse image, the enlarged images of the position and / or area can be easily compared.

[0126] In addition, you can also Fig.19 and Fig. 20 In the example described in , the second microscope image 402 and the second enlarged image 412 are only displayed over time without changing the first microscope image 401 and the first enlarged image 411 .

[0127] Fig.21 1 is a block diagram of a microscope system 100 including another image processing device 102. The image processing device 102 and the image processing device 102 are similar to each other except for the parts described below. Figure 1 The image processing device 101 shown has the same configuration. Elements common to the image processing device 101 are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0128] The image processing device 102 is different from the image processing device 101 in that it includes a processing receiving unit 151 that receives display conditions inputted from the input device 130 in addition to the receiving unit 150 that receives the position designated by the input device 130. The display conditions are conditions for changing the display on the display device 170 by performing image processing on the temporarily generated first enlarged image 411. The display conditions include the enlarged area in the first enlarged image 411, the image magnification, the image brightness, the image contrast, the image rotation, and the image inversion.

[0129] Fig. 22 5 is a diagram for explaining a case where the enlarged area in the first enlarged image 411 is designated as a display condition. The user drags the outline of the first enlarged image 411 displayed on the display device 170 with the cursor 530 to designate a size after the change. Fig. 22 As shown by the middle arrow 641, by dragging the cursor 530 radially outward of the first enlarged image 411, the user can instruct to enlarge the display area of ​​the first enlarged image 411. In addition, the user can determine the size of the first enlarged image 411 after image processing by releasing the drag at a desired position.

[0130] Fig.23 It is explained next Fig. 22 FIG. 1 is a diagram of a process of processing. The image generating unit 160 enlarges the initial first enlarged image 411 to the range of the first enlarged image 421 corresponding to the enlarged display area based on the size of the first enlarged image 411 received by the process receiving unit 151. In conjunction with this, the image processing device 102 enlarges the initial second enlarged image 412 to the same size as the enlarged first enlarged image 421, and generates an enlarged second enlarged image 422. In this case, the size, direction and position of the first microscope image 401 and the second microscope image 402 are consistent, so the enlargement process is performed on the first microscope image 401 (sample image) and the second microscope image 402 (sample image) at the same position, with the same magnification and in the same area. As a result, the enlarged first enlarged image 421 and the enlarged second enlarged image 422 are arranged side by side.

[0131] Fig.24 FIG. 1 is a flowchart showing the execution order of the processing in the image processing device 102. Fig.24 In the order of Fig.13 The same, therefore the description is omitted.

[0132] With the first microscope image 401 , the second microscope image 402 , the first enlarged image 411 , and the second enlarged image 412 displayed on the display device 170 , the process accepting unit 151 accepts input of the presence or absence of display conditions in this state ( S121 ).

[0133] When the processing receiving unit 151 receives the display condition (S121: Yes), the image generating unit 160 performs image processing according to the display condition on the first enlarged image 411 (S122). Furthermore, the image generating unit 160 also reflects the image processing performed on the first enlarged image 411 on the second enlarged image 412 generated corresponding to the first enlarged image 411 and performs the image processing (S123).

[0134] Next, the image generating unit 160 generates the first enlarged image 421 and the second enlarged image 422 after image processing (S124), and displays them on the display device 170 (S125). Thus, the processing executed when the processing receiving unit 151 receives the display condition is completed.

[0135] In this way, the image processing device 102 can further perform image processing on the first enlarged image 411 and the second enlarged image 412 according to the input display conditions after the processing by the image generating unit 160 is temporarily completed. As a result, the first enlarged image 411 and the second enlarged image 412 after the processing displayed on the display device 170 can be observed more easily, and the efficiency of comparative observation can be further improved.

[0136] Furthermore, when the display conditions for the first enlarged image 411 are input, the image processing device 102 also performs the same image processing on the second enlarged image 412 as on the first enlarged image 411. This saves the trouble of specifying the display conditions for the first enlarged image 411 and the second enlarged image 412, and enables efficient comparison and observation of the two.

[0137] In addition, the designation method when designating the display conditions in the above step S121 is not limited to the method of operating the cursor movement on the first enlarged image 411. For example, a menu listing the types of image processing and parameter values ​​as options may be displayed, and the content indicated by the menu may be selected. Moreover, when, for example, the display area of ​​the enlarged image is designated, the actual size of the sample may be designated by a numerical value, or may be designated by the number of pixels on the display device 170, the magnification ratio relative to the size of the original area, and the like. Furthermore, a plurality of first enlarged images 411 of different sizes may be displayed, and a selection may be made therefrom.

[0138] exist Fig.24 In step S121, the processing receiving unit 151 may receive Fig.234. The display conditions other than the area enlarged in the first enlarged image 411 described in the above are used as display conditions. For example, when the processing receiving unit 151 receives the rotation of the first enlarged image 411 as a display condition, the image generating unit 160 rotates the first enlarged image 411 and overlaps it with the first microscope image 401. Similarly, the image generating unit 160 also rotates the second enlarged image 412 and overlaps it with the second microscope image 402. In this case, the size, direction and position of the first microscope image 401 and the second microscope image 402 are the same, so the image generating unit 160 only needs to rotate the second enlarged image 412 by the same angle as the first enlarged image 411.

[0139] exist Fig.24 In step S121, the processing accepting unit 151 may also accept the image magnification, image brightness, image contrast, and image inversion as other display conditions. When accepting any display condition, the image generating unit 160 generates the first enlarged image 411 and the second enlarged image 412 after image processing according to the display condition, overlaps the first microscope image 401 and the second microscope image 402, and displays them on the display device 170. Thus, the trouble of specifying the processing of the first enlarged image 411 and the second enlarged image 412 separately can be omitted, so that the two can be compared and observed efficiently.

[0140] Can replace Fig.24 In step S121, the receiving unit 150 receives an instruction to move the first enlarged image 411. In this case, when the user drags and moves the first enlarged image 411 on the display device 170 via the input device 130, the receiving unit 150 receives the movement and outputs the movement direction and movement amount to the image generating unit 160. The image generating unit 160 moves the first enlarged region 631 in the first microscope image 401 in the movement direction and movement amount received by the receiving unit 150, and generates the first enlarged image 411 that enlarges the moved first enlarged region 631. In addition, the image generating unit 160 moves the second enlarged region 632 in the second microscope image 402 in the movement direction and movement amount received by the receiving unit 150, and generates the second enlarged image 412 that enlarges the moved second enlarged region 632. The image generating unit 160 displays the first enlarged image 411 and the second enlarged image 412 on the display device 170. Thus, the second enlarged image 412 moves in synchronization with the movement of the first enlarged image 411 and is displayed on the display device 170. In response to the instruction to move the first enlarged image 411, the second enlarged image 412 also moves together, so that the user can move the first enlarged image 411 and the second enlarged image 412 to a position where the user can easily observe them, and can easily compare and observe the two.

[0141] Can replace Fig.24 In step S121, the processing receiving unit 151 receives a change in the position at which the first enlarged image 411 is displayed. In this case, when the user drags the first enlarged image 411 on the display device 170 via the input device 130, the processing receiving unit 151 receives the movement and outputs the movement direction and movement amount to the image generating unit 160. The image generating unit 160 displays the first enlarged image 411 at a position after the movement is performed in accordance with the movement direction and movement amount received by the processing receiving unit 151, and also displays the second enlarged image 412 at a position after the movement is performed in the same movement direction and movement amount. Thus, the second enlarged image 412 is also moved synchronously with the movement of the first enlarged image 411 and is displayed on the display device 170. According to the instruction to move the first enlarged image 411, the second enlarged image 412 is also moved together, so that the user can move the first enlarged image 411 and the second enlarged image 412 to a position where the user can easily observe them, and can easily compare and observe the two.

[0142] Fig.25 and Fig.26 Is to replace the description in Fig.24 In step S121 , the display condition is accepted, and the scanning process on the first microscope image 401 and the second microscope image 402 is accepted.

[0143] In this example, the user can instruct the scanning mode to the processing receiving unit 151 when comparing and observing the first enlarged image 411 and the second enlarged image 412. In the scanning mode, the scanning start position, scanning direction, and scanning end position are set by the system default, but can also be specified by the user via the input device 130.

[0144] In the scan mode, the image generator 160 scans the first microscope image 401 from the starting point of the first microscope image 401 along the scan direction, and sequentially generates and displays the first enlarged image 411 including the scan position.

[0145] Fig.25 The first enlarged image 411 and the second enlarged image at the starting point are shown. Fig.26 The first enlarged image 411 and the second enlarged image in the middle of scanning are shown. The image generator 160 sequentially generates and displays the second enlarged images 412 of the positions corresponding to the second microscope image 402 in synchronization with the scanning of the first microscope image 401 .

[0146] Therefore, the first magnified image 411 and the second magnified image 412 repeatedly move horizontally in the same direction in synchronization with each other and gradually descend on the first microscope image 401 and the second microscope image 402. By adopting such a scanning pattern, the image processing device 102 can improve the efficiency of complete comparative observation of the first microscope image 401 and the second microscope image 402.

[0147] use Fig.25 and Fig.26 The scanning mode described above can also be used to compare and observe samples in a plurality of wells arranged two-dimensionally on a well plate. The amount, concentration, type, and time elapsed from drug injection of the drugs added to the samples are different for the plurality of wells.

[0148] In this case, first, the image matching unit 140 obtains a first microscope image 401 obtained by photographing a sample of a specific hole, and a plurality of second microscope images 402 obtained by photographing samples of a plurality of holes different from the specific hole. The receiving unit 150 receives the designation of the position of the point to be enlarged in the first microscope image 401, and the image generating unit 160 generates a first enlarged image 411 in which the area including the point of the first microscope image 401 is enlarged. Fig.10 As described in the example of , the receiving unit 150 may receive the designation of the position of the region of the first microscope image 401 instead of the designation of the position of the point of the first microscope image 401 .

[0149] The image generating unit 160 determines a point in each of the plurality of second microscope images 402 corresponding to a point in the first microscope image 401 whose designation is accepted by the accepting unit 150. Specifically, the image generating unit 160 determines a point in the second microscope image 402 whose position (position coordinates in the first microscope image 401) is the same as the position of the point in the first microscope image 401 whose designation is accepted. The image generating unit 160 also generates a plurality of second enlarged images 412 in which the regions including the point in the plurality of second microscope images 402 are enlarged, and sequentially displays the plurality of second enlarged images 412 on the display device 170 together with the first enlarged image 411. Instead of determining a point in the second microscope image 402 whose position is the same as the position of the point in the first microscope image 401 whose designation is accepted, the image generating unit 160 may be used for each of the first microscope image 401 and the plurality of second microscope images 402. Fig.14 and Fig.15 The pattern matching described above specifies the position at which the second enlarged image 412 is generated in each of the plurality of second microscope images 402 .

[0150] When the plurality of second enlarged images 412 are displayed sequentially, the image generating unit 160 and Fig.25 and Fig.26 Similarly to the method described in , based on the input from the user or the system default setting, the position to start scanning, the scanning direction, and the position to end scanning are determined, and the second enlarged images 412 of the wells corresponding to the positions on the scan are sequentially displayed. This makes it easy to compare and observe samples of a plurality of wells that are different in the amount, concentration, type, and time elapsed from the injection of the drug.

[0151] Fig. 27 FIG. 1 is a block diagram of another image processing device 103. The image processing device 103 has the same Figure 1 Therefore, the same reference numerals are given to the common components and duplicate descriptions are omitted.

[0152] The image processing device 103 further includes a transparency accepting unit 152 and an overlay accepting unit 153 that accept instructions from the input device 130. The transparency accepting unit 152 accepts a transparency instruction from a user input through the input device 130. The overlay accepting unit 153 accepts an image overlay instruction from a user input through the input device 130.

[0153] Fig.28 1 is a diagram for explaining the processing of the image processing device 103. Before the transparency instruction is input, the image corresponding part 140 is displayed on the display device 170. Fig.28 In FIG. 4 , a first enlarged image 411 and a second enlarged image 412 are displayed overlapping the first microscope image 401 and the second microscope image 402 , respectively. Fig.28 Also shown is a button 491 that supports transparency.

[0154] When receiving the instruction of transparency through the key 491, the image generating unit 160 which has received the instruction through the transparency receiving unit 152 selects which of the first enlarged image 411 and the second enlarged image 412 to be made transparent using the cursor 530 operated by a pointing device such as a mouse. In the example shown in the figure, the first enlarged image 411 is selected and made transparent.

[0155] The transparency mentioned here means to make the image semi-transparent to the extent that the content of the first enlarged image 411 can be clearly seen and the underlying layer can be seen through, rather than to make the first enlarged image 411 disappear. In addition, the image processing device 103 displays a slide bar 551 for adjusting the transmittance on the display device 170, so that the user can set the transmittance.

[0156] When the user inputs an instruction to overlay via the input device 130, the overlay accepting unit 153 accepts the instruction. When the overlay accepting unit 153 accepts the instruction to overlay, the image generating unit 160 performs overlay processing to generate an overlay image 414 in which the first enlarged image 411 and the second enlarged image 412 are overlaid.

[0157] When the user places the cursor 530 on the first enlarged image 411 and drags it to move the first enlarged image 411 and overlap the second enlarged image 412, the overlap receiving unit 153 determines that there is an overlap instruction. Instead of moving the first enlarged image 411 by dragging it, the overlap instruction may be received by a button, a menu item, etc. When the overlap instruction is received by a button, a menu item, etc., the overlap image may be displayed at the position of one of the first enlarged image 411 and the second enlarged image 412, or may be displayed at a position different from the position of either the first enlarged image 411 or the second enlarged image 412.

[0158] Fig.29 4 shows a state where the superimposed image 414 is displayed. The image generating unit 160 performs superimposition processing in which the first enlarged image 411 that has been made transparent is used as the upper layer and the second enlarged image 412 that has not been made transparent is used as the background in the region where the first enlarged image 411 and the second enlarged image 412 overlap. As a result, the different parts between the first enlarged image 411 and the second enlarged image 412 can be easily grasped. Therefore, the efficiency of comparative observation of the first microscope image 401 and the second microscope image 402 can be improved. In addition, the image generating unit 160 can also interchange the enlarged image that has been made transparent and the enlarged image that is the background according to the user's instruction.

[0159] Furthermore, when generating the superimposed image 414, image processing may be performed so that one of the first enlarged image 411 and the second enlarged image 412 has a different color and / or brightness from the other. For example, when image processing is performed so that one of the first enlarged image 411 and the second enlarged image 412 has a different color from the other, image processing may be performed so that one of the first enlarged image 411 and the second enlarged image 412 has a different tone from the other. This makes it easier to compare the first enlarged image 411 and the second enlarged image.

[0160] Fig.30 1 is a flowchart showing the operation sequence of the image processing device 103. Fig.30 In the order of Fig.13 The same, therefore the description is omitted.

[0161] In a state where the first microscope image 401 , the second microscope image 402 , the first enlarged image 411 , and the second enlarged image 412 are displayed on the display device 170 , the transparency accepting unit 152 accepts a transparency instruction in this state ( S131 ).

[0162] Next, when the transparency accepting unit 152 accepts a transparency instruction (S131: Yes), the image generating unit 160 performs transparency on, for example, the first enlarged image 411 (S132). Next, the overlap accepting unit 153 monitors whether there is an overlap instruction and waits (S133: No).

[0163] When the superposition receiving unit 153 receives the instruction to superimpose (S133: Yes), the image generating unit 160 superimposes the first enlarged image 411 and the second enlarged image 412 to generate the superimposed image 414 (S134). The image generating unit 160 superimposes the generated superimposed image 414 on the second microscope image 402 and displays the superimposed image 414 on the display device 170 (S135). Thus, the image processing performed by the image processing device 103 ends.

[0164] Fig.31 1 is a diagram for explaining other operations of the image processing device 101. Fig.31 In, with Fig.12 Similarly, a first microscope image 401 , a second microscope image 402 , a first enlarged image 411 , and a second enlarged image 412 are displayed.

[0165] exist Fig.31 , repeat each time a new designation is received from the user about the location to be zoomed in Fig.13 In steps S105 to S108, the first enlarged image 411 and the second enlarged image 412 corresponding to the position are displayed on the display device 170. In this way, the user sequentially observes the first enlarged image 411 and the second enlarged image 412, but sometimes the user wants to observe the first enlarged image 411 and the second enlarged image 412 of a specific position that has been observed before. Therefore, each time the receiving unit 150 Fig.13 When the designation of the position is received in step S104, steps S105 to S108 are executed, and the image generation unit 160 displays the first enlarged image 411 and the second enlarged image 412 on the display device 170, and displays the mark 475 at each position of the first microscope image 401. In addition, the first enlarged image 411 and the second enlarged image 412 are stored in the image processing device 101 corresponding to each position.

[0166] Next, when the user specifies any of the marks 475, the image generator 160 displays the first enlarged image 411 and the second enlarged image 412 stored at the position corresponding to the mark 475 on the display device 170. In addition, based on the instruction from the user, the image generator 160 may automatically sequentially display the first enlarged image 411 and the second enlarged image 412 for a plurality of marks 475. In addition, in the second microscope image 402, the mark 475 may be displayed at a point corresponding to the point specified by the user in the first microscope image 401, or may be displayed on both the first microscope image 401 and the second microscope image 402.

[0167] According to this embodiment, the mark 475 can be retained during the operation of comparing the enlarged images, so when you want to re-observe the previously enlarged position later, you can easily compare and observe the enlarged images. Therefore, it is easy to compare and observe the first microscope image 401 and the second microscope image 402.

[0168] In any of the above-mentioned embodiments and modifications, the image generator 160 may display the first enlarged image 411 separately from the first microscope image 401 on the display device 170, or may display the second enlarged image 412 separately from the second microscope image 402 on the display device 170. Furthermore, when the first enlarged image 411 overlaps with the first microscope image 401, the image generator 160 may overlap the first enlarged image 411 at a position different from the position designated to be enlarged in the first microscope image 401. Similarly, when the second enlarged image 412 overlaps with the second microscope image 402, the image generator 160 may overlap the second enlarged image 412 at a position different from the position designated to be enlarged in the second microscope image 402.

[0169] In addition, in any of the above-mentioned embodiments and modifications, the first microscope image 401 and the second microscope image 402 obtained by SIM, STORM and electron microscopy can be replaced by images obtained by other microscope methods as the first microscope image 401 and the second microscope image 402. For example, microscope images generated by a stereo fluorescence microscope, an epifluorescence microscope, a confocal microscope, a STED (Stimulated Emission Depletion) microscope, a PALM (PhotoActivated Localization Microscopy), an Expansion Microscopy, an atomic force microscope, etc. can also be used. In addition, images processed under different conditions using the same microscope method can also be used as the first microscope image 401 and the second microscope image. In this case, super-resolution images with different conditions set when reconstructed in SIM can be used as the first microscope image and the second microscope image.

[0170] When using an image obtained by any microscopy method, the image matching unit 140 may also use Figures 3 to 7 The method for establishing correspondence between microscope images described in Fig.14 and Fig.15 The first microscope image 401 is associated with the second microscope image 402 by using any of the methods for associating microscope images with each other described in , and the method for setting conditions of the microscope when capturing microscope images. Furthermore, in any of the embodiments and modifications, the first microscope image 401 and the second microscope image 402 may be images obtained by capturing samples of the same type but of different individuals, and / or by capturing samples of different types.

[0171] In any of the above-mentioned embodiments and modifications, at least a portion of the first microscope image 401 designated by the receiving unit 150 and at least a portion of the second microscope image 402 corresponding to the portion may not be located at the same position. Specifically, the positions of the point in the first microscope image 401 designated by the receiving unit 150 and the point in the second microscope image 402 determined by the image generating unit 160 on the microscope image may be different. For example, when comparing and observing a first enlarged image obtained by enlarging a region including the point in the first microscope image 401 designated by the receiving unit 150 and a second enlarged image obtained by enlarging a region including the point in the second microscope image 402 determined by the image generating unit 160, the positions of the point in the first microscope image 401 designated by the receiving unit 150 and the point in the second microscope image 402 determined by the image generating unit 160 on the microscope image may be separated within a range that does not affect the comparison.

[0172] In any of the above-mentioned embodiments and variations, the image generation unit 160 may perform brightness conversion so that the brightness values ​​of the first enlarged image 411 and the second enlarged image 412 are distributed within a specific range. The image generation unit 160 preferably changes the brightness of each of the first enlarged image 411 and the second enlarged image 412 so that the distribution of the brightness of each is appropriate. For example, when the brightness values ​​in the first enlarged image 411 (in other words, the brightness values ​​of each pixel of the first enlarged image 411) are concentrated on the lower (darker) side of the 256 gray scales, the image generation unit 160 performs brightness conversion so that the distribution is extended to the entire 256 gray scales. Specifically, the brightness of each pixel is converted so that the brightness value of the pixel with a high brightness value among the brightness values ​​of each pixel of the first enlarged image 411 becomes higher (brighter). On the other hand, for example, when the brightness value of the second enlarged image 412 (in other words, the brightness value of each pixel of the second enlarged image 412) is concentrated on the higher (brighter) side of the 256 gray scales, the image generation unit 160 transforms the brightness of each pixel so that the distribution is extended to the entire 256 gray scales and the brightness value of the pixel with a low brightness value among the pixels of the second enlarged image 412 becomes lower (darker). As a result, it is possible to express the brightness difference that has not been observed in the first enlarged image 411 and the second enlarged image 412, and to observe the image of the fine structure of the sample. By transforming the brightness in a manner appropriate for the distribution of the brightness in each of the first enlarged image 411 and the second enlarged image 412, it is easy to compare and observe the two. In this case, a known LUT (Look-Up-Table) or the like that describes the relationship between the brightness value before the transformation and the brightness value after the transformation is used.

[0173] In addition, the second enlarged image 412 can be said to be associated with the first enlarged image 411 in that the position of the second microscope image 402 corresponding to the position designated for enlargement in the first microscope image 401 and at which the first enlarged image 411 was generated is enlarged. However, the second enlarged image 412 generated by the image generating unit 160 in association with the first enlarged image 411 is not limited to the image corresponding to each position, and it is sufficient that there is some kind of association.

[0174] The present invention has been described above using the embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that the implementation of such changes or improvements is also included in the technical scope of the present invention.

[0175] It should be noted that the execution order of each process such as the actions, sequence, steps, and stages of the devices, systems, programs, and methods shown in the claims, specifications, and drawings can be implemented in any order unless it is specifically stated that "before...", "before...", etc., and the output of the pre-processing is used for post-processing. Even if the action flow in the claims, specifications, and drawings is described using "first", "next", etc. for convenience, it does not mean that it must be implemented in this order.

[0176] Description of Reference Numerals

[0177] 100, 105 microscope system, 101, 102, 103 image processing device, 110 first microscope, 120 second microscope, 130 input device, 140 image corresponding unit, 150 receiving unit, 151 processing receiving unit, 152 transparent receiving unit, 153 superposition receiving unit, 160 image generating unit, 170 display device, 401 first microscope image, 402 second microscope image, 414 superposition image, 411, 421, 461 first enlarged image, 412, 422 second enlarged image, 450 microscope image group, 451, 452 sample images, 453, 454 area, 475, 511, 512, 521, 522 marks, 491 button, 501, 502 sample images, 530 cursor, 531 mark, 551 slider, 552 horizontal bar, 601, 602 straight line, 611, 641 arrows, 621, 622 radius, 631, 632 enlarged area, 700 microscope, 710 first microscope, 720 second microscope.

Claims

1. An image processing device, characterized in that: include: a receiving unit that receives a designation of a position of at least a portion of the first microscope image; and an image generating unit that generates image data for displaying the first microscope image, the second microscope image, the first enlarged image, and the second enlarged image on a display device, wherein the second microscope image is an image obtained by photographing the same sample as the first microscope image and establishing a correspondence with the first microscope image, the first enlarged image is an image obtained by enlarging a part of the first microscope image based on the designation of the position, and the second enlarged image is an image obtained by enlarging a region of the second microscope image that is associated with the region of the first microscope image and includes a position corresponding to the position, The receiving unit receives an instruction inputted from the user to the input unit to designate a part of the first enlarged image and move the first enlarged image, The image generating unit generates image data for causing the display device to display a first enlarged image in accordance with the movement of the region on the first microscope image based on the movement instruction, and causing the display device to display a second enlarged image obtained by enlarging a region in the second microscope image that is associated with the moved region.

2. The image processing device according to claim 1, wherein: The first microscope image and the second microscope image include images obtained by different microscopic methods.

3. The image processing device according to claim 1, wherein: The first microscope image and the second microscope image include images of focal planes different from each other.

4. The image processing device according to claim 1, wherein: The first microscope image and the second microscope image include images including the observation object processed under different processing conditions.

5. The image processing device according to claim 1, wherein: The first microscope image and the second microscope image both include images obtained using a structured illumination microscope, The first microscope image and the second microscope image include images generated under conditions different from each other.

6. The image processing device according to claim 1, wherein: The image generating unit generates image data for displaying the first enlarged image and the first microscope image in a superimposed manner and the second enlarged image and the second microscope image in a superimposed manner on the display device.

7. The image processing device according to claim 1, wherein: The image generating unit generates image data for displaying on the display device such that the first microscope image and the second microscope image are adjacent to each other and the first enlarged image and the second enlarged image are adjacent to each other.

8. The image processing device according to claim 1, wherein: The system further includes an image corresponding unit that corresponds the first microscope image to the second microscope image.

9. The image processing device according to claim 8, wherein: The receiving unit receives a plurality of positions in the first microscope image and a plurality of positions in the second microscope image for establishing correspondence, The image association unit associates the plurality of positions in the first microscope image with the plurality of positions in the second microscope image.

10. The image processing device according to claim 1, wherein: The size, inclination, and position of the sample image in the second microscope image correspond to the size, inclination, and position of the sample image in the first microscope image.

11. A microscope system, characterized in that: have: The image processing device according to any one of claims 1 to 9; a microscope that outputs a microscope image to the image processing device; and A display device displays an image generated based on the image data output from the image processing device.

Images