Microscope system, electronic equipment capable of controlling the microscope, control method and program

Abstract

To provide a technique that can quickly and accurately perform superimposed display on an observation image through a microscope.SOLUTION: The present disclosure relates to a microscope system having change instruction means that instructs a change in an observation range. The microscope system has: acquisition means that acquires an image obtained by photographing the observation range including an observation object; inference means that infers the type of the observation object included in the image; creation means that creates a superimposed image obtained by superimposing an inference result from the inference means on a position of the observation object of the image; and correction amount determination means that determines a correction amount related to a difference between an amount of change instructions issued by the change instruction means and an actual movement amount of the observation range. When creating a superimposed image based on a second image photographed with the changed observation range after acquisition of a first image, the creation means creates a superimposed image in which a position is corrected according to the amount of the change instructions and the correction amount for an observation object included in the first image out of observation objects included in the second image, and that is obtained by superimposing, on the second image, the inference result of the inference means with respect to the first image.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a microscope system, an electronic device capable of controlling a microscope, a control method, and a program.

Background Art

[0002] Conventionally, a system is known in which a digital camera is attached to a microscope, a specimen image magnified through the microscope is imaged, and the captured image is displayed on a monitor.

[0003] When a high-magnification lens is used to photograph a fine observation target such as a cell, the observation range becomes narrow. For this reason, a technique is known in which the observation range is sequentially moved by driving and controlling a stage, an image is photographed, and the photographed images are combined to generate an observation image of the entire sample (Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By applying an inference technique using, for example, deep learning (deep learning algorithm) to an observation image and superimposing information such as a classification result of an observation target on the area of the observation target, the convenience when using the observation image can be improved. However, generally, since the inference processing using deep learning has a large amount of calculation, if it is applied to the technique of Patent Document 1, it will take time until images are sequentially captured, combined, and the inference result is superimposed.

[0006] In addition, in an actual system, since there is a system control error, when information on the inference result is superimposed and displayed at the movement position calculated from the design value after the stage movement, the information on the inference result may be displayed shifted.

[0007] This invention has been made in view of the above problems, and its objective is to realize a technology that can perform high-speed and accurate superimposition on an image observed through a microscope. [Means for solving the problem]

[0008] To solve this problem, for example, the microscope system of the present invention has the following configuration. That is, a microscope system having a change instruction means for instructing a change in the observation range, comprising: an acquisition means for acquiring an image of the observation range including an object to be observed; an inference means for inferring the type of object to be observed included in the image; a generation means for generating a superimposed image by superimposing the inference result by the inference means onto the position of the object to be observed in the image; and a correction amount determination means for determining a correction amount related to the difference between the amount of change instruction by the change instruction means and the actual amount of movement of the observation range, wherein the generation means, when generating a superimposed image based on a second image taken after changing the observation range following the acquisition of a first image, generates a superimposed image by superimposing the inference result of the inference means on the second image, with the position of objects to be observed included in the second image that are also included in the first image corrected according to the amount of change instruction and the correction amount. [Effects of the Invention]

[0009] According to the present invention, it becomes possible to superimpose an image onto an image observed through a microscope at high speed and with accuracy. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the appearance of the microscope system according to the present invention. [Figure 2] This figure shows an example of the functional configuration of a microscope system according to the embodiment. [Figure 3] This diagram shows the four classifications of bacterial species based on Gram staining. [Figure 4] This flowchart shows the operation of the AI ​​inference frame overlay display process. [Figure 5] It is a diagram showing an example of displaying an image with an AI inference frame superimposed. [Figure 6] It is a flowchart showing the operation of the superimposed display process when changing the observation range. [Figure 7] It is a diagram showing an example of display when changing the observation range. [Figure 8] It is a diagram showing an example of the superimposed display in the case without correction processing. [Figure 9] It is a diagram showing an example of the superimposed display in the case with correction processing. [Figure 10] It is a flowchart showing the operation of the correction amount determination process. [Figure 11] It is a diagram showing an example of a table for storing AI inference results. [Figure 12] It is a diagram for explaining the template matching process. [Figure 13] It is a diagram showing the state where the difference is minimized in the template matching process. [Figure 14] It is a diagram showing an example of a user interface for setting the search range and setting values. [Figure 15] It is a diagram showing an example of the search range in the case of a set value of 3σ for the search range. [Figure 16] It is a diagram for explaining the template usage frame selection process. [Figure 17] It is a diagram for explaining the history of the observation range when performing the image synthesis process. [Figure 18] It is a diagram showing an example of the recorded content of the observation range history table. [Figure 19] It is a diagram showing an example of the image synthesis result.

Mode for Carrying Out the Invention

[0011] (Embodiment 1) Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicate explanations are omitted.

[0012] <Overview of Microscope System> FIG. 1 schematically shows an example of the appearance of a microscope system 10 according to this embodiment.

[0013] 101 is a microscope device. The microscope device 101 has an optical path capable of electronically photographing a microscopic enlarged image during observation with a digital camera or the like.

[0014] 102 is a personal computer (PC) as an example of an electronic device. The PC 102 acquires (receives) an image photographed by the microscope device 101 via an external IF such as USB, or transmits a control signal for controlling the position of the XY stage to the XY stage 106. Further, the PC 10 executes inference processing (also referred to as AI inference) using a machine learning model on the image photographed by the microscope device 101. The machine learning model may be, for example, a neural network formed by deep learning (deep learning algorithm). In the following description, an example using a personal computer (PC) will be described, but this embodiment is not limited to a PC and is also applicable to other electronic devices capable of controlling the position of a microscope slide and acquiring a photographed image via wired or wireless. These devices may include, for example, tablet terminals, smartphones, medical devices, and the like.

[0015] 103 is a display. The display 103 is an example of a display unit 217 described later, and displays a superimposed image in which the result of AI inference is superimposed on a microscopic image photographed by the digital camera of the microscope device 101.

[0016] 104 is an input device for the user 105 to input instructions to change the observation range of the microscope device 101 (observation range change instruction). In this embodiment, the case in which a joystick 104 is used as an example of an input device is described, but the input device may be any other input device that can input direction, and it may also be configured to input from a mouse or keyboard.

[0017] 106 is an XY stage. The XY stage 106 operates, for example, a linear actuator 110 in the X direction and a linear actuator 109 in the Y direction according to control signals from the PC 102, to move the observation range 108 of the sample 107.

[0018] In such a microscope system 10, the user 105 can operate the joystick 104 to move the position of the observation range 108 on the sample 107 to a desired position and observe various parts of the sample 107 while looking at the display 103.

[0019] <Example of a microscope and PC functional configuration> Next, with reference to Figure 2, an example of the functional configuration of the microscope system according to this embodiment will be described. Note that one or more of the functional blocks shown in Figure 2 may be implemented by hardware such as an ASIC or a programmable logic array (PLA), or by a programmable processor such as a CPU or MPU executing software. Alternatively, they may be implemented by a combination of software and hardware.

[0020] First, let's describe an example of the functional configuration of the microscope apparatus 101. 201 is the light source, which focuses light emitted from a halogen lamp or tungsten lamp (not shown) using a condenser lens (not shown) and irradiates the sample 107.

[0021] 202 is a stage, a platform for stably setting up the sample 107, the XY stage 106, etc. In this embodiment, the XY stage 106 is set up and operated on the stage 202 which is pre-installed in the microscope apparatus 101. The sample 107 is placed on the XY stage 106, and when the user 105 operates the joystick 104, the stage control unit 220 transmits a control signal 218 corresponding to the amount of user operation to the XY stage 106. The observation range 108 is changed as the XY stage 106 operates according to the control signal 218.

[0022] 203 is the objective lens. By rotating the revolving nosepiece 204, multiple objective lenses with different magnifications can be switched and used. 205 is the imaging lens. The imaging lens 205 has optical properties such that the light of the transmitted image that has passed through the sample 107 is focused onto the imaging plane of the image sensor of the digital camera 206.

[0023] The digital camera 206 includes an image sensor in which multiple pixels having photoelectric conversion elements are arranged in two dimensions. The image sensor may be a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, etc. For example, the image sensor photoelectrically converts the image of a sample formed on the imaging surface of the image sensor at each pixel, and then converts it from analog to digital using an A / D conversion circuit to output pixel-level image data.

[0024] The microscope device captures images of the sample 107 as digital images (observation images) using a digital camera 206, and transmits the video signal 219 of the observation image to the PC 102 via a communication means such as a USB cable.

[0025] Next, we will describe an example of the functional configuration of PC102. 212 is the video input unit. The video input unit 212 transmits the video signal 219 received from the digital camera 206 to various parts within PC102 via the internal bus 211.

[0026] 213 is an AI inference unit. The AI ​​inference unit 213 performs inference using a machine learning model on the microscope image (observation image) obtained from the video input unit 212. Specifically, in this embodiment, deep learning is used to infer bacterial species from the observation image of a sample 107 that has been Gram-stained on a glass slide coated with sputum. Gram staining involves staining the sample with a predetermined dye solution. When the stained sample is observed with an optical microscope, Gram-positive bacteria appear purple and Gram-negative bacteria appear red, as will be described later. Note that the sample to be observed using Gram staining is not limited to sputum; blood, urine, etc., can also be used.

[0027] In this embodiment, we will explain, as an example, the classification of bacterial species in a Gram-stained image into the following four classifications (GNR, GNC, GPR, GPC) using a general object detection technique based on deep learning. In Gram staining, bacteria can be classified into four categories, "GNR," "GNC," "GPR," and "GPC," according to the color and shape of the bacteria after staining. Here, GNR (Gram negative rods) is an abbreviation for Gram-negative bacilli, GNC (Gram negative cocci) is an abbreviation for Gram-negative cocci, GPR (Gram positive rods) is an abbreviation for Gram-positive bacilli, and GPC (Gram positive cocci) is an abbreviation for Gram-positive cocci. Each type of bacterium can be classified into four categories, as shown in Figure 3, according to its color and shape after Gram staining. Gram-negative bacteria are 301 and 311 in Figure 3, and turn red after staining. Gram-positive bacteria are 321 and 331, and turn purple after staining. Bacilli are 301 and 321, and have an elongated shape. Cocci are 311 and 333 and are spherical in shape.

[0028] In this embodiment, deep learning training is performed in advance using images of Gram-stained samples. The trained model generated as a result of the training is stored in the secondary memory unit 215. The data of the trained model includes, for example, the weighting parameters of the neural network. The AI ​​inference unit 213 reads this trained model from the secondary memory unit 215 and performs classification processing of the fungal species. In this embodiment, a network model called object detection is used as an example of a deep learning algorithm. Any object detection network model can be used, but for example, SSD: Single Shot MultiBox Detector can be used as a model that performs well in detecting small objects such as fungi.

[0029] 214 is a superimposed image generation unit. The superimposed image generation unit 214 generates an image (superimposed image) by superimposing the AI ​​inference frame inferred by the AI ​​inference unit 213 onto the image data obtained from the video input unit 212. The AI ​​inference frame, as will be described later, is information regarding the position and size of the AI ​​inference frame surrounding the bacteria detected by the inference.

[0030] 215 is a secondary storage unit composed of a non-volatile storage medium such as an SSD (Solid State Drive). The secondary storage unit 215 can store the operating system, various programs according to this embodiment, data of the trained model used in the AI ​​inference unit described above, and configuration data for the PC 102.

[0031] 216 is a primary storage unit composed of a volatile storage medium such as DRAM. Each part of the PC102 uses the primary storage unit 216 to temporarily store programs and various data as working memory. 217 is a display unit. The display unit 217 is, for example, an LCD monitor, which is configured either as an integral part of the PC102 or as a separate unit.

[0032] 224 is the control unit. The control unit 224 includes one or more processors, such as a CPU (Central Processing Unit), and controls the operation of each part of the PC 102 by loading programs stored in the secondary storage unit 215 into the primary storage unit 216 and executing them. The control unit 224 may further include a GPU (Graphics Processing Unit) for high-speed execution of processing using machine learning models in the AI ​​inference unit 213.

[0033] The stage control unit 220, together with the user input unit 223, constitutes the observation range change instruction unit 221. The observation range change instruction unit 221 functions as an observation range change instruction means in the microscope system 10. The joystick 104 described above constitutes the user input unit 223, which receives user input operations. The stage control unit 220 transmits a control signal 218 to the microscope device 101 to control the position of the XY stage 106.

[0034] 222 is a correction amount determination unit. When the observation range is moved according to the instruction of the observation range change instruction unit 221, the correction amount determination unit 222 determines the difference between the instruction of the control signal 218 and the actual amount of movement of the XY stage 106 as the correction amount.

[0035] <Operation of the AI ​​inference frame overlay processing during initial display> Next, referring to Figure 4, the operation of the AI ​​inference frame superimposition process, which superimposes the AI ​​inference results onto the microscope image as the initial display, will be explained. This process can be achieved by the control unit 224 of the PC102 deploying and executing the program stored in the secondary storage unit 215 to the primary storage unit 216, and controlling each part of the PC102.

[0036] In S401, the video input unit 212 receives an observation image from the microscope device 101 and stores the received observation image in the primary storage unit 216. In S402, the AI ​​inference unit 213 reads the observation image from the primary storage unit 216, performs AI inference processing, and stores the AI ​​inference result (also called the inference result) in the primary storage unit 216.

[0037] Here, the AI ​​inference results stored in the primary memory unit 216 include, for example, the following information for each detected bacterium. Note that the following "types" refer to the types of objects being observed, and in this embodiment, they refer to bacterial species. • Type: Classification information indicating which type (bacterial species) it is (GNR / GNC / GPR / GPC). • AI inference frame: Information regarding the position and size of the AI ​​inference frame surrounding the bacteria within the image. • AI Inference Score: The probability that the inference of the fungal species is correct (a value between 0 and 1)

[0038] In S403, the superimposed image generation unit 214 reads the observed image and the AI ​​inference result stored in the primary storage unit 216 and generates an image in which the AI ​​inference result is superimposed on the observed image. The superimposed image generation unit 214 writes the generated image as the AI ​​inference result superimposed image to the primary storage unit 216.

[0039] In S404, the display unit 217 acquires the AI ​​inference result superimposed image stored in the primary storage unit 216 and displays it on the display panel. Alternatively, the superimposed image generation unit 214 may control the display unit 217 to display the AI ​​inference result superimposed image. Once the display unit 217 displays the AI ​​inference result superimposed image, this process ends.

[0040] Figure 5 illustrates an example of the display on the display unit 217 after the AI ​​inference frame superimposed as the initial display. 501 in Figure 5 indicates the current display frame shown on the display unit 217. That is, the area within 501 is displayed on the display panel of the display unit 217. Areas other than 501 are illustrated for illustrative purposes and represent the state of an actual sample outside the observation range. In this embodiment, the display frame 501 corresponds, for example, to an observation range of 30 μm in the X direction and 20 μm in the Y direction.

[0041] 502 is an AI inference frame indicating the presence of GNR, and the inference result shows that GNR is present at the location of the frame, roughly the size of the frame. For example, the bacterial species of GNR is enclosed in a red frame. 503 is an AI inference frame indicating the presence of GPC.

[0042] Image 504 is a neutrophil, a non-bacterial object. Typically, many non-bacterial objects are present in observed images. Neutrophils are abundant when bacteria causing the subject's symptoms are present, and a high number of neutrophils indicates that the sample being observed is suitable for identifying the causative organism.

[0043] <Operation of the AI ​​inference frame overlay process when changing the observation range> Next, referring to Figure 6, the operation of the microscope system 10 when the user 105 operates the joystick 104 to instruct a change in the observation range from the initial display state shown in Figure 5 will be described. In this process as well, unless otherwise specified, the control unit 224 of the PC 102 can be deployed and executed by the primary storage unit 216 from the secondary storage unit 215, thereby controlling each part of the PC 102.

[0044] In S601, PC102 accepts user input via user input unit 223. For example, let's consider the case where user 105 operates joystick 104. Here, let's assume user 105 wants to move the observation range to the right and tilts joystick 104 to the right. The amount of operation at this time is, for example, tilting it to the right for 1 second.

[0045] In S602, the observation range change instruction unit 221 generates a control signal 218 according to the amount of user input and transmits it to the XY stage 106. Here, the stage control unit 220 controls the stage by 10 μm in the same direction for each second the joystick 104 is tilted. For example, if the joystick is tilted to the right for 1 second, the stage control unit 220 transmits a control signal 218 to the XY stage 106 only once, instructing it to move 10 μm in the X direction.

[0046] In step S603, the correction amount determination unit 222 performs the correction amount determination process. In this microscope system 10, similar to a typical XY stage control system, the actual stage movement amount when the stage is controlled includes an error with respect to the movement amount specified by the control signal. The correction amount determination unit 222 determines the correction amount related to this error. Details of the correction amount determination process will be described separately.

[0047] In S604, the superimposed image generation unit 214 performs superimposition processing. At this stage, for the area where AI inference was performed before movement (i.e., the area where inference was performed in S403), the superimposed image generation unit 214 generates a superimposed image in which the AI ​​inference frame is displayed at a position moved by the amount of movement due to the control signal and the correction amount determined in S603. In S605, the display unit 217 displays the image with the inferred AI inference frame superimposed on it on the display panel.

[0048] In S606, the AI ​​inference unit 213 performs AI inference processing on the newly displayed area. That is, the AI ​​inference unit 213 performs AI inference processing (inference of the type of observed object) on the observed object in the area included in the image after movement that was not included in the image before movement. AI inference, which is an inference process using machine learning models, generally requires a large amount of computation. Therefore, by performing AI inference only on the newly displayed area, the AI ​​inference unit 213 can reduce the inference time and display the AI ​​inference results on the observed image with better responsiveness.

[0049] In S607, the superimposed image generation unit 214 performs superimposition processing on the newly displayed area using the AI ​​inference frame newly generated by the AI ​​inference unit 213. Since correction processing by the correction amount determination unit 222 is not necessary for the new area, the AI ​​inference frame obtained as an inference result from the AI ​​inference unit 213 is superimposed directly onto the observed image.

[0050] On the other hand, in the superposition process for the displayed area in S604, the AI ​​inference frame is displayed at coordinates obtained by adding the amount of movement equal to the control signal (10 μm in the X direction) and the correction amount determined by the correction amount determination unit 222 to the coordinates of the AI ​​inference frame before movement. In this way, it becomes possible to display an AI inference frame (using existing inference results) that takes system system errors into account.

[0051] In S608, the display unit 217 displays the superimposed image generated by the superimposed image generation unit 214 on the display panel. The control unit 224 terminates this process when the display unit 217 displays the superimposed image.

[0052] In this embodiment, the process shown in Figure 6 was explained using the example of displaying the AI ​​inference frame that has already been inferred in S605, assuming that the AI ​​inference process for the new area takes time. However, the above process is not limited to this example, and if the AI ​​inference time does not cause any usability problems, the processes in S604 and S605 may be skipped. That is, the AI ​​inferred area and the new area to be AI inferred may be superimposed simultaneously in S607 and displayed once in S608.

[0053] The following describes the appearance of the observation image and the AI ​​inference frame when the AI ​​inference frame is superimposed (processing shown in Figure 6) while changing the observation range, referring to Figure 7.

[0054] Figure 7 shows that 501 is the observation range when changing the observation range shown in Figure 5. 701 is the observation range indicated by the control signal 218 from the observation range change instruction unit 221. If there are no system system errors in this microscope system 10, the observation range after moving the observation range will be as shown in 701. On the other hand, 702 is the actual observation range after the move. Figure 7 shows that an error 703 (2 μm) in the X direction and an error 704 (1 μm) in the Y direction occur with respect to the observation range 701.

[0055] Figure 8 shows an example of the superimposed display when no correction is performed using the correction amount determined by the correction amount determination unit 222. The observation ranges 701 and 702 are as described above. Frames 21 to 26 are AI-inferred frames for bacteria 1 to 6 before movement, respectively. As shown in the figure, if errors are not considered, the AI-inferred frames 21 to 26 will be displayed shifted 10 μm to the left in the X direction as relative coordinates within the observation range. In other words, the AI-inferred frames 21 to 26 will be displayed shifted by 2 μm in the X direction and 1 μm in the Y direction relative to bacteria 1 to 6, respectively. Frames 27 to 31 are AI-inferred frames resulting from new AI inference processing performed by the AI ​​inference unit 213 for bacteria that have newly appeared due to the movement of the observation range. Since these frames do not reuse the inferred frames, they are correctly superimposed around the bacteria.

[0056] Figure 9 shows an example of the display of an overlaid image when the correction amount is determined by the correction amount determination unit 222 and the overlaid image generation unit 214 performs the overlaid processing of the AI ​​inference frame taking the correction amount into consideration. As shown in Figure 9, by correcting the position of the AI ​​inference frame by the determined correction amount, the corresponding AI inference frames 21 to 26 are displayed in the correct position around bacteria 1 to 6.

[0057] <Operation of the correction amount determination process> Next, with reference to Figure 10, the correction amount determination process in S603 by the correction amount determination unit 222 will be described.

[0058] In S1001, the correction amount determination unit 222 selects an AI inference frame to be used for template matching processing. As an example of how to select an AI inference frame, for example, the correction amount determination unit 222 determines the AI ​​inference score of each AI inference frame and selects the AI ​​inference frame with the highest AI inference score (i.e., the AI ​​inference frame with the highest probability of being correct in its inference of the fungal species). However, the method of selecting an AI inference frame is not limited to this example, and for example, the complexity (activity) of the images within the AI ​​inference frame may be added to the evaluation value for selection. In addition, other selection methods that enable good template matching can be used, such as selecting from AI inference frames whose AI inference score is above a predetermined threshold, taking into account the state of the surrounding observed objects.

[0059] Furthermore, the template image used for template matching may not only be the image of the AI ​​inference frame itself, but may also be an image that includes the AI ​​inference frame and a few pixels around it, or an image that includes other areas determined based on the positional information of the AI ​​inference frame.

[0060] Furthermore, in this embodiment, for simplicity, the template image is selected from among the AI ​​inference frames located in the overlapping area of ​​the source and destination areas assumed from the control signal 218. However, since there is actually a movement error in the XY stage, it is also possible to set an overlapping area considering a predetermined error and select the AI ​​inference frames located within that area as candidates for the template image.

[0061] The table shown in Figure 11 is an example of a table that stores the AI ​​inference results in the initial display state. This table is stored, for example, in the primary storage unit 216. In the example in Figure 11, the fungal number 1101 is the temporary ID of the fungus to which the AI ​​inference frame was added during the AI ​​inference process. The position X and position Y of position 1102 indicate the coordinates of the upper left corner of the AI ​​inference frame in μm, respectively. The width 1103 and height 1104 indicate the width and height of the AI ​​inference frame in μm, respectively. The fungal species 1105 indicates the fungal species obtained as an AI inference result, and corresponds to the class resulting from performing deep learning as a classifier. The AI ​​inference score 1106 is the AI ​​inference score resulting from performing deep learning. As described above, in this embodiment, the AI ​​inference frame with the highest AI inference score is used for template matching. In this case, the AI ​​inference frame for fungal number 4 with an AI inference score of 0.95783 will be used for template matching. Regarding bacterial numbers other than 1-6, explanations are omitted because, due to the expected predetermined error, these represent bacteria in areas not used for template matching.

[0062] Furthermore, in this embodiment, the maximum value of the system error is set to be within 5 μm per 10 μm of movement, as a predetermined expected error. If an error exceeding this expectation occurs and the evaluation value of the template matching does not exceed the threshold, the correction amount determination unit 222 performs the following exception processing. For example, as an exception processing, the correction amount determination unit 222 can perform AI inference processing on the entire observation range after observation range movement without using the inferred AI inference frame (although the display response will be reduced), and then perform superimposed display.

[0063] In S1002, the correction amount determination unit 222 determines the search range for template matching. As mentioned above, the instruction to change the observation range in this case is 10 μm in the X direction, so the error range for this amount of movement is set to 5 μm, and the search range is determined to be within a range of 5 μm in all directions (front, back, left, and right). In a system where the system error increases with the amount of movement, the correction amount determination unit 222 can perform more suitable template matching by increasing the search range according to the amount of movement.

[0064] In step S1003, the correction amount determination unit 222 performs image conversion on both the template image and the search image to create images suitable for template matching. Here, as is commonly used in template matching, the luminance value is calculated from the RGB video signal transmitted from the digital camera 206, and its high-frequency component is calculated and used. As a result, image contour information is obtained, enabling template matching that is suitable in terms of speed and accuracy.

[0065] In this embodiment, the following ITU-R BT709 formula will be used to calculate the luminance value (Y). Y = 0.299×R+0.587×G+0.114×B

[0066] In S1004, the correction amount determination unit 222 performs template matching. Then, in S1005, the correction amount determination unit 222 determines the correction amount based on the results of template matching. The template matching and correction amount determination in the correction amount determination process will be explained below with reference to Figure 12.

[0067] Figure 12 shows 1201, which is a template image for performing template matching. Template image 1201 is an image obtained by calculating the high-frequency components of the brightness components through image processing for the image within frame 24 in Figure 7, and is an image that contains information about the contour features of the bacteria.

[0068] Image 1202 is an image obtained by calculating the high-frequency components of the brightness component using the same image processing as the image of the observation range after movement. Image 1203 shows the coordinates of the upper left of the AI ​​inference frame for bacterium number 4 when no correction processing is performed. From the table in Figure 11, the coordinates are (X,Y)=(15.9,9.26). As shown in the figure, it is in a different position from bacterium 4.

[0069] In this embodiment, the correction amount determination unit 222 performs template matching within a range of 5 μm in all directions (front, back, left, and right) based on the coordinates (15.9, 9.26) when no correction processing is performed. For this reason, the correction amount determination unit 222 performs a difference calculation with the observed image for all pixels of the template image, shifting the template one pixel at a time from the top left to the bottom right of the search range, as shown in 1205. The correction amount determination unit 222 determines the position where the sum of these differences is the smallest within the search range 1204 as the position where the bacteria 4 exist.

[0070] In this embodiment, the difference calculation is performed while shifting the template by one pixel at a time, but this is not limited to this. If a slight shift in the display position of the AI ​​inference frame is acceptable, the template matching process may be sped up by shifting it by two pixels at a time, for example.

[0071] Figure 13 shows the position of the template frame when the difference calculation in template matching is minimized. As shown in the figure, the top left of the template frame is located at the point where it has moved by the difference (X,Y)=(-2,-1) from the assumed position 1203 when system system errors are not considered. In this way, the correction amount determination unit 222 determines the correction amount to be (X,Y)=(-2,-1) from the difference caused by system system errors.

[0072] In this embodiment, a template matching algorithm that is currently in common use is employed, but this is not limited to this; other algorithms that determine the position correction amount from the correlation between images may also be used.

[0073] Furthermore, in this embodiment, the correction amount was determined using one AI inference frame. However, if there are errors in the rotational direction in addition to the XY direction, template matching may be performed using multiple AI inference frames to determine the XY correction amount and the rotational direction correction amount.

[0074] Furthermore, although this embodiment describes an example where the AI ​​inference unit 213 performs inference using deep learning, the AI ​​inference unit 213 may use algorithms of other machine learning models capable of classifying objects observed in an image.

[0075] As described above, in this embodiment, when the observation range is moved, the amount of movement due to the control signal is corrected by the correction amount determination unit 222 according to the correction amount determined using the images before and after the movement, and the AI ​​inference frame obtained by the AI ​​inference processing before the movement is superimposed on the corrected position. In this way, even if there is an error in the movement of the microscope's observation range, the superimposed display can be performed quickly and accurately.

[0076] (Embodiment 2) Next, a microscope system according to Embodiment 2 will be described. In Embodiment 2, the template matching search range is changed according to the situation. This makes it possible to determine a suitable correction amount even when there is a large system error due to wear of actuator parts, vibration or tilt of the installation environment, etc. Therefore, although the template matching process by the correction amount determination unit 222 in this embodiment differs from that of Embodiment 1, the other configurations and processes are substantially the same as those of Embodiment 1. For this reason, the same or substantially the same configurations and processes are denoted by the same reference numerals and their descriptions are omitted.

[0077] Figure 14 shows 1401, which is a list box displayed on the display 103. The control unit 224 receives user operations on the list box 1401 via an input device such as a mouse, and sets the template matching search range according to the user operation. In other words, the control unit 224 functions as a tolerance specification means that can specify the template matching search range according to the tolerance. The control unit 224 stores the template matching search range selected by the user in, for example, the secondary storage unit 215.

[0078] The system error of the microscope system in this embodiment follows a normal distribution, for example, as shown in 1402, and when moved by 10 μm by the control signal 218, σ = 2 μm. For example, by storing information on the distribution of the error when the observation range is actually moved by the control signal 218 in the secondary storage unit 215, the value of σ related to that distribution can be obtained.

[0079] For example, suppose user 105 operates the list box 1401 and selects 3σ. In this case, if the user instructs a change in the observation range by only 10 μm, the template matching search range will be 6 μm in all directions (front, back, left, and right). The correction amount determination unit 222 will perform template matching across the search range shown by 1501 in Figure 15.

[0080] As described above, in the microscope system according to this embodiment, the search range of the template matching applied is varied depending on the settings. In this way, it becomes possible to adjust the balance between the response when changing the observation range and the accuracy of template matching according to the state and performance of the equipment in the system, the conditions of the installation environment, etc.

[0081] (Embodiment 3) Furthermore, Embodiment 3 will be described. In the microscope system according to this embodiment, in order to perform template matching with higher accuracy, template matching is performed using color information in addition to template matching using brightness information. Therefore, in this embodiment, the template matching process by the correction amount determination unit 222 differs from that of Embodiment 1, but the other configurations and processes are substantially the same as those of Embodiment 1. For this reason, the same or substantially the same configurations and processes are denoted by the same reference numerals and their descriptions are omitted.

[0082] In the correction amount determination process shown in Figure 10, in the image conversion in S1003 of Embodiment 1, the luminance value was calculated and its high-frequency component was calculated and used. In contrast, in this embodiment, the correction amount determination unit 222 performs template matching for the high-frequency component of the luminance value in the template matching process in S1004, and also performs image conversion to chrominance signals Cr and Cb. The correction amount determination unit 222 then evaluates the sum of the differences between the template and the observed image and the sum of the differences in luminance values ​​for these density values ​​and performs template matching. When performing image conversion of Cr and Cb, the ITU-R BT709 conversion formula shown in the following formula shall be used. Cb = -0.168736×R-0.331264×G+0.5×B Cr = 0.5×R-0.418688×G-0.081312×B

[0083] Gram-stained bacteria include GPCs and GNCs, which have similar shapes but different colors. Therefore, to address cases where template matching using only the high-frequency component of luminance does not function well, a high-speed mode using only luminance and a high-precision mode that evaluates both luminance and color information for template matching may be provided. In other words, the high-speed mode uses only luminance, requiring less computation than the high-precision mode, and thus enabling faster template matching. On the other hand, the high-precision mode uses both luminance and color information, requiring more computation than the high-speed mode, and thus enabling more accurate template matching.

[0084] Furthermore, the control unit 224 may display a user interface on the display unit 217 for selecting a high-precision mode, and accept the user 105's selection of the above mode.

[0085] As described above, in this embodiment, the correction amount determination unit 222 is provided with a high-precision mode that evaluates brightness and color information and performs template matching, making it possible to display a more suitable AI inference frame superimposed.

[0086] (Embodiment 4) Furthermore, Embodiment 4 will be described. In this embodiment, in the template usage frame selection process of S1001 by the correction amount determination unit 222, highly accurate template matching is performed even for observation targets with few shape features, such as non-chained GPCs. Therefore, in this embodiment, the template matching process by the correction amount determination unit 222 differs from that of Embodiment 1, but the other configurations and processes are substantially the same as those of Embodiment 1. For this reason, the same or substantially the same configurations and processes are denoted by the same reference numerals and their descriptions are omitted.

[0087] Figure 16 shows the observation of unlinked GPCs with few morphological features. When the AI ​​inference score for bacterium 1 in the figure is the highest, selecting the AI ​​inference frame of bacterium 1 as the template image results in high template matching evaluation values ​​at all positions of bacterium 2-6, increasing the likelihood of incorrect template matching.

[0088] Therefore, in the template usage frame selection process according to this embodiment, in S1001, the frame indicated by 1601 is selected as the template image for template matching in order to perform more accurate template matching.

[0089] As shown in Figure 16, for areas where AI inference frames are geographically concentrated, the correction amount determination unit 222 selects the area surrounding the AI ​​inference frames constituting the concentrated area as the template usage frame 1601. Here, as an example of detecting areas where AI inference frames are concentrated, the AI ​​inference result table in Figure 11 is referred to, and overlapping frames are collected in a chain and grouped based on their position, width, and height. As a result, for the group with the most frames, the frames located at the outermost edges within the group are determined vertically and horizontally, and the area surrounding their coordinates is set as the template usage frame.

[0090] Furthermore, if there are no groups that encompass more than the predetermined frame, such as when bacteria are sparsely distributed, the number of overlapping frames may be increased by finding overlaps in frames that are expanded by doubling or tripling the width and height of the AI ​​inference frame.

[0091] As explained above, in this embodiment, even if a single AI inference frame has few features, the template selection process enables highly accurate template matching by using a frame that includes multiple AI inference frames.

[0092] In actual observational images, Gram staining irregularities and non-bacterial objects often appear in the background. Therefore, even a template image of a single bacterium may contain features, and template matching may be successful. On the other hand, there are cases where Gram staining is effective and there are no (or few) staining irregularities. In these cases, high-precision template matching can be performed by selecting the template usage frame using the high-precision mode.

[0093] (Embodiment 5) Furthermore, Embodiment 5 will be described. This embodiment differs from the above-described embodiment in that the superimposed image generation unit 214 performs a process to generate an image synthesis result from additionally captured images. Therefore, other configurations and processes are substantially the same as those of Embodiment 1. For this reason, the same reference numerals are used for the same or substantially the same configurations and processes, and their descriptions are omitted.

[0094] The image synthesis process according to this embodiment will be explained with reference to Figure 17. In Figure 17, 1701 shows the entire sample. 1702 to 1708 show the trajectory of the observation range as it moved by the observation range change instruction unit 221. The control unit 224 stores information indicating the history of the observation range position (observation range history table 1801), as shown in Figure 18, in the primary storage unit 216.

[0095] In Figure 18, ID 1802 corresponds to the numbers of observation ranges 1702 to 1708. The position X and position Y of position 1803 record the upper-left coordinates of each observation range when the lower left of the initial display state observation range shown in 1702 is set to (X,Y)=(0,0). The file name 1804 indicates that the images taken in each observation range are stored in the secondary storage unit 215 with the file names shown in the table. Note that the coordinates of each observation range may also be coordinates corrected by a correction amount determined by the correction amount determination unit 222 using adjacent images (images before and after movement).

[0096] In this embodiment, the microscope system 10 generates a composite image by combining images of the observation ranges recorded in the observation range history table 1801, in accordance with instructions from the user 105, and displays it on the display unit 217. The microscope system 10 also saves the composite image to the secondary storage unit 215 in accordance with instructions from the user 105.

[0097] Figure 19 schematically shows an example of image synthesis result 1901 generated from captured images. As shown in the figure, it is possible to show observation results over a wider range than the observation range by microscope, so the user can easily grasp the overall picture of the sample. Furthermore, after a simple diagnosis immediately after obtaining a sample from a patient, and after culturing the sample for a more accurate diagnosis, comparing the diagnosis from the culture with the Gram stained image while looking at the image synthesis result 1901 can help improve the accuracy of subsequent diagnoses.

[0098] As described above, in this embodiment, by using the position history of the observation range to synthesize images of multiple observation ranges, an image showing observation results wider than the observation range of the microscope is synthesized. This makes it possible to easily grasp the overall picture of the sample as described above.

[0099] In this embodiment, a composite image is generated from observation images based on observation range movement instructions given by user 105. However, this embodiment is not limited to this, and the observation range may be moved automatically from the upper left to the lower left of the sample so that the observation ranges overlap, and the image synthesis process may be performed in this manner. Even in this way, a composite image of the entire sample can be obtained without requiring any intervention from user 105.

[0100] Furthermore, for the sake of clarity in the diagram, the AI ​​inference frame is not shown in the example of the image synthesis result 1901 described above. However, the superimposed image generation unit 214 may also generate an image in which the AI ​​inference frame is superimposed on the image synthesis result 1901 by referring to the AI ​​inference result table in Figure 11.

[0101] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0102] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0103] 10...Microscope system, 101...Microscope device, 102...PC, 103...Display, 212...Video input unit, 213...AI inference unit, 214...Superimposed image generation unit, 222...Correction amount determination unit

Claims

1. A microscope system having a change instruction means for instructing a change in the observation range, An acquisition means for acquiring an image of the observation range including the object to be observed, An inference means for inferring the type of object to be observed included in the aforementioned image, A generation means for generating a superimposed image by superimposing the inference result from the inference means onto the position of the object being observed in the image, The system includes a correction amount determination means for determining a correction amount related to the difference between the amount of change instruction by the change instruction means and the actual amount of movement of the observation range, The microscope system is characterized in that, when generating a superimposed image based on a second image taken after changing the observation range following the acquisition of a first image, the generation means generates a superimposed image in which the inference results of the inference means for the first image are superimposed on the second image, with the position of the objects to be observed included in the second image that are also included in the first image corrected according to the amount of the change instruction and the correction amount.

2. The microscope system according to claim 1, characterized in that, when the inference means generates a superimposed image based on the second image, it infers the type of object among the objects to be observed that are not included in the first image, which are included in the second image.

3. The microscope system according to claim 1, characterized in that the correction amount determination means determines the correction amount by performing a template matching process on a predetermined range of the second image based on the amount of the change instruction.

4. The inference means includes, as an inference result by the inference means, information on the position and size of the object being observed in the image, The microscope system according to claim 3, characterized in that the correction amount determination means generates a template image to be used in the template matching process based on information of the position and size of the object to be observed.

5. The inference means includes, as an inference result by the inference means, a score indicating the probability that the inference of the observed object is correct. The microscope system according to claim 4, characterized in that the correction amount determination means selects information on the position and size of an object with a higher score from among a plurality of detected objects to be used as the template image.

6. The system further includes a selection means for selecting a mode for the template matching process, The microscope system according to claim 3, characterized in that the correction amount determination means performs template matching processing using luminance values ​​in the first mode, and performs template matching processing using luminance values ​​and color information in the second mode.

7. The system further includes a means for specifying the search range of the template matching process according to the tolerance error, The microscope system according to claim 3, characterized in that the correction amount determination means performs the template matching process based on the search range specified by the designation means.

8. The microscope system according to claim 7, further comprising a storage means for storing information regarding errors when the observation range is actually moved, for specifying the search range of the template matching process according to the tolerance error.

9. The microscope system according to claim 1, further characterized in that the generation means generates a composite image by combining images of multiple observation ranges based on images of multiple observation ranges, the amount of change instruction for each observation range, and the correction amount determined by the correction amount determination means.

10. An electronic device capable of controlling a microscope, A transmission means for transmitting a control signal based on a change instruction means for instructing a change in the observation range for the control of the microscope, An acquisition means for acquiring an image of the observation range including the object to be observed, An inference means for inferring the type of object to be observed included in the aforementioned image, A generation means for generating a superimposed image by superimposing the inference result from the inference means onto the position of the object being observed in the image, The system includes a correction amount determination means for determining a correction amount related to the difference between the amount of change instruction by the change instruction means and the actual amount of movement of the observation range, The generating means is characterized in that, when generating a superimposed image based on a second image taken after changing the observation range following the acquisition of a first image, it generates a superimposed image in which the inference results of the inference means for the first image are superimposed on the second image, with respect to the objects of observation included in the second image that are also included in the first image, the positions of which have been corrected according to the amount of the change instruction and the correction amount.

11. A control method for a microscope system having a change instruction means for instructing a change in the observation range, An acquisition step of acquiring an image of the observation range including the object to be observed, An inference step for inferring the type of object to be observed in the aforementioned image, A generation step that generates a superimposed image by superimposing the inference result from the inference step onto the position of the object being observed in the image, The system includes a correction amount determination step, which determines a correction amount related to the difference between the amount of change instruction by the change instruction means and the actual amount of movement of the observation range. A method for controlling a microscope system, characterized in that the generation step, when generating a superimposed image based on a second image taken after changing the observation range following the acquisition of a first image, includes generating a superimposed image in which the inference results of the inference step for the first image are superimposed on the second image, with the position of objects to be observed that are also included in the first image among the objects to be observed included in the second image corrected according to the amount of the change instruction and the correction amount.

12. A program that causes a computer to function as one of the means of the electronic device described in claim 10.

Images