Microscope system

Abstract

To stably perform appropriate apparatus settings according to a scene in a microscope system.SOLUTION: A microscope system 1 comprises: a microscope 10; a digital camera 20 that picks up an image of a sample S through the microscope 10; a scene recognition unit 31 that performs scene recognition of a plurality of scenes by using a trained machine learning model on the basis of the image of the sample S obtained by the digital camera 20; a scene determination unit 32 that performs scene determination on the basis of a result of recognition performed by the scene recognition unit 31; a stabilization unit 40 that temporally stabilizes a result of determination made by the scene determination unit 32; and a setting unit 33 that changes the settings of the digital camera 20 according to a result of determination made by the scene determination unit 32.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a microscope system. [Background technology]

[0002] In observations using a microscope system, the optimal device settings vary depending on the scene, which consists of the observation method, the subject, or a combination of these, etc. Therefore, users of the microscope system are required to appropriately change the device settings depending on the scene, but it is not necessarily easy for users to set the optimal settings for each scene by themselves.

[0003] A technology related to such a problem is described, for example, in Patent Document 1. Patent Document 1 describes a technology that uses artificial intelligence (learning model) to estimate (infer) shooting parameters intended by a user from a live image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6960445 Summary of the Invention [Problem to be solved by the invention]

[0005] However, even if the technology described in Patent Document 1 is used, it is not easy to automate device settings according to the scene with high accuracy. This is because images acquired with various device settings may be used as live images used to infer shooting parameters, and learning all patterns in advance requires a huge amount of learning data and a lot of time. This situation can occur particularly prominently in devices such as microscope systems that have advanced functions and allow various device settings.

[0006] Furthermore, the use of a learning model with low inference accuracy has the disadvantages not only of not being able to set the appropriate equipment according to the scene, but also of causing other problems such as frequent switching of equipment settings, which may hinder observation work, because the inference results may suffer from hunting due to the low inference accuracy of the learning model even when the same scene is being photographed.

[0007] In view of the above circumstances, an object of one aspect of the present invention is to provide a technique for stably performing appropriate device settings according to a scene in a microscope system. [Means for solving the problem]

[0008] A microscope system according to one aspect of the present invention includes a microscope, an imaging unit that images a subject through the microscope, a scene recognition unit that performs scene recognition using a machine learning model that has learned multiple scenes based on an image of the subject obtained by the imaging unit, a scene determination unit that performs scene determination based on the recognition result of the scene recognition unit, a stabilization unit that stabilizes the determination result of the scene determination unit over time, and a setting unit that changes the settings of the imaging unit in accordance with the determination result of the scene determination unit. Effect of the Invention

[0009] According to the above aspect, it is possible to provide a technique for stably performing appropriate device settings according to a scene in a microscope system. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a configuration of a microscope system according to an embodiment of the present invention; [Diagram 2] 4A and 4B are diagrams illustrating the configuration of a stabilization unit according to an embodiment of the present invention; [Diagram 3] FIG. 2 is a diagram showing an example of a flowchart of an automatic setting process performed by a control device according to the first embodiment of the present invention. [Figure 4]FIG. 2 is a diagram for explaining the timing at which a scene determination is performed in the control device according to the first embodiment of the present invention. [Diagram 5] FIG. 4 is a diagram showing an example of a flowchart of frame processing performed by the control device according to the first embodiment of the present invention. [Figure 6] FIG. 3 is a diagram showing an example of a flowchart of a determination process performed by a control device according to the first embodiment of the present invention. [Figure 7] FIG. 11 is a diagram showing an example of a flowchart of frame processing performed by a control device according to a second embodiment of the present invention. [Figure 8] FIG. 11 is a diagram showing an example of a flowchart of a determination process performed by a control device according to a third embodiment of the present invention. [Figure 9] FIG. 13 is a diagram showing an example of a flowchart of a determination process performed by a control device according to a fourth embodiment of the present invention. [Figure 10] FIG. 13 is a diagram showing an example of a flowchart of a determination process performed by a control device according to a fifth embodiment of the present invention. [Figure 11] FIG. 13 is a diagram showing an example of a flowchart of a determination process performed by a control device according to a sixth embodiment of the present invention. [Figure 12] FIG. 13 is a diagram illustrating the configuration of a microscope system according to a seventh embodiment of the present invention. [Figure 13] FIG. 23 is a diagram showing an example of a flowchart of a determination process performed by a control device according to an eighth embodiment of the present invention. [Figure 14] FIG. 13 is a diagram illustrating the configuration of a microscope system according to a ninth embodiment of the present invention. [Figure 15] FIG. 2 is a diagram illustrating a hardware configuration of a computer for implementing a control device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Fig. 1 is a diagram illustrating the configuration of a microscope system according to an embodiment of the present invention. The microscope system 1 shown in Fig. 1 includes a microscope 10, a digital camera 20 attached to the microscope 10, and a control device 30 that controls the microscope 10 and the digital camera 20.

[0012] The microscope 10 may be any microscope equipped with a digital camera 20, and may be a microscope that is compatible with multiple observation methods, such as bright field, dark field, fluorescent observation, polarized light observation, and differential interference observation.

[0013] The digital camera 20 is an example of an imaging unit of the microscope system 1. The digital camera 20 captures an image of the specimen S, which is a subject, via the microscope 10, and acquires an image of the specimen S. The digital camera 20 includes an image sensor, and the image sensor is, for example, a CCD or a CMOS. The image of the specimen S acquired by the digital camera 20 is output to the control device 30.

[0014] The digital camera 20 performs predetermined image processing on the image of the specimen S to adjust brightness and color, and then outputs the image to the control device 30. The image processing performed by the digital camera 20 is not particularly limited, but includes, for example, white balance correction, color matrix correction, gamma correction, and the like.

[0015] The control device 30 changes the settings of the microscope system 1 based on the image acquired by the digital camera 20. The control device 30 includes a scene recognition unit 31, a scene determination unit 32, a setting unit 33, and a stabilization unit 40 as components related to the setting change, as shown in FIG.

[0016] The scene recognition unit 31 performs scene recognition based on the image of the specimen S obtained by the digital camera 20, using a machine learning model that has learned a plurality of scenes. The scene determination unit 32 performs scene determination based on the recognition result of the scene recognition unit 31. The setting unit 33 changes the settings of the digital camera 20 according to the determination result of the scene determination unit 32. The setting unit 33 may further change the settings of the microscope 10 according to the determination result of the scene determination unit 32. The stabilization unit 40 stabilizes the determination result of the scene determination unit 32 over time. The stabilization unit 40 will be described in detail later with reference to FIG. 2.

[0017] The image input to the scene recognition unit 31 is, for example, a live image (moving image) before actual shooting (still image shooting) performed during observation with the microscope system 1. The scene recognition unit 31 recognizes the scene for each frame of the input live image, for example.

[0018] The scene recognized by the scene recognition unit 31 relates to, for example, the type of specimen S, the observation method, a combination thereof, etc. However, the scene is not necessarily limited to the subject and the observation method as long as it has a relatively strong correlation with the settings of the microscope system 1.

[0019] Specifically, the machine learning model used in the scene recognition unit 31 is a machine learning model that has learned scenes such as the type of sample S, the observation method, and combinations thereof by deep learning, and estimates the type of sample S, the observation method, and combinations thereof, that is, the scene. More specifically, the machine learning model used in the scene recognition unit 31 uses an object detection algorithm such as YOLO (You Only Look Once). The output from the scene recognition unit 31, that is, the recognition result of the scene recognition unit 31, is, for example, the sum of the reliability of each class of one or more bounding boxes detected from an image (hereinafter, referred to as total reliability). That is, it includes the same number of total reliability as the number of classes. Note that the classes classified by the machine learning model each correspond to a scene.

[0020] The scene determination unit 32 performs scene determination by comparing the recognition result (or the accumulated recognition result described later) of the scene recognition unit 31 with a threshold. The threshold is set in advance as an initial value such that the recognition result (or the accumulated recognition result described later) of the scene recognition unit 31 can be considered sufficiently reliable if the total reliability (or the accumulated total reliability described later) is equal to or greater than the threshold. Then, the scene determination unit 32 is configured to reserve scene determination if the total reliability (or the accumulated total reliability described later) is less than the threshold. This enables scene determination with a higher accuracy than scene recognition by the scene recognition unit 31 alone.

[0021] The control device 30 stores a parameter list corresponding to optimal settings for each scene in advance. The setting unit 33 reads out the parameter list corresponding to the scene determined by the scene determination unit 32, and sets it in the digital camera 20 and the microscope 10. An example of the setting of the digital camera 20 performed by the setting unit 33 is, but is not limited to, for example, setting the exposure time. An example of the setting of the microscope 10 performed by the setting unit 33 is, but is not limited to, for example, inserting or removing an optical element (such as a polarizer or analyzer) depending on the observation method.

[0022] In this way, by including the scene recognition unit 31 and the scene determination unit 32, the control device 30 can perform scene determination with an accuracy higher than that of a single machine learning model used for scene recognition, and can automatically perform optimal settings for the microscope system 1 according to the scene. Furthermore, by including a stabilization unit 40 that stabilizes the determination result over time, the control device 30 can further improve the scene determination accuracy and suppress hunting. This makes it possible to avoid adverse effects on the observation work caused by frequent setting changes.

[0023] 2 is a diagram illustrating the configuration of a stabilization unit according to an embodiment of the present invention. Hereinafter, the setting change process performed by the control device 30 will be described in more detail with reference to FIG. 2, focusing on the stabilization unit 40.

[0024] As shown in FIG. 2, the stabilization unit 40 includes a scene recognition pre-processing unit 41 that performs pre-processing on the image obtained by the digital camera 20, a scene recognition post-processing unit 42 that performs post-processing on the recognition result of the scene recognition unit 31, and a threshold adjustment unit 43 that adjusts the threshold used in the scene determination unit 32.

[0025] The scene recognition pre-processing unit 41 is configured to perform the reverse conversion of the image processing performed by the digital camera 20 on the image. The image processing performed by the digital camera 20 is performed for various purposes, such as preventing whiteout, improving color reproducibility, and reproducing a natural appearance similar to that seen by the human eye. These image processing processes have a relatively large effect on the image, and are dependent on the device itself and device settings that are not necessarily optimized, so that predictability at the time of learning is low, and it is not easy to learn the effects in advance. For this reason, if sufficient learning has not been performed on the image acquired by the device or on the image acquired by the device with the current settings, the recognition accuracy of the scene recognition unit 31 may decrease. The scene recognition pre-processing unit 41 takes this point into consideration.

[0026] The scene recognition pre-processing unit 41 can cancel the influence of the image processing performed by the digital camera 20 by performing an inverse transformation of the image processing performed by the digital camera 20. The content of the image processing performed by the digital camera 20 can be specified from the identification information of the digital camera 20 and the settings of the digital camera 20. The scene recognition unit 31 performs scene recognition based on the image obtained by the inverse transformation in the scene recognition pre-processing unit 41 (hereinafter, referred to as the inverse transformed image), so that a decrease in the recognition accuracy of the machine learning model caused by the image processing performed by the digital camera 20 can be suppressed.

[0027] In the above, it has been described that the scene recognition unit 31 performs inverse conversion during inference using a trained machine learning model, but it is also desirable to perform inverse conversion of the image processing performed on the digital camera that acquired the learning image during learning of the machine learning model used in the scene recognition unit 31. However, information (settings, etc.) on the digital camera at the time of image acquisition is not necessarily provided together with the learning image (processed image) as learning data. For this reason, for learning of the machine learning model, an image that has been inversely converted from the image processing performed on the digital camera may be used as the learning image, an image that has not been inversely converted from the image processing performed on the digital camera, or the image itself output from the digital camera may be used as the learning image, or both of these may be used as the learning image.

[0028] As described above, the scene recognition pre-processing unit 41 performs pre-processing before the recognition processing performed by the scene recognition unit 31, thereby improving the recognition result (scene recognition result) itself of the scene recognition unit 31. In contrast, the scene recognition post-processing unit 42 performs post-processing after the recognition processing performed by the scene recognition unit 31, thereby improving not the recognition result (scene recognition result) itself of the scene recognition unit 31, but the determination result based on the recognition result (scene determination result).

[0029] The scene recognition post-processing unit 42 is configured to calculate a cumulative recognition result (hereinafter, referred to as a cumulative recognition result) for a period corresponding to a plurality of frames based on a plurality of recognition results of the scene recognition unit 31 corresponding to a plurality of frames. In live shooting, where the image changes from moment to moment due to the movement of the specimen S and changes in the environment (e.g., focus, lighting, etc.), the scene recognition result may vary over time. The scene recognition post-processing unit 42 takes this into consideration.

[0030] The scene recognition post-processing unit 42 calculates a cumulative recognition result based on the recognition results for a plurality of frames, for example by accumulating the total reliability for each class over a plurality of frames to calculate a cumulative total reliability for each class, and outputs the cumulative recognition result to the scene determination unit 32. That is, the cumulative recognition result includes the cumulative total reliability for each class. This makes it possible to prevent irregularly occurring erroneous recognition results from being immediately reflected in the settings of the microscope system 1. In addition, the recognition results for each frame are averaged by the accumulation process. Therefore, by the scene determination unit 32 performing scene determination based on the cumulative recognition results calculated by the scene recognition post-processing unit 42, the influence of blurring of scene recognition occurring during live shooting can be suppressed, and the determination result can be stabilized over time.

[0031] The number of frames for which the post-scene recognition processing unit 42 performs the cumulative processing is not particularly limited, but may be determined in advance. For example, the scene determination may be performed based on the cumulative recognition results corresponding to five frames, and the settings may be updated. Alternatively, the number of frames may not be determined in advance, and the scene determination may be performed when the cumulative recognition results satisfy a predetermined condition, and the settings may be changed based on the determination results.

[0032] The scene recognition post-processing unit 42 may further be configured to set a period of a predetermined number of frames (e.g., a period of five frames) from the timing of the setting change made by the setting unit 33 as a dead zone, so that the recognition results of the scene recognition unit 31 corresponding to the frames in the dead zone are not reflected in the cumulative recognition result. Furthermore, the scene recognition post-processing unit 42 may set a period of a predetermined number of frames from the timing when the setting change is scheduled (e.g., every five frames) as a dead zone, regardless of whether the setting change has been made or not. By the scene recognition post-processing unit 42 setting the dead zone, the recognition results of the scene acquired in an unstable state by the microscope system 1 are not reflected in the scene determination. Therefore, the determination result can be stabilized over time.

[0033] As described above, the scene recognition pre-processing unit 41 and the scene recognition post-processing unit 42 improve the scene determination result by directly or indirectly improving the cumulative scene recognition result, whereas the threshold adjustment unit 43 improves the scene determination result by adjusting the threshold value that is compared with the cumulative scene recognition result.

[0034] The threshold adjustment unit 43 is configured to increase the threshold when the scene with the highest evaluation in the cumulative recognition result is different from the scene corresponding to the judgment result output immediately before. The threshold adjustment unit 43 is also configured to reset the threshold to an initial value when the scene with the highest evaluation in the cumulative recognition result is the same as the scene corresponding to the judgment result output immediately before. As a result, when it is determined that a scene has been changed, the judgment is made with stricter standards than when it is determined that the scene has not been changed. In a situation where an erroneous scene judgment is likely to occur, the cumulative recognition result is generally less likely to become large compared to when a scene judgment is performed correctly. For this reason, it is possible to avoid erroneous judgment in advance by making a judgment with an increased threshold. In addition, in the above configuration, when scene changes occur continuously, the threshold increases more and more, so that the occurrence of hunting can be effectively suppressed. Furthermore, when it is determined that a scene has not been changed, the threshold is reset and judgment is made with the initial value, so that it is possible to avoid a situation in which the threshold for maintaining a scene is raised in order to avoid an erroneous scene judgment and the judgment is reserved more than necessary.

[0035] The threshold adjustment unit 43 may be further configured to reset the threshold to an initial value when movement of the specimen S is detected, regardless of whether the scene with the highest evaluation in the cumulative recognition result is the same as the scene corresponding to the judgment result output immediately before. When movement of the specimen S is detected, the possibility that a scene change such as replacement of the specimen S or a change in the observation method has occurred is higher than usual. Therefore, in such a case, the threshold adjustment unit 43 may reset the threshold, so that the scene judgment unit 32 may perform scene judgment using the same criteria as when judging that the scene has not been changed, without excessively raising the criteria for judging that the scene has been changed. This configuration also makes it possible to avoid a situation in which judgment is withheld more than necessary.

[0036] The method of detecting the movement of the specimen S is not particularly limited. The threshold adjustment unit 43 may detect the movement of the specimen S based on an image, for example, may detect the movement of the specimen S based on a motion vector calculated from a live image. The threshold adjustment unit 43 may also detect the movement of the specimen S based on an output from a sensor provided on the stage of the microscope 10. Furthermore, the threshold adjustment unit 43 may detect the movement of the specimen S based on a control signal (for example, a signal for controlling the stage) output from the control device 30 to the microscope 10.

[0037] As described above, the microscope system 1 can accurately determine the scene while avoiding hunting in the scene determination result. Therefore, the microscope system 1 can stably perform appropriate device settings according to the scene.

[0038] <First embodiment> FIG. 3 is a diagram showing an example of a flowchart of the automatic setting process performed by the control device according to the present embodiment. FIG. 4 is a diagram for explaining the timing at which scene determination is performed in the control device according to the present embodiment. FIG. 5 is a diagram showing an example of a flowchart of frame processing performed by the control device according to the present embodiment. FIG. 6 is a diagram showing an example of a flowchart of determination processing performed by the control device according to the present embodiment. The automatic setting process performed by the control device 30 will be specifically described with reference to FIGS. 3 to 6. Note that the configuration of the microscope system according to the present embodiment is similar to that of the microscope system 1, and therefore the components of the microscope system according to the present embodiment will be referred to by the same reference numerals as the components of the microscope system 1.

[0039] When the microscope system according to this embodiment starts live shooting, the processor of the control device 30 executes a predetermined program to perform the automatic setting process shown in Fig. 3. Note that, here, as shown in Fig. 4, a case will be described as an example in which evaluation target frames and dead zone frames are set to be repeated for five frames each, and after the five evaluation target frames are completed, scene determination is performed and the settings of the microscope system 1 are updated.

[0040] First, the control device 30 performs frame processing shown in FIG. 5 on each frame of a live image captured by the digital camera 20 via the microscope 10 (step S10).

[0041] In the frame processing of step S10, first, the control device 30 acquires an image of the latest frame of the live image (step S11). Next, the control device 30 calculates a motion vector based on the image acquired in step S11 and a reference image (for example, the image of the first evaluation target frame), and judges whether the specimen S has moved (step S12). If it is judged that the specimen S has moved (step S12 YES), the threshold value used in the scene judgment unit 32 is reset and initialized (step S13). If it is judged that the specimen S has not moved (step S12 NO), the processing of step S13 is skipped.

[0042] Thereafter, the control device 30 judges whether or not the period of the current frame is in the dead zone (step S14). If it is judged that the period of the current frame is in the dead zone (step S14 YES), the frame process shown in FIG.

[0043] On the other hand, if it is determined that the period of the current frame is not a dead zone but a period to be evaluated (step S14 NO), the control device 30 performs an inverse transformation of the image processing performed by the digital camera 20 on the image acquired in step S11 to generate an inversely transformed image (step S15).

[0044] When the inversely transformed image is generated, the control device 30 inputs the inversely transformed image into a machine learning model to perform inference on the scene (step S16), and calculates an evaluation value for each class as a recognition result (step S17). Specifically, the control device 30 calculates an evaluation value for each class, which is a total reliability obtained by adding up the total reliability for each class calculated for a plurality of bounding boxes, as the recognition result. Finally, the control device 30 adds up the evaluation value for each class of the current frame calculated in step S17 to the cumulative evaluation value for each class calculated in the previous frame, calculates the updated cumulative evaluation value for each class as a cumulative recognition result (step S18), and ends the frame processing shown in FIG. 5.

[0045] When the frame processing shown in Fig. 5 is completed, the control device 30 judges whether or not to perform scene determination (step S20). In this example, the control device 30 judges to perform scene determination when five evaluation target frames occur consecutively, as shown in Fig. 5. Therefore, the control device 30 repeatedly performs frame processing on the five evaluation target frames, and calculates a cumulative evaluation value that is an accumulation of the recognition results for the five frames.

[0046] When it is determined that scene determination is to be performed (step S20: YES), the control device 30 performs the determination process shown in FIG. 6 (step S30).

[0047] In the judgment process of step S30, first, the control device 30 identifies the scene (class) with the highest evaluation in the accumulated recognition result (step S31). Next, the control device 30 judges whether or not there has been a change in the scene (step S32). Here, the control device 30 judges whether or not there has been a change in the scene based on whether or not the scene identified in step S31 is different from the scene judged in the immediately preceding judgment process. If the control device 30 judges that there has been a change in the scene, it increases the threshold value used in step S35 (step S33), and if it judges that there has been no change in the scene, it resets the threshold value used in step S35 to the initial value (step S34).

[0048] Thereafter, the control device 30 compares the cumulative evaluation value (maximum cumulative evaluation value) corresponding to the scene identified in step S31, which is included in the cumulative recognition result, with a threshold value (step S35). If it is determined that the maximum cumulative evaluation value exceeds the threshold value, the control device 30 outputs the scene corresponding to the maximum cumulative evaluation value as the determination result (step S36), and ends the determination process shown in Fig. 6. On the other hand, if it is determined that the maximum cumulative evaluation value does not exceed the threshold value, the control device 30 reserves the scene determination (step S37), and ends the determination process shown in Fig. 6.

[0049] 6 is completed, the control device 30 judges whether or not to perform settings (step S40). In step S40, if a judgment result is output in the judgment process of step S30, the control device 30 performs setting processing according to the judgment result (step S50). Specifically, the control device 30 reads out a parameter list corresponding to the judgment result (scene) and updates the settings of the digital camera 20 (for example, exposure time settings) according to the parameter list. On the other hand, if the scene judgment is reserved in the judgment process of step S30, the control device 30 judges that settings will not be performed (step S40 NO) and skips the setting process of step S50.

[0050] Finally, the control device 30 determines whether or not to end the automatic setting process shown in Fig. 3 (step S60). For example, when the control device 30 detects an input of an end instruction from the user (step S60 YES), it ends the automatic setting process, and when there is no end instruction (step S60 NO), it returns to step S10 and repeats the automatic setting process.

[0051] According to the microscope system of this embodiment, the automatic setting process shown in Fig. 3 is performed, and the scene can be accurately determined while avoiding hunting in the scene determination result. Therefore, it is possible to stably perform appropriate device settings according to the scene, and the burden on the user can be reduced by automating the setting work.

[0052] <Second embodiment> Fig. 7 is a diagram showing an example of a flowchart of frame processing performed by the control device according to this embodiment. The configuration of the microscope system according to this embodiment is similar to that of the microscope system 1. The automatic setting processing performed by the control device 30 according to this embodiment differs from the automatic setting processing performed by the control device 30 according to the first embodiment in that the automatic setting processing performed by the control device 30 according to this embodiment includes the frame processing shown in Fig. 7 instead of the frame processing shown in Fig. 5.

[0053] The frame process shown in Fig. 7 includes processes from step S111 to step S118 corresponding to the processes from step S11 to step S18 constituting the frame process shown in Fig. 5. The frame process shown in Fig. 7 differs from the frame process shown in Fig. 5 in that if it is determined in step S112 that the specimen S has not moved (step S112 NO), the frame process ends without performing the subsequent processes.

[0054] The microscope system according to this embodiment can also provide the same effects as the microscope system 1 according to the first embodiment. Furthermore, in the microscope system according to this embodiment, unless movement of the specimen S is detected, the scene recognition process is not performed and the cumulative evaluation value is not updated. Therefore, the cumulative evaluation value does not exceed the threshold value, the scene determination result is not output, and the settings are not changed. As a result, the settings during the period when movement of the specimen S is not detected are fixed, so that hunting of the determination results in the same scene can be reliably prevented.

[0055] <Third embodiment> Fig. 8 is a diagram showing an example of a flowchart of the determination process performed by the control device according to this embodiment. The configuration of the microscope system according to this embodiment is similar to that of the microscope system 1. The automatic setting process performed by the control device 30 according to this embodiment differs from the automatic setting process performed by the control device 30 according to the first embodiment in that the automatic setting process performed by the control device 30 according to this embodiment includes the determination process shown in Fig. 8 instead of the determination process shown in Fig. 6.

[0056] The determination process shown in Fig. 8 includes a process of step S138 in addition to the processes of steps S131 to S137 corresponding to the processes of steps S31 to S37 constituting the determination process shown in Fig. 6. In the determination process shown in Fig. 8, when the control device 30 determines that the maximum accumulated evaluation value exceeds the threshold value (step S135 YES), the control device 30 further determines whether or not the scene corresponding to the highest evaluation is the background (step S138).

[0057] When it is determined that the scene corresponding to the highest evaluation is not the background (step S138 NO), the control device 30 (scene determination unit 32) outputs the scene corresponding to the maximum accumulated evaluation value as the determination result (step S136), and ends the determination process shown in Fig. 8. On the other hand, when it is determined that the scene corresponding to the highest evaluation is the background (step S138 YES), the control device 30 (scene determination unit 32) reserves the scene determination (step S137) as in the case where it is determined that the maximum accumulated evaluation value does not exceed the threshold value (step S135 NO), and ends the determination process shown in Fig. 8.

[0058] The microscope system according to this embodiment can also provide the same effects as those of the microscope system 1 according to the first embodiment. Furthermore, in the microscope system according to this embodiment, when the current scene is estimated to be the background from the accumulated evaluation value, scene determination is reserved. This makes it possible to avoid unnecessary setting changes in situations where it is not desirable to change the settings or where changing the settings is meaningless, such as when the field of view deviates from the specimen S while searching for a desired location in the specimen S or when only the background is captured while replacing the specimen S.

[0059] <Fourth embodiment> Fig. 9 is a diagram showing an example of a flowchart of the determination process performed by the control device according to the present embodiment. The configuration of the microscope system according to the present embodiment is similar to that of the microscope system 1. The automatic setting process performed by the control device 30 according to the present embodiment differs from the automatic setting process performed by the control device 30 according to the first embodiment in that the automatic setting process performed by the control device 30 according to the present embodiment includes the determination process shown in Fig. 9 instead of the determination process shown in Fig. 6.

[0060] The judgment process shown in Fig. 9 includes steps S139 and S140 in addition to steps S131 to S137 corresponding to steps S31 to S37 constituting the judgment process shown in Fig. 6. In the judgment process shown in Fig. 9, when the control device 30 judges that the maximum cumulative evaluation value exceeds the threshold value (step S135 YES), the control device 30 further acquires time-series data of the judgment result that has already been output (step S139) and judges whether hunting between specific scenes occurs (step S140). Here, the specific scenes refer to hunting between scenes that would not normally occur, such as hunting between a scene related to a biological slide specimen and a scene related to an industrial printed circuit board.

[0061] When it is determined that there is no hunting between specific scenes (step S140 NO), the control device 30 (scene determination unit 32) outputs the scene corresponding to the maximum accumulated evaluation value as the determination result (step S136) and ends the determination process shown in Fig. 9. On the other hand, when it is determined that there is hunting between specific scenes (step S140 YES), the control device 30 (scene determination unit 32) reserves the scene determination (step S137) as in the case where it is determined that the maximum accumulated evaluation value does not exceed the threshold value (step S135 NO), and ends the determination process shown in Fig. 9.

[0062] That is, in this embodiment, in the processing from step S135 onwards, when the evaluation value of the highest rated scene exceeds a threshold value, the control device 30 determines whether to reserve the scene judgment or output the scene corresponding to that evaluation value as the judgment result based on the time series data of the scene corresponding to that evaluation value and the judgment result that has already been output.

[0063] The microscope system according to this embodiment can also provide the same effects as the microscope system 1 according to the first embodiment. Furthermore, in the microscope system according to this embodiment, even if the cumulative evaluation value exceeds a threshold, if a state transition that cannot occur and is registered in advance is detected, scene determination is withheld. This makes it possible to avoid, for example, changing settings to match an impossible state transition and prevent hunting due to erroneous recognition.

[0064] <Fifth embodiment> Fig. 10 is a diagram showing an example of a flowchart of the determination process performed by the control device according to this embodiment. The configuration of the microscope system according to this embodiment is similar to that of the microscope system 1. The automatic setting process performed by the control device 30 according to this embodiment differs from the automatic setting process performed by the control device 30 according to the first embodiment in that the automatic setting process performed by the control device 30 according to this embodiment includes the determination process shown in Fig. 10 instead of the determination process shown in Fig. 6.

[0065] The determination process shown in Fig. 10 includes steps S139 and S141 in addition to steps S131 to S137 corresponding to steps S31 to S37 constituting the determination process shown in Fig. 6. In the determination process shown in Fig. 10, when the control device 30 determines that the maximum accumulated evaluation value exceeds the threshold value (YES in step S135), the control device 30 further acquires time-series data of the determination results that have already been output (step S139).

[0066] Thereafter, the control device 30 performs scene determination based on the scene identified in step S131 and the time series data of the determination result acquired in step S139 (step S141). In step S141, the control device 30 uses a machine learning model of a recurrent neural network (RNN) trained by deep learning to infer the scene based on the time series data of the scene. Finally, the control device 30 (scene determination unit 32) outputs the scene determined in step S141 as a determination result (step S136), and ends the determination process shown in FIG. 10.

[0067] The microscope system according to this embodiment can also provide the same effects as the microscope system 1 according to the first embodiment. Furthermore, in the microscope system according to this embodiment, the scene corresponding to the highest evaluation is not used as the judgment result as it is, but the scene is judged taking into consideration the time-series data. This allows for more accurate scene judgment.

[0068] Sixth embodiment Fig. 11 is a diagram showing an example of a flowchart of the determination process performed by the control device according to this embodiment. The configuration of the microscope system according to this embodiment is similar to that of the microscope system 1. The automatic setting process performed by the control device 30 according to this embodiment differs from the automatic setting process performed by the control device 30 according to the first embodiment in that the automatic setting process performed by the control device 30 according to this embodiment includes the determination process shown in Fig. 11 instead of the determination process shown in Fig. 6.

[0069] The determination process shown in Fig. 11 includes a process of step S142 in addition to the processes of steps S131 to S137 corresponding to the processes of steps S31 to S37 constituting the determination process shown in Fig. 6. In the determination process shown in Fig. 11, when the control device 30 determines that the maximum accumulated evaluation value exceeds the threshold value (step S135 YES), the control device 30 determines whether a limit setting (one-time function) that limits the setting of the digital camera 20 to being changed only once is enabled and whether the setting has already been changed once (step S142).

[0070] If the one-time function is not enabled or the settings have not been changed even once (step S142 NO), the control device 30 (scene determination unit 32) outputs the scene corresponding to the maximum accumulated evaluation value as the determination result (step S136) and ends the determination process shown in Fig. 11. On the other hand, if the one-time function is enabled and the settings have already been changed (step S142 YES), the control device 30 (scene determination unit 32) reserves the scene determination (step S137) as in the case where it is determined that the maximum accumulated evaluation value does not exceed the threshold value (step S135 NO), and ends the determination process shown in Fig. 11.

[0071] The microscope system according to this embodiment can also provide the same effects as the microscope system 1 according to the first embodiment. Furthermore, in the microscope system according to this embodiment, when the one-time function (restriction setting) is enabled, the control device 30 (setting unit 33) changes the settings of the digital camera 20 according to the determination result of the scene determination unit 32, and then prohibits any further changes to the settings. This restricts excessive changes to the settings, thereby making it possible to prevent hunting from occurring.

[0072] <Seventh embodiment> Fig. 12 is a diagram illustrating the configuration of a microscope system according to this embodiment. The microscope system 2 shown in Fig. 12 differs from the microscope system 1 in that it includes a digital camera 50 into which an infrared cut filter can be inserted instead of the digital camera 20, and in that the scene recognition unit 31 uses different machine learning models depending on whether an infrared cut filter is inserted into the digital camera 50. The other points are the same as those of the microscope system 1.

[0073] 12, digital camera 50 includes image sensor 51 and image processing unit 52 that processes the output from image sensor 51 to generate an image. Digital camera 50 further includes filter insert / removal unit 53 that can insert / remove an infrared filter onto the optical path between image sensor 51 and microscope 10, and filter detection unit 54 that detects whether an infrared cut filter is inserted onto the optical path.

[0074] There is no particular limitation on the mechanism by which filter inserter / remover 53 detects the insertion of the infrared cut filter. For example, a magnet may be provided in filter inserter / remover 53. Filter detector 54 may include a Hall sensor and detect that the infrared filter is inserted into the optical path by detecting or not detecting the magnet provided in filter inserter / remover 53.

[0075] The digital camera 50 is configured such that, for example, when the microscope system 1 performs observation using visible light, the filter inserting / removing unit 53 inserts an infrared cut filter into the optical path in order to improve color reproducibility. Also, the digital camera 50 is configured such that, for example, when the microscope system 1 performs observation using near-infrared light, the filter inserting / removing unit 53 removes the infrared cut filter from the optical path in order to improve sensitivity.

[0076] The digital camera 50 outputs the detection result of the filter detection unit 54 to the control device 30. The scene recognition unit 31 of the control device 30 is configured to switch the machine learning model used for scene recognition based on the detection result of the filter detection unit 54. Specifically, when the filter detection unit 54 detects that an infrared cut filter is inserted into the optical path, the scene recognition unit 31 performs scene recognition using a machine learning model that has learned the scene from an image acquired using the infrared cut filter. On the other hand, when the filter detection unit 54 detects that an infrared cut filter is not inserted into the optical path, the scene recognition unit 31 performs scene recognition using a machine learning model that has learned the scene from an image acquired without using the infrared cut filter.

[0077] The microscope system 2 according to this embodiment can also provide the same effects as the microscope system 1 according to the first embodiment. Furthermore, in the microscope system according to this embodiment, a machine learning model is used depending on whether the infrared cut filter is inserted or removed, so that more accurate scene determination is possible.

[0078] <Eighth embodiment> Fig. 13 is a diagram showing an example of a flowchart of the determination process performed by the control device according to the present embodiment. The configuration of the microscope system according to the present embodiment is similar to that of the microscope system 2 according to the seventh embodiment. The automatic setting process performed by the control device 30 according to the present embodiment differs from the automatic setting process performed by the control device 30 according to the seventh embodiment in that the automatic setting process performed by the control device 30 according to the present embodiment includes the determination process shown in Fig. 13 instead of the determination process shown in Fig. 6.

[0079] The determination process shown in Fig. 13 includes processes of steps S143 and S144 in addition to processes of steps S131 to S137 corresponding to the processes of steps S31 to S37 constituting the determination process shown in Fig. 6. In the determination process shown in Fig. 13, when the control device 30 outputs a determination result in step S136, the control device 30 determines whether or not a combination of the output determination result and the filter detection result of the filter detection unit 54 is appropriate (step S143).

[0080] Note that combinations of the inserted / removed states of the infrared cut filter that can be used in observation and scenes are preregistered in the control device 30, and in step S143, the control device 30 determines whether the combination is appropriate based on whether the combination of the output judgment result and the filter detection result of the filter detection unit 54 is included in the preregistered combinations. Examples of preregistered combinations are not particularly limited, but may include, for example, both the inserted and removed states for fluorescence observation, phase contrast observation, differential interference observation, and polarized observation, and only the inserted state for brightfield observation.

[0081] Thereafter, when the control device 30 determines that the combination is appropriate (step S143 YES), it ends the determination process shown in FIG. 13. On the other hand, when the control device 30 determines that the combination is inappropriate (step S143 NO), it outputs an alert to the notification unit (step S144). As a result, the notification unit notifies the user that the insertion / removal state of the infrared cut filter is inappropriate for the scene. Note that the notification unit may be, for example, a display device of the control device 30, and the control device 30 may display an alert (for example, a message such as "An infrared cut filter is not inserted. Please insert an infrared cut filter for bright field observation.") on a live image displayed on the display device. That is, the microscope system 2 according to this embodiment includes a notification unit that outputs an alert based on the determination result of the scene determination unit 32 and the detection result of the filter detection unit 54.

[0082] The microscope system according to this embodiment can also provide the same effects as the microscope system 1 according to the first embodiment. Furthermore, the microscope system according to this embodiment can notify the user of a situation in which the combination of the inserted / removed state of the infrared cut filter and the scene is not appropriate. This can prevent the user from using the microscope system with inappropriate settings by encouraging the user to insert or remove the infrared cut filter.

[0083] <Ninth embodiment> Fig. 14 is a diagram illustrating the configuration of a microscope system according to this embodiment. The microscope system 3 shown in Fig. 14 differs from the microscope system 1 in that a digital camera 60 is provided instead of the digital camera 20 and the control device 30. The other points are the same as those of the microscope system 1.

[0084] The digital camera 60 includes a control unit 70 that performs scene recognition, stabilization processing, scene determination, and setting change, in addition to the image sensor 51 and image processing unit 52. That is, like the control device 30 of the microscope system 1, the digital camera 60 has a function of automating setting changes according to the scene (scene recognition unit 31, scene determination unit 32, setting unit 33, and stabilization unit 40).

[0085] The microscope system 3 according to this embodiment can also provide the same effects as those of the microscope system 1 according to the first embodiment.

[0086] Fig. 15 is a diagram illustrating a hardware configuration of a computer 100 for realizing the control device or control unit according to the above-mentioned embodiment. The hardware configuration illustrated in Fig. 15 includes, for example, a processor 101, a memory 102, a storage device 103, a reading device 104, a communication interface 106, and an input / output interface 107. The processor 101, the memory 102, the storage device 103, the reading device 104, the communication interface 106, and the input / output interface 107 are connected to each other, for example, via a bus 108.

[0087] The processor 101 may be any processor, and may include a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a general-purpose computing on graphics processing units (GPGPU), and the like.

[0088] The processor 101 may function as the scene recognition unit 31, the scene determination unit 32, the setting unit 33, and the stabilization unit 40 by reading and executing a program stored in the storage device 103. Note that, as an example, the scene recognition unit 31 may be configured using, for example, a GPGPU or an ASIC, and the scene determination unit 32, the setting unit 33, and the stabilization unit 40 may be configured using a CPU or an FPGA.

[0089] The memory 102 is, for example, a semiconductor memory, and may include a RAM area and a ROM area. The storage device 103 is, for example, a semiconductor memory such as a hard disk or a flash memory, or an external storage device.

[0090] The reader 104 accesses the removable storage medium 105 according to, for example, an instruction from the processor 101. The removable storage medium 105 is realized by, for example, a semiconductor device, a medium in which information is input and output by magnetic action, a medium in which information is input and output by optical action, or the like. The semiconductor device is, for example, a Universal Serial Bus (USB) memory. The medium in which information is input and output by magnetic action is, for example, a magnetic disk. The medium in which information is input and output by optical action is, for example, a Compact Disc (CD)-ROM, a Digital Versatile Disk (DVD), a Blu-ray Disc, etc. (Blu-ray is a registered trademark).

[0091] The communication interface 106 communicates with other devices (e.g., the microscope 10, the digital camera 20, etc.) according to, for example, instructions from the processor 101. The input / output interface 107 is, for example, an interface between an input device and an output device. The input device is, for example, a device such as a mouse, keyboard, or foot switch that accepts instructions from a user. The output device is, for example, a display device and an audio device such as a speaker.

[0092] The program executed by the processor 101 is provided to the computer in the following form, for example. (1) It is pre-installed in the storage device 103. (2) Provided by a removable storage medium 105. (3) Provided from a server such as a program server.

[0093] It should be noted that the hardware configuration of the computer for realizing the control device described with reference to FIG. 15 is merely an example, and the embodiment is not limited to this.

[0094] The above-mentioned embodiments are specific examples shown to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Modifications of the above-mentioned embodiments and alternative forms that replace the above-mentioned embodiments may be included. That is, the components of each embodiment can be modified within the scope of the embodiment. In addition, new embodiments can be implemented by appropriately combining multiple components disclosed in one or more embodiments. In addition, some components may be deleted from the components shown in each embodiment, or some components may be added to the components shown in the embodiment. Furthermore, the processing procedures shown in each embodiment may be performed in a different order as long as there is no contradiction.

Claims

1. A microscope and an imaging unit that images a subject through the microscope; a scene recognition unit that performs scene recognition using a machine learning model that has learned a plurality of scenes based on the image of the subject obtained by the imaging unit; a scene determination unit that performs scene determination based on a recognition result of the scene recognition unit; a stabilization unit that temporally stabilizes a determination result of the scene determination unit; a setting unit that changes a setting of the imaging unit in response to the determination result of the scene determination unit. A microscope system comprising:

2. 2. The microscope system according to claim 1, the stabilization unit includes a scene recognition pre-processing unit that performs pre-processing on the image of the subject; the scene recognition preprocessing unit performs an inverse transformation of the image processing performed by the imaging unit on the image of the subject; The scene recognition unit performs the scene recognition based on an inversely transformed image obtained by the inverse transformation in the scene recognition preprocessing unit. A microscope system comprising:

3. 3. The microscope system according to claim 2, the stabilization unit further includes a scene recognition post-processing unit that performs post-processing on the recognition result of the scene recognition unit, the scene recognition post-processing unit calculates a cumulative recognition result, which is a cumulative recognition result for a period corresponding to the plurality of frames, based on a plurality of recognition results of the scene recognition unit corresponding to the plurality of frames; The scene determination unit performs the scene determination based on the accumulated recognition result. A microscope system comprising:

4. 4. The microscope system according to claim 3, The scene determination unit When the evaluation value of the scene having the highest evaluation in the accumulated recognition result does not exceed a threshold, the scene determination is reserved. A microscope system comprising:

5. 5. The microscope system according to claim 4, When the evaluation value of the highest rated scene exceeds the threshold, the scene determination unit outputs the scene corresponding to the evaluation value as the determination result. A microscope system comprising:

6. 5. The microscope system according to claim 4, The scene determination unit When the evaluation value of the highest-rated scene exceeds the threshold value and the scene corresponding to the evaluation value is not a background, the scene corresponding to the evaluation value is output as the determination result; When the evaluation value of the highest rated scene exceeds the threshold and the scene corresponding to the evaluation value is a background, the scene determination is withheld. A microscope system comprising:

7. 5. The microscope system according to claim 4, When the evaluation value of the highest-rated scene exceeds the threshold, the scene determination unit determines whether to reserve the scene determination or output the scene corresponding to the evaluation value as the determination result based on time-series data of the scene corresponding to the evaluation value and the previously output determination result. A microscope system comprising:

8. 5. The microscope system according to claim 4, When the evaluation value of the highest-rated scene exceeds the threshold value, the scene determination unit performs the scene determination based on the scene corresponding to the evaluation value and time-series data of the determination result that has already been output, and outputs the determined scene as the determination result. A microscope system comprising:

9. 9. The microscope system according to claim 4, the stabilization unit includes a threshold adjustment unit that adjusts the threshold; The threshold adjustment unit is increasing the threshold value when the scene with the highest evaluation in the cumulative recognition result is different from the scene corresponding to the judgment result output immediately before; When the scene with the highest evaluation in the accumulated recognition result is the same as the scene corresponding to the judgment result output immediately before, the threshold value is reset to an initial value. A microscope system comprising:

10. 10. The microscope system according to claim 9, The threshold adjustment unit resets the threshold to the initial value when a movement of the subject is detected. A microscope system comprising:

11. 4. The microscope system according to claim 3, The scene recognition post-processing unit includes: setting a predetermined number of frame periods from the timing of the setting change made by the setting unit or the timing when the setting change is scheduled to be made by the setting unit as a dead band; The recognition results of the scene recognition unit corresponding to the frames in the dead zone are not reflected in the cumulative recognition result. A microscope system comprising:

12. 2. The microscope system according to claim 1, The setting unit is A restriction setting for restricting changes to settings of the imaging unit, When the restriction setting is enabled, after changing the setting of the imaging unit according to the determination result of the scene determination unit, the change of the setting is prohibited. A microscope system comprising:

13. 2. The microscope system according to claim 1, The setting unit further changes settings of the microscope in accordance with the determination result of the scene determination unit. A microscope system comprising:

14. 2. The microscope system according to claim 1, The imaging unit includes: a filter insertion / removal section capable of inserting and removing an infrared cut filter into and from the optical path; a filter detection unit that detects whether the infrared cut filter is inserted in the optical path, The scene recognition unit switches a machine learning model used for the scene recognition based on a detection result of the filter detection unit. A microscope system comprising:

15. The microscope system according to claim 14, further comprising: a notification unit that outputs an alert based on the determination result of the scene determination unit and the detection result of the filter detection unit; A microscope system comprising:

Images