Microscope system, setting changing method, storage medium, and digital camera

Abstract

The invention provides a microscope system, a setting changing method, a storage medium, and a digital camera. The present invention addresses the problem of stably performing appropriate device settings in accordance with a scene in a microscope system. A microscope system (1) is provided with: a microscope (10); a digital camera (20) that captures an image of the specimen (S) via the microscope (10); a scene recognition unit (31) that recognizes a plurality of scenes using the learned machine learning model on the basis of the image of the specimen (S) obtained by the digital camera (20); a scene determination unit (32) that performs scene determination on the basis of the recognition result of the scene recognition unit (31); a stabilization unit (40) that stabilizes the determination result of the scene determination unit (32) over time; and a setting unit (33) that changes the setting of the digital camera (20) in accordance with the determination result of the scene determination unit (32).

Description

Technical Field

[0001] The disclosure of this specification relates to a microscope system, a setting change method, a storage medium, and a digital camera. Background Art

[0002] In observation using a microscope system, the optimal device settings differ depending on the scene consisting of the observation method, the subject, or a combination thereof, etc. Therefore, the user of the microscope system is required to appropriately change the device settings according to the scene, but it is not always easy for the user to make the optimal settings according to the scene.

[0003] A technique related to such a problem is described in, for example, Patent Document 1. Patent Document 1 describes a technique for estimating (inferring) a user's desired imaging parameter from a real-time image by utilizing artificial intelligence (learning model).

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent No. 6960445 Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, even with the technology described in Patent Document 1, it is not easy to automatically perform device settings corresponding to the scene with high accuracy. This is because images obtained by various device settings may be used in the real-time images used for inference of the shooting parameters, and a large amount of learning data and time are required to learn all modes in advance. This situation occurs particularly significantly in devices such as microscope systems that have advanced functions and can perform multiple device settings.

[0009] In addition, the disadvantages of using a learning model with low inference accuracy are not limited to not performing appropriate device settings corresponding to the scene. Although the same scene is photographed, due to the low inference accuracy of the learning model, the inference results sometimes fluctuate, which may cause other problems such as frequent switching of device settings and hindering observation work.

[0010] In view of the above-mentioned actual situation, an object of one aspect of the present invention is to provide a technology for stably performing appropriate device settings according to the scene in a microscope system.

[0011] Means for solving problems

[0012] A microscope system according to one embodiment of the present invention comprises: a microscope; an imaging unit which captures an image of a subject through the microscope; a scene recognition unit which performs scene recognition on a plurality of scenes using a learned machine learning model based on an image of the subject obtained by the imaging unit; a scene determination unit which performs scene determination based on a recognition result of the scene recognition unit; a stabilization unit which makes the determination result of the scene determination unit stable over time; and a setting unit which changes the setting of the imaging unit based on the determination result of the scene determination unit.

[0013] A setting change method according to one embodiment of the present invention is performed by a computer as follows: based on an image of a subject obtained by a camera unit, a learned machine learning model is used to perform scene recognition on multiple scenes, the camera unit photographs the subject through a microscope, and scene judgment is performed based on the result of the scene recognition, so that the result of the scene judgment is stable in time, and the setting of the camera unit is changed based on the result of the scene judgment.

[0014] A storage medium according to one embodiment of the present invention is a computer-readable storage medium storing a program that causes a computer to execute the following processing:

[0015] Based on the image of the subject obtained by the camera unit, scene recognition is performed on multiple scenes using a learned machine learning model. The camera unit photographs the subject through a microscope, and scene judgment is performed based on the result of the scene recognition, so that the result of the scene judgment is stabilized in time, and the setting of the camera unit is changed based on the result of the scene judgment.

[0016] A digital camera according to one embodiment of the present invention comprises: an image sensor that captures an image of a subject through a microscope; a scene recognition unit that performs scene recognition on a plurality of scenes using a learned machine learning model based on an image of the subject generated by processing an output from the image sensor; a scene determination unit that performs scene determination based on a recognition result of the scene recognition unit; a stabilization unit that stabilizes the determination result of the scene determination unit over time; and a setting unit that changes the setting of the digital camera based on the determination result of the scene determination unit.

[0017] Effects of the Invention

[0018] According to the above aspect, it is possible to provide a technique for stably performing appropriate device settings according to the scene in a microscope system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a diagram illustrating the configuration of a microscope system according to one embodiment of the present invention.

[0020] Figure 2This is a diagram illustrating the structure of a stabilization unit according to one embodiment of the present invention.

[0021] Figure 3 This is a diagram showing an example of a flowchart of the automatic setting process performed by the control device according to the first embodiment of the present invention.

[0022] Figure 4 This is a diagram for explaining the timing of performing scene determination in the control device according to the first embodiment of the present invention.

[0023] Figure 5 This is a diagram showing an example of a flowchart of a frame process performed by the control device according to the first embodiment of the present invention.

[0024] Figure 6 This is a diagram showing an example of a flowchart of a determination process performed by the control device according to the first embodiment of the present invention.

[0025] Figure 7 This is a diagram showing an example of a flowchart of a frame process performed by the control device according to the second embodiment of the present invention.

[0026] Figure 8 This is a diagram showing an example of a flowchart of a determination process performed by a control device according to the third embodiment of the present invention.

[0027] Fig. 9 This is a diagram showing an example of a flowchart of a determination process performed by a control device according to a fourth embodiment of the present invention.

[0028] Fig.10 This is a diagram showing an example of a flowchart of a determination process performed by a control device according to a fifth embodiment of the present invention.

[0029] Fig.11 This is a diagram showing an example of a flowchart of a determination process performed by a control device according to the sixth embodiment of the present invention.

[0030] Fig.12 It is a diagram illustrating the configuration of a microscope system according to a seventh embodiment of the present invention.

[0031] Fig.13 This is a diagram showing an example of a flowchart of a determination process performed by a control device according to the eighth embodiment of the present invention.

[0032] Fig.14 It is a diagram illustrating the configuration of a microscope system according to a ninth embodiment of the present invention.

[0033] Fig.15 This is a diagram illustrating a hardware configuration of a computer for realizing a control device. DETAILED DESCRIPTION

[0034] Figure 1 This is a diagram illustrating the configuration of a microscope system according to one embodiment of the present invention. Figure 1 The microscope system 1 shown includes a microscope 10 , a digital camera 20 attached to the microscope 10 , and a control device 30 that controls the microscope 10 and the digital camera 20 .

[0035] The microscope 10 is an arbitrary microscope equipped with a digital camera 20 , and may be a microscope capable of handling a plurality of observation methods such as bright field, dark field, fluorescence observation, polarization observation, and differential interference observation, for example.

[0036] The digital camera 20 is an example of an imaging unit of the microscope system 1. The digital camera 20 captures an image of a specimen S as a subject via the microscope 10 to obtain an image of the specimen S. The digital camera 20 includes an image sensor such as a CCD or a CMOS. The image of the specimen S obtained by the digital camera 20 is output to the control device 30.

[0037] The digital camera 20 performs predetermined image processing on the image of the specimen S to adjust brightness and color, and then outputs the image to the control device 30. The image processing performed by the digital camera 20 is not particularly limited, and includes, for example, white balance correction, color matrix correction, gamma correction, and the like.

[0038] The control device 30 changes the settings of the microscope system 1 based on the image acquired by the digital camera 20. As a configuration associated with the setting change, Figure 1 As shown, the control device 30 includes a scene recognition unit 31 , a scene determination unit 32 , a setting unit 33 , and a stabilization unit 40 .

[0039] The scene recognition unit 31 performs scene recognition on a plurality of scenes using a learned machine learning model based on an image of the specimen S obtained by the digital camera 20. The scene determination unit 32 performs scene determination based on the recognition result of the scene recognition unit 31. The setting unit 33 changes the setting of the digital camera 20 according to the determination result of the scene determination unit 32. The setting unit 33 may also change the setting of the microscope 10 according to the determination result of the scene determination unit 32. The stabilization unit 40 stabilizes the determination result of the scene determination unit 32 over time. For details about the stabilization unit 40, see Figure 2 This will be described in detail later.

[0040] The image input to the scene recognition unit 31 is, for example, a real-time image (moving image) before actual photography (still image photography) during observation by the microscope system 1. For example, the scene recognition unit 31 recognizes the scene of each frame of the input real-time image.

[0041] The scene recognized by the scene recognition unit 31 is associated with, for example, the type of specimen S, the observation method, or a combination thereof. However, the scene may be any scene that has a relatively strong correlation with the setting of the microscope system 1 and is not necessarily limited to the subject and the observation method.

[0042] The machine learning model used in the scene recognition unit 31 is, for example, a machine learning model that has learned scenes such as the type of specimen S, observation methods, and combinations thereof through deep learning, and the machine learning model estimates the type of specimen S, observation methods, and combinations thereof, i.e., the scene. More specifically, the machine learning model used in the scene recognition unit 31 uses, for example, object detection algorithms such as YOLO (You Only Look Once). The output from the scene recognition unit 31, i.e., the recognition result of the scene recognition unit 31, is, for example, a result obtained by summing up the confidence of each category of one or more bounding boxes detected in the image for each category (hereinafter referred to as the total confidence). That is, it includes the same number of total confidences as the number of categories. In addition, the categories classified by the machine learning model correspond to the scenes respectively.

[0043] The scene determination in the scene determination unit 32 is performed by comparing the recognition result of the scene recognition unit 31 (or the cumulative recognition result described later) with a threshold. For the threshold, if the total confidence (or the cumulative total confidence described later) is above the threshold, a value that can be considered to be sufficiently reliable for the recognition result of the scene recognition unit 31 (or the cumulative total confidence described later) is set as an initial value in advance. On this basis, the scene determination unit 32 is configured to suspend scene determination when the total confidence (or the cumulative total confidence described later) is less than the threshold. In this way, scene determination with a higher accuracy than the scene recognition of the scene recognition unit 31 alone can be performed.

[0044] A parameter list corresponding to the optimal setting for each scene is pre-stored in the control device 30. The setting unit 33 reads the parameter list corresponding to the scene determined by the scene determination unit 32, and sets the digital camera 20 and the microscope 10. In addition, an example of the setting of the digital camera 20 performed by the setting unit 33 is not particularly limited, and is, for example, the setting of the exposure time. In addition, an example of the setting of the microscope 10 performed by the setting unit 33 is not particularly limited, and is, for example, the insertion and removal of an optical element (such as a polarizer, an analyzer, etc.) according to the observation method.

[0045] Thus, the control device 30 can perform scene determination with a higher accuracy than the machine learning model alone used for scene determination by including the scene recognition unit 31 and the scene determination unit 32, and thus can automatically perform the best setting corresponding to the scene for the microscope system 1. Furthermore, the control device 30 can further improve the scene determination accuracy and suppress fluctuations by including the stabilization unit 40 that stabilizes the determination result over time. Thus, the adverse effects of frequent setting changes on the observation operation can be avoided.

[0046] Figure 2 FIG. 1 is a diagram illustrating the structure of a stabilization unit according to an embodiment of the present invention. Figure 2 , focusing on the stabilization unit 40, the setting change processing performed by the control device 30 is described in further detail.

[0047] like Figure 2 As shown, the stabilization unit 40 includes: a scene recognition pre-processing unit 41, which pre-processes the image obtained by the digital camera 20; a scene recognition post-processing unit 42, which post-processes the recognition result of the scene recognition unit 31; and a threshold adjustment unit 43, which adjusts the threshold used in the scene determination unit 32.

[0048] The scene recognition pre-processing unit 41 is configured to perform an inverse transformation on the image of the image processing performed by the digital camera 20. The image processing performed by the digital camera 20 is performed for various purposes, such as preventing whitening, improving color reproducibility, and reproducing the same natural appearance as when observed by the human eye. These image processings not only have a relatively large impact on the image, but also depend on the device itself and are not necessarily the best device settings. Therefore, the predictability of the learning time is low, and it is not easy to learn the impact in advance. Therefore, if sufficient learning is not performed on the images obtained by the device or on the images obtained by the device with the current settings, the recognition accuracy in the scene recognition unit 31 may be reduced. The scene recognition pre-processing unit 41 takes this into consideration.

[0049] The scene recognition pre-processing unit 41 can eliminate the influence of the image processing performed by the digital camera 20 by performing the inverse transformation of the image processing performed by the digital camera 20. In addition, the content of the image processing performed by the digital camera 20 can be determined based on the recognition information of the digital camera 20 and the settings of the digital camera 20. The scene recognition unit 31 performs scene recognition based on the image obtained by the inverse transformation in the scene recognition pre-processing unit 41 (hereinafter referred to as the inverse transformation image), thereby suppressing the reduction in the recognition accuracy of the machine learning model caused by the image processing performed by the digital camera 20.

[0050] In the above, it is explained that the scene recognition unit 31 performs inverse transformation when performing inference using the learned machine learning model, but when learning the machine learning model used in the scene recognition unit 31, it is also preferable to perform inverse transformation of the image processing performed by the digital camera that acquired the learning image. However, it is not necessary to provide the information (settings, etc.) of the digital camera when the image was acquired as learning data together with the learning image (the image after image processing). Therefore, in the learning of the machine learning model, the image after the image processing performed by the digital camera is inversely transformed and used as the learning image, the image without inverse transformation of the image processing performed by the digital camera, the image output from the digital camera itself, or both of them can be used as the learning image.

[0051] As described above, the scene recognition pre-processing unit 41 improves the recognition result (scene recognition result) of the scene recognition unit 31 itself by performing pre-processing before the recognition process performed by the scene recognition unit 31. In contrast, the scene recognition post-processing unit 42 improves the determination result (scene determination result) based on the recognition result, rather than the recognition result (scene recognition result) of the scene recognition unit 31 itself, by performing post-processing after the recognition process performed by the scene recognition unit 31.

[0052] The scene recognition post-processing unit 42 is configured to calculate the cumulative recognition results (hereinafter referred to as the cumulative recognition results) in the period corresponding to the plurality of frames based on the plurality of recognition results of the scene recognition unit 31 corresponding to the plurality of frames. In real-time shooting where the image changes momentarily due to the movement of the specimen S, changes in the environment (e.g., focus, lighting, etc.), etc., the scene recognition results may change over time. The scene recognition post-processing unit 42 takes this into consideration.

[0053] The scene recognition post-processing unit 42 calculates the cumulative total confidence of each category by, for example, accumulating the total confidence of each category in a plurality of frames, calculates the cumulative recognition result based on the recognition results of a plurality of frames, and outputs the cumulative recognition result to the scene determination unit 32. That is, the cumulative recognition result includes the cumulative total confidence of each category. Thus, it is possible to avoid that the erroneous recognition results generated irregularly are immediately reflected in the settings of the microscope system 1. In addition, the recognition results of each frame are averaged by the accumulation processing. Therefore, by performing scene determination based on the cumulative recognition results calculated by the scene recognition post-processing unit 42 by the scene determination unit 32, the influence of the jitter of scene recognition generated during real-time shooting can be suppressed, and the determination result can be stabilized in time.

[0054] The number of frames to be accumulated in the scene recognition post-processing unit 42 is not particularly limited and may be predetermined. For example, scene determination may be performed based on the accumulated recognition results corresponding to 5 frames, and the settings may be updated. Alternatively, the number of frames may not be predetermined, and scene determination may be performed when the accumulated recognition results satisfy a predetermined condition, and the settings may be changed based on the determination results.

[0055] The scene recognition post-processing unit 42 may also be configured to set a predetermined number of frame periods (e.g., a period of 5 frames) from the time when the setting section 33 makes a setting change as a dead zone, and not reflect the recognition results of the scene recognition unit 31 corresponding to the frames in the dead zone in the accumulated recognition results. In addition, the scene recognition post-processing unit 42 may also set a predetermined number of frame periods from the time when the setting change is scheduled (e.g., every 5 frames) as a dead zone, regardless of whether the setting change is made. By setting the dead zone by the scene recognition post-processing unit 42, the recognition results of the scene obtained by the microscope system 1 in an unstable state will not be reflected in the judgment of the scene. Therefore, the judgment result can be stabilized in time.

[0056] As described above, the scene recognition pre-processing unit 41 and the scene recognition post-processing unit 42 improve the scene determination result by directly or indirectly improving the cumulative recognition result of the scene. In contrast, the threshold adjustment unit 43 improves the scene determination result by adjusting the threshold value compared with the cumulative recognition result.

[0057] The threshold adjustment unit 43 is configured to increase the threshold value when the scene with the highest evaluation in the cumulative recognition result is different from the scene corresponding to the determination result output immediately before. In addition, the threshold adjustment unit 43 is configured to reset the threshold value to the initial value when the scene with the highest evaluation in the cumulative recognition result is the same as the scene corresponding to the determination result output immediately before. Thus, when it is determined that the scene has been changed, the judgment is made with a stricter standard than when it is determined that the scene has not been changed. In a situation where an erroneous scene judgment is likely to occur, the cumulative recognition result is less likely to become larger than when the scene judgment is generally performed correctly. Therefore, by increasing the threshold value for judgment, erroneous judgments can be prevented in advance. In addition, in the above-mentioned structure, when the scene change occurs continuously, the threshold value will gradually increase, so the occurrence of fluctuations can be effectively suppressed. Moreover, when it is determined that the scene has not been changed, the threshold value is reset and the judgment is made with the initial value, so that it is possible to avoid the situation where the threshold value of the scene judgment is maintained to avoid erroneous scene judgment and the judgment is unnecessary.

[0058] The threshold adjustment unit 43 may also be configured to reset the threshold to an initial value when the movement of the specimen S is detected, regardless of whether the scene with the highest evaluation in the cumulative recognition result is the same as the scene corresponding to the judgment result output immediately before. When the movement of the specimen S is detected, the possibility of scene changes such as replacement of the specimen S and change of the observation method is higher than usual. Therefore, in such a case, the threshold adjustment unit 43 resets the threshold, and the benchmark for determining that the scene has been changed will not be excessively increased, and the scene determination unit 32 may also perform scene determination based on the same benchmark as the case where the scene has not been changed. According to this structure, it is also possible to avoid unnecessary suspension of determination.

[0059] In addition, the detection method of the movement of the specimen S is not particularly limited. The threshold adjustment unit 43 may detect the movement of the specimen S based on an image, for example, the movement of the specimen S may be detected based on a motion vector calculated from a real-time image. In addition, the threshold adjustment unit 43 may detect the movement of the specimen S based on an output from a sensor provided on the stage of the microscope 10. Furthermore, the threshold adjustment unit 43 may detect the movement of the specimen S based on a control signal (for example, a signal for controlling the stage) output from the control device 30 to the microscope 10.

[0060] As described above, the microscope system 1 can accurately determine the scene while avoiding fluctuations in the determination results of the scene. Therefore, according to the microscope system 1, it is possible to stably perform appropriate device settings corresponding to the scene.

[0061] <First Embodiment>

[0062] Figure 3 This is a diagram showing an example of a flowchart of the automatic setting process performed by the control device of the present embodiment. Figure 4 This is a diagram for explaining the timing of performing scene determination in the control device of this embodiment. Figure 5 This is a diagram showing an example of a flowchart of frame processing performed by the control device according to the present embodiment. Figure 6 FIG. 2 is a diagram showing an example of a flowchart of a determination process performed by the control device of the present embodiment. Figures 3 to 6 , the automatic setting process performed by the control device 30 will be specifically described. In addition, the structure of the microscope system of this embodiment is the same as that of the microscope system 1, so the components of the microscope system of this embodiment are referred to with the same reference numerals as those of the components of the microscope system 1.

[0063] When the microscope system of this embodiment starts real-time photography, the processor of the control device 30 executes a predetermined program to perform Figure 3 In addition, here, as shown in Figure 4As shown, the following case is described as an example: the evaluation target frame and the dead zone frame are set to be repeated for 5 frames each, and after the 5 evaluation target frames are completed, the scene determination is performed to update the setting of the microscope system 1.

[0064] First, the control device 30 performs a real-time image processing on each frame of the real-time image obtained by the digital camera 20 via the microscope 10. Figure 5 The frame processing shown (step S10).

[0065] In the frame processing of step S10, first, the control device 30 obtains the image of the latest frame of the real-time image (step S11). Next, the control device 30 calculates a motion vector based on the image obtained in step S11 and the image used as a reference (for example, the image of the first evaluation target frame), and determines whether the specimen S moves (step S12). When it is determined that the specimen S is moving (step S12: Yes), the threshold used in the scene determination unit 32 is reset and initialized (step S13). When it is determined that the specimen S is not moving (step S12: No), the processing of step S13 is skipped.

[0066] Then, the control device 30 determines whether the period of the current frame is a dead zone (step S14). When it is determined that the period of the current frame is a dead zone (step S14: Yes), the process ends. Figure 5 Frame processing shown.

[0067] On the other hand, if it is determined that the period of the current frame is not a dead zone but an evaluation target period (step S14: No), the control device 30 performs an inverse transformation of the image processing performed by the digital camera 20 on the image acquired in step S11 to generate an inversely transformed image (step S15).

[0068] When the inverse transformation image is generated, the control device 30 inputs the inverse transformation image into the machine learning model, infers the scene (step S16), and calculates the evaluation value of each category as the recognition result (step S17). Specifically, the control device 30 calculates the total confidence obtained by summing the confidence of each category calculated for multiple bounding boxes for each category, that is, the evaluation value of each category as the recognition result. Finally, the control device 30 accumulates the evaluation value of each category of the current frame calculated in step S17 to the cumulative evaluation value of each category calculated in the previous frame, calculates the updated cumulative evaluation value of each category as the cumulative recognition result (step S18), and ends. Figure 5 Frame processing shown.

[0069] exist Figure 5 When the frame processing shown in FIG. 1 is performed, the control device 30 determines whether to perform scene determination (step S20). Figure 5As shown, the control device 30 determines to perform scene determination when five evaluation target frames are generated continuously. Therefore, the control device 30 repeatedly performs frame processing on the five evaluation target frames and calculates a cumulative evaluation value of the recognition results accumulated from the five frames.

[0070] When it is determined that scene determination is to be performed (step S20: Yes), the control device 30 performs Figure 6 The determination process shown (step S30).

[0071] In the determination process of step S30, first, the control device 30 determines the scene (category) with the highest evaluation in the cumulative recognition results (step S31). Next, the control device 30 determines whether the scene has changed (step S32). Here, the control device 30 determines whether the scene has changed based on whether the scene determined in step S31 is different from the scene determined in the immediately previous determination process. If the control device 30 determines that the scene has changed, it increases the threshold used in step S35 (step S33), and if it determines that the scene has not changed, it resets the threshold used in step S35 to the initial value (step S34).

[0072] Then, the cumulative evaluation value (maximum cumulative evaluation value) corresponding to the scene determined in step S31 included in the cumulative recognition result is compared with the threshold value (step S35). If it is determined that the maximum cumulative evaluation value exceeds the threshold value, the control device 30 outputs the scene corresponding to the maximum cumulative evaluation value as the determination result (step S36), and the process ends. Figure 6 On the other hand, if it is determined that the maximum cumulative evaluation value does not exceed the threshold value, the control device 30 puts the scene determination (step S37) on hold and ends. Figure 6 The determination process shown.

[0073] exist Figure 6 When the determination process shown is completed, the control device 30 determines whether to perform setting (step S40). When it is determined in step S40 that the determination result in the determination process of step S30 is output, the control device 30 performs setting processing corresponding to the determination result (step S50). Specifically, the control device 30 reads a parameter list corresponding to the determination result (scene), and updates the setting of the digital camera 20 (for example, the setting of the exposure time) according to the parameter list. On the other hand, when the scene determination is shelved in the determination process of step S30, the control device 30 determines that the setting is not performed (step S40: No), and skips the setting processing of step S50.

[0074] Finally, the control device 30 determines whether to end Figure 3The control device 30 terminates the automatic setting process when, for example, it detects an end instruction input from the user (step S60: Yes), and returns to step S10 to repeat the automatic setting process if there is no end instruction (step S60: No).

[0075] According to the microscope system of this embodiment, by performing Figure 3 The automatic setting process shown can determine the scene with high accuracy while avoiding fluctuations in the scene determination results. Therefore, it is possible to stably perform appropriate device settings corresponding to the scene, and it is possible to automate the setting work and reduce the burden on the user.

[0076] <Second Embodiment>

[0077] Figure 7 1 is a diagram showing an example of a flowchart of frame processing performed by the control device of the present embodiment. The structure of the microscope system of the present embodiment is the same as that of the microscope system 1. The automatic setting processing performed by the control device 30 of the present embodiment is different from the automatic setting processing performed by the control device 30 of the first embodiment in that Figure 5 The frame processing shown includes Figure 7 Frame processing shown.

[0078] Figure 7 The frame processing shown includes the processing of step S111 to step S118, and the processing of step S111 to step S118 is the same as the composition. Figure 5 The processes of step S11 to step S18 of the frame processing shown correspond to each other. Figure 7 The frame processing shown is similar to Figure 5 The frame processing shown is different in that, when it is determined in step S112 that the specimen S has not moved (step S112 : No), the subsequent processing is not performed and the frame processing is terminated.

[0079] The microscope system according to the present embodiment can also obtain the same effect as the microscope system 1 according to the first embodiment. In addition, in the microscope system of the present embodiment, unless the movement of the sample S is detected, the scene recognition process is not performed and the cumulative evaluation value is not updated. Therefore, the cumulative evaluation value does not exceed the threshold value, the scene determination result is not output, and the setting is not changed. Thus, the setting during the period when the movement of the specimen S is not detected is fixed, so that the fluctuation of the determination result in the same scene can be reliably prevented.

[0080] <Third Embodiment>

[0081] Figure 81 is a diagram showing an example of a flowchart of the determination process performed by the control device of the present embodiment. The structure of the microscope system of the present embodiment is the same as that of the microscope system 1. The automatic setting process performed by the control device 30 of the present embodiment is different from the automatic setting process performed by the control device 30 of the first embodiment in that Figure 6 The determination process shown includes Figure 8 The determination process shown.

[0082] Figure 8 The determination process shown in the figure includes the process of step S138 in addition to the process of step S131 to step S137. The process of step S131 to step S137 is similar to the process of step S138. Figure 6 The processing of step S31 to step S37 of the determination process shown in FIG. Figure 8 In the determination process shown, when the control device 30 determines that the maximum cumulative evaluation value exceeds the threshold value (step S135: Yes), it further determines whether the scene corresponding to the highest evaluation is the background (step S138).

[0083] When it is determined that the scene corresponding to the highest evaluation value is not the background (step S138: No), the control device 30 (scene determination unit 32) outputs the scene corresponding to the maximum cumulative evaluation value as the determination result (step S136), and the process ends. Figure 8 On the other hand, when it is determined that the scene corresponding to the highest evaluation is the background (step S138: Yes), the control device 30 (scene determination unit 32) puts aside the scene determination (step S137) in the same manner as when it is determined that the maximum cumulative evaluation value does not exceed the threshold value (step S135: No), and ends the process. Figure 8 The determination process shown.

[0084] The microscope system according to this embodiment can also obtain the same effects as the microscope system 1 according to the first embodiment. In addition, in the microscope system according to this embodiment, when the current scene is estimated to be the background based on the cumulative evaluation value, the scene determination is put on hold. This can avoid unnecessary setting changes in situations where it is not desirable to change the setting or there is no point in changing the setting, such as when the field of view deviates from the specimen S during the search for the desired part of the specimen S or when only the background is captured during the replacement of the specimen S.

[0085] <Fourth Embodiment>

[0086] Fig. 91 is a diagram showing an example of a flowchart of the determination process performed by the control device of the present embodiment. The structure of the microscope system of the present embodiment is the same as that of the microscope system 1. The automatic setting process performed by the control device 30 of the present embodiment is different from the automatic setting process performed by the control device 30 of the first embodiment in that it includes Fig. 9 The determination process shown is replaced by Figure 6 The determination process shown.

[0087] Fig. 9 The determination process shown in the figure includes the processes of step S139 and step S140 in addition to the processes of step S131 to step S137. The processes of step S131 to step S137 are similar to the processes of step S139 and step S140. Figure 6 The processing of step S31 to step S37 of the determination process shown in FIG. Fig. 9 In the judgment process shown, when the control device 30 judges that the maximum cumulative evaluation value exceeds the threshold value (step S135: Yes), it further obtains the time series data of the judgment result that has been output (step S139), and judges whether there is fluctuation between specific scenes (step S140). Here, the fluctuation between specific scenes refers to fluctuation between scenes that do not usually occur, such as fluctuation between scenes related to biological slide specimens and scenes related to industrial printed circuit boards.

[0088] When it is determined that there is no fluctuation between specific scenes (step S140: No), the control device 30 (scene determination unit 32) outputs the scene corresponding to the maximum cumulative evaluation value as the determination result (step S136), and the process ends. Fig. 9 On the other hand, if it is determined that there is fluctuation between specific scenes (step S140: Yes), the control device 30 (scene determination unit 32) puts aside the scene determination (step S137) in the same manner as when it is determined that the maximum cumulative evaluation value does not exceed the threshold value (step S135: No), and ends the process. Fig. 9 The determination process shown.

[0089] That is, in this embodiment, in the processing after step S135, when the evaluation value of the highest-evaluated scene exceeds the threshold, the control device 30 decides whether to put on hold the scene judgment or output the scene corresponding to the evaluation value as the judgment result based on the scene corresponding to the evaluation value and the time series data of the judgment result that has been output.

[0090] The microscope system according to this embodiment can also obtain the same effect as the microscope system 1 according to the first embodiment. In addition, in the microscope system according to this embodiment, even when the cumulative evaluation value exceeds the threshold value, when a state transition that is not likely to occur and is registered in advance is detected, the scene determination is suspended. Thus, for example, it is possible to avoid setting changes corresponding to a state transition that is not likely to occur and prevent fluctuations caused by misrecognition.

[0091] <Fifth Embodiment>

[0092] Fig.10 1 is a diagram showing an example of a flowchart of the determination process performed by the control device of the present embodiment. The structure of the microscope system of the present embodiment is the same as that of the microscope system 1. The automatic setting process performed by the control device 30 of the present embodiment is different from the automatic setting process performed by the control device 30 of the first embodiment in that it includes Fig.10 The determination process shown is replaced by Figure 6 The determination process shown.

[0093] Fig.10 The determination process shown in the figure includes the processes of step S139 and step S141 in addition to the processes of step S131 to step S137. The processes of step S131 to step S137 are similar to the processes of step S141. Figure 6 The processing of step S31 to step S37 of the determination process shown in FIG. Fig.10 In the determination process shown, when the control device 30 determines that the maximum cumulative evaluation value exceeds the threshold value (step S135 : Yes), it further acquires the time series data of the determination result that has been output (step S139 ).

[0094] Then, the control device 30 performs scene determination based on the scene determined in step S131 and the time series data of the determination result obtained in step S139 (step S141). In addition, in step S141, the control device 30 uses the learned RNN (Recurrent Neural Network) machine learning model, which is obtained by deep learning in a way of inferring scenes based on the time series data of the scenes. Finally, the control device 30 (scene determination unit 32) outputs the scene determined in step S141 as the determination result (step S136), and ends. Fig.10 The determination process shown.

[0095] The microscope system of this embodiment can also obtain the same effect as the microscope system 1 of the first embodiment. In addition, in the microscope system of this embodiment, the scene corresponding to the highest evaluation is not directly used as the judgment result, but the scene is judged in consideration of the time series data. Thus, the scene can be judged with higher accuracy.

[0096] <Sixth Embodiment>

[0097] Fig.11 1 is a diagram showing an example of a flowchart of the determination process performed by the control device of the present embodiment. The structure of the microscope system of the present embodiment is the same as that of the microscope system 1. The automatic setting process performed by the control device 30 of the present embodiment is different from the automatic setting process performed by the control device 30 of the first embodiment in that Figure 6 The determination process shown includes Fig.11 The determination process shown.

[0098] Fig.11 The determination process shown in the figure includes the process of step S142 in addition to the process of step S131 to step S137. The process of step S131 to step S137 is similar to the process of step S143. Figure 6 The processing of step S31 to step S37 of the determination process shown in FIG. Fig.11 In the judgment process shown, when the control device 30 determines that the maximum cumulative evaluation value exceeds the threshold value (step S135: Yes), it determines whether the restriction setting (one-time function) that limits the setting change of the digital camera 20 to only one time is set to be effective and the setting change has been performed once (step S142).

[0099] If the primary function is not enabled or the setting change is not performed (step S142: No), the control device 30 (scene determination unit 32) outputs the scene corresponding to the maximum cumulative evaluation value as the determination result (step S136), and the process ends. Fig.11 On the other hand, when the one-time function is enabled and the setting has been changed (step S142: Yes), the control device 30 (scene determination unit 32) puts the scene determination on hold (step S137) in the same manner as when it is determined that the maximum cumulative evaluation value does not exceed the threshold value (step S135: No), and ends the process. Fig.11 The determination process shown.

[0100] According to the microscope system of this embodiment, the same effects as those of the microscope system 1 of the first embodiment can be obtained. In addition, in the microscope system of this embodiment, when the one-time function (limit setting) is set to be effective, the control device 30 (setting unit 33) prohibits the subsequent setting changes after the setting of the digital camera 20 is changed according to the determination result of the scene determination unit 32. Thus, excessive setting changes are restricted, so that the generation of fluctuations can be prevented.

[0101] <Seventh Embodiment>

[0102] Fig.12This is a diagram illustrating the configuration of a microscope system according to this embodiment. Fig.12 The microscope system 2 shown is different from the microscope system 1 in that a digital camera 50 capable of inserting and removing an infrared cut filter is provided instead of the digital camera 20, and the scene recognition unit 31 distinguishes and uses different machine learning models depending on whether an infrared cut filter is inserted in the digital camera 50. Other aspects are the same as those of the microscope system 1.

[0103] like Fig.12 As shown, the digital camera 50 includes an image sensor 51 and an image processing unit 52 that processes the output from the image sensor 51 to generate an image. The digital camera 50 also includes: a filter insertion and removal unit 53 that can insert and remove an infrared cut filter in the optical path between the image sensor 51 and the microscope 10; and a filter detection unit 54 that detects whether the infrared cut filter is inserted in the optical path.

[0104] In addition, the structure of the filter insertion and extraction unit 53 for detecting the insertion of the infrared cut filter is not particularly limited. For example, a magnet may be provided on the filter insertion and extraction unit 53. The filter detection unit 54 may also include a Hall sensor, and detects whether the infrared cut filter is inserted into the optical path by detecting whether the magnet provided on the filter insertion and extraction unit 53 is detected.

[0105] The digital camera 50 is configured such that, for example, when the microscope system 1 uses visible light for observation, the filter insertion and extraction unit 53 inserts an infrared cut filter into the optical path in order to improve color reproducibility. In addition, when the microscope system 1 uses near-infrared light for observation, the digital camera 50 removes the infrared cut filter from the optical path in order to improve sensitivity.

[0106] The digital camera 50 outputs the detection result of the filter detection unit 54 to the control device 30. The scene recognition unit 31 of the control device 30 is configured to switch the machine learning model used for scene recognition based on the detection result of the filter detection unit 54. Specifically, when the filter detection unit 54 detects that the infrared cut filter is inserted into the optical path, the scene recognition unit 31 uses the following machine learning model to perform scene recognition, and the machine learning model is obtained by learning the scene by using the image obtained by using the infrared cut filter. On the other hand, when the filter detection unit 54 detects that the infrared cut filter is not inserted into the optical path, the scene recognition unit 31 uses the following machine learning model to perform scene recognition, and the machine learning model is obtained by learning the scene by using the image obtained without using the infrared cut filter.

[0107] The microscope system 2 of this embodiment can also achieve the same effects as the microscope system 1 of the first embodiment. In addition, in the microscope system of this embodiment, the machine learning model is used according to the insertion and removal status of the infrared cut filter, so that more accurate scene judgment can be performed.

[0108] <Eighth Implementation Method>

[0109] Fig.13 1 is a diagram showing an example of a flowchart of the determination process performed by the control device of the present embodiment. The structure of the microscope system of the present embodiment is the same as that of the microscope system 2 of the seventh embodiment. The automatic setting process performed by the control device 30 of the present embodiment is different from the automatic setting process performed by the control device 30 of the seventh embodiment in that Figure 6 The determination process shown includes Fig.13 The determination process shown.

[0110] Fig.13 The determination process shown in the figure includes the processes of step S143 and step S144 in addition to the processes of step S131 to step S137. The processes of step S131 to step S137 are similar to the processes of step S143 and step S144. Figure 6 The processing of step S31 to step S37 of the determination process shown in FIG. Fig.13 In the determination process shown, when the control device 30 outputs the determination result in step S136, it determines whether the combination of the output determination result and the filter detection result of the filter detection unit 54 is appropriate (step S143).

[0111] In addition, the control device 30 pre-registers combinations of the insertion and removal states of the infrared cut filter that can be used in observation and scenes. In step S143, the control device 30 determines whether the combination is appropriate based on whether the combination of the output determination result and the filter detection result of the filter detection unit 54 is included in the pre-registered combination. Examples of the pre-registered combination are not particularly limited, and for example, both the insertion and removal states may be registered for fluorescence observation, phase difference observation, differential interference observation, and polarization observation, and only the insertion state may be registered for bright field observation.

[0112] Then, when the control device 30 determines that the combination is appropriate (step S143: Yes), the control device 30 ends the Fig.13The judgment process shown. On the other hand, when it is judged that the combination is inappropriate (step S143: No), an alarm is output to the notification unit (step S144). Thereby, the notification unit notifies the user that the plug-in and unplug-out state of the infrared cut filter is inappropriate for the scene. In addition, the notification unit can be, for example, a display device of the control device 30, and the control device 30 can display an alarm on the real-time image displayed by the display device (for example, a message "The infrared cut filter is not inserted. Please insert the infrared cut filter during bright field observation.", etc.). That is, the microscope system 2 of the present embodiment has a notification unit that outputs an alarm based on the judgment result of the scene judgment unit 32 and the detection result of the filter detection unit 54.

[0113] According to the microscope system of this embodiment, the same effects as those of the microscope system 1 of the first embodiment can be obtained. In addition, in the microscope system of this embodiment, it is possible to notify the user that the combination of the insertion and removal state of the infrared cut filter and the scene is inappropriate. Thus, by prompting the user to insert or remove the infrared cut filter, it is possible to avoid using the microscope system under inappropriate settings.

[0114] <Ninth Embodiment>

[0115] Fig.14 This is a diagram illustrating the configuration of a microscope system according to this embodiment. Fig.14 The microscope system 3 shown is different from the microscope system 1 in that a digital camera 60 is provided instead of the digital camera 20 and the control device 30. The other points are the same as those of the microscope system 1.

[0116] The digital camera 60 includes a control unit 70 for performing scene recognition, stabilization processing, scene determination, and setting changes in addition to the image sensor 51 and the image processing unit 52. That is, the digital camera 60 has a function of automating setting changes according to the scene (scene recognition unit 31, scene determination unit 32, setting unit 33, stabilization unit 40), similarly to the control device 30 of the microscope system 1.

[0117] According to the microscope system 3 of this embodiment, the same effects as those of the microscope system 1 of the first embodiment can be obtained.

[0118] Fig.15 This is a diagram illustrating a hardware configuration of a computer 100 for realizing the control device or the control unit according to the above-described embodiment. Fig.15The hardware configuration shown includes, for example, a processor 101, a memory 102, a storage device 103, a reader 104, a communication interface 106, and an input / output interface 107. The processor 101, the memory 102, the storage device 103, the reader 104, the communication interface 106, and the input / output interface 107 are connected to each other via, for example, a bus 108.

[0119] The processor 101 is any processor and may include a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), and a GPGPU (General-purpose computing on graphics processing units).

[0120] The processor 101 can also function as the scene recognition unit 31, the scene determination unit 32, the setting unit 33, and the stabilization unit 40 by reading and executing the program stored in the storage device 103. In addition, as an example, the scene recognition unit 31 can be configured using a GPGPU, an ASIC, etc., and the scene determination unit 32, the setting unit 33, and the stabilization unit 40 can be configured using a CPU or an FPGA.

[0121] The memory 102 is, for example, a semiconductor memory, and may include a RAM area and a ROM area. The storage device 103 is, for example, a semiconductor memory such as a hard disk or a flash memory, or an external storage device.

[0122] The reading device 104 accesses the removable storage medium 105 according to the instruction of the processor 101. The removable storage medium 105 is implemented by, for example, a semiconductor device, a medium that inputs and outputs information by magnetic action, a medium that inputs and outputs information by optical action, etc. The semiconductor device is, for example, a USB (Universal Serial Bus) memory. In addition, the medium that inputs and outputs information by magnetic action is, for example, a magnetic disk. The medium that inputs and outputs information by optical action is, for example, a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disk), a Blu-ray Disc, etc. (Blu-ray is a registered trademark).

[0123] The communication interface 106 communicates with other devices (e.g., the microscope 10, the digital camera 20, etc.) according to the instructions of the processor 101. The input / output interface 107 is, for example, an interface between an input device and an output device. The input device is, for example, a device such as a mouse, a keyboard, a foot switch, etc. that receives instructions from a user. The output device is, for example, a display device and an audio device such as a speaker.

[0124] The program executed by the processor 101 is provided to the computer in the following manner, for example.

[0125] (1) Pre-installed in the storage device 103.

[0126] (2) Provided by the removable storage medium 105.

[0127] (3) Provided by a server such as a program server.

[0128] In addition, for reference Fig.15 The hardware configuration of the computer of the control device described above is an example, and the embodiment is not limited to this.

[0129] The above-mentioned embodiments show specific examples for easy understanding of the invention, and the present invention is not limited to these embodiments. It may include a deformation method after the above-mentioned embodiments are deformed and an alternative method replacing the above-mentioned embodiments. That is, each embodiment can deform the constituent elements within the scope of its purpose and scope. In addition, by appropriately combining multiple constituent elements disclosed in more than one embodiment, a new embodiment can be implemented. In addition, several constituent elements can be deleted from the constituent elements shown in each embodiment, or several constituent elements can be added to the constituent elements shown in the embodiment. Moreover, the processing procedures shown in each embodiment can also be performed in a reversed order as long as there is no contradiction.

Claims

1. A microscope system, characterized in that: The microscope system comprises: microscope; An imaging unit for photographing a subject via the microscope; A scene recognition unit, which uses the learned machine learning model to perform scene recognition on multiple scenes based on the image of the subject obtained by the camera unit; a scene determination unit, which performs scene determination according to the recognition result of the scene recognition unit; a stabilization unit that stabilizes the determination result of the scene determination unit in terms of time; as well as A setting unit changes the setting of the imaging unit according to the determination result of the scene determination unit.

2. The microscope system according to claim 1, characterized in that The stabilization unit includes a scene recognition pre-processing unit that pre-processes the image of the subject. The scene recognition pre-processing unit performs an inverse transformation of the image processing performed by the imaging unit on the image of the subject. The scene recognition unit performs the scene recognition based on an inverse transformation image obtained by the inverse transformation performed by the scene recognition pre-processing unit.

3. The microscope system according to claim 1 or 2, characterized in that: The stabilization unit further includes a scene recognition post-processing unit, which performs post-processing on the recognition result of the scene recognition unit. The scene recognition post-processing unit calculates a cumulative recognition result based on a plurality of recognition results corresponding to a plurality of frames of the scene recognition unit, the cumulative recognition result being the cumulative recognition results within a period corresponding to the plurality of frames, The scene determination section performs the scene determination based on the accumulated recognition result.

4. The microscope system according to claim 3, characterized in that The scene determination unit suspends the scene determination when the evaluation value of the highest-evaluated scene in the cumulative recognition result does not exceed a threshold value.

5. The microscope system according to claim 4, characterized in that When the evaluation value of the highest-evaluated scene exceeds the threshold value, the scene determination unit outputs the scene corresponding to the evaluation value as the determination result.

6. The microscope system according to claim 4, characterized in that: When the evaluation value of the highest-evaluated scene exceeds the threshold and the scene corresponding to the evaluation value is not the background, the scene determination unit outputs the scene corresponding to the evaluation value as the determination result, The scene determination unit suspends the scene determination when the evaluation value of the highest-evaluated scene exceeds the threshold value and the scene corresponding to the evaluation value is the background.

7. The microscope system according to claim 4, characterized in that: When the evaluation value of the highest-evaluated scene exceeds the threshold, the scene determination unit decides whether to shelve the scene determination or output the scene corresponding to the evaluation value as the determination result based on the scene corresponding to the evaluation value and time series data of the determination result output immediately before.

8. The microscope system according to claim 4, characterized in that When the evaluation value of the highest-evaluated scene exceeds the threshold, the scene determination unit performs the scene determination based on the scene corresponding to the evaluation value and the time series data of the already output determination results, and outputs the determined scene as the determination result.

9. The microscope system according to any one of claims 4 to 8, characterized in that: The stabilization unit includes a threshold value adjustment unit that adjusts the threshold value. When the scene with the highest evaluation in the accumulated recognition result is different from the scene corresponding to the determination result output immediately before, the threshold adjustment unit increases the threshold, The threshold adjustment unit resets the threshold to an initial value when the scene with the highest evaluation in the cumulative recognition result is the same as the scene corresponding to the determination result output immediately before.

10. The microscope system according to claim 9, characterized in that The threshold value adjustment unit resets the threshold value to the initial value when movement of the subject is detected.

11. The microscope system according to claim 3, characterized in that: The scene recognition post-processing unit sets a predetermined number of frame periods from a timing when the setting unit changes settings or a timing when the setting unit is scheduled to change settings as a dead zone, and does not reflect the recognition results of the scene recognition unit corresponding to frames in the dead zone in the accumulated recognition results.

12. The microscope system according to claim 1, characterized in that: The setting unit has a restriction setting for restricting changes in the settings of the imaging unit, When the restriction setting is enabled, after the setting of the imaging unit corresponding to the determination result of the scene determination unit is changed, the change of the setting is prohibited.

13. The microscope system according to claim 1, characterized in that The setting unit further changes the setting of the microscope according to the determination result of the scene determination unit.

14. The microscope system according to claim 1, characterized in that The camera unit comprises: A filter insertion and removal unit capable of inserting and removing an infrared cutoff filter from an optical path; and a filter detection unit for detecting whether the infrared cut filter is inserted in the optical path, The scene recognition section switches a machine learning model used for the scene recognition based on a detection result of the optical filter detection section.

15. The microscope system according to claim 14, characterized in that The microscope system further includes a notification unit that outputs an alarm based on the determination result of the scene determination unit and the detection result of the filter detection unit.

16. A setting change method, characterized in that: The following processing is performed by the computer: Based on an image of a subject obtained by a camera unit, the camera unit photographs the subject through a microscope, and performs scene recognition on a plurality of scenes using a learned machine learning model. Performing scene determination according to the result of the scene recognition, Make the result of the scene determination stable in time, The setting of the imaging unit is changed according to the result of the scene determination.

17. A computer-readable storage medium storing a program, characterized in that: The program causes the computer to execute the following processing: Based on an image of a subject obtained by a camera unit, the camera unit photographs the subject through a microscope, and performs scene recognition on a plurality of scenes using a learned machine learning model. Performing scene determination according to the result of the scene recognition, Make the result of the scene determination stable in time, The setting of the imaging unit is changed according to the result of the scene determination.

18. A digital camera, characterized in that: The digital camera comprises: An image sensor that captures a subject through a microscope; a scene recognition unit that performs scene recognition on a plurality of scenes using a learned machine learning model based on an image of the subject generated by processing an output from the image sensor; and a scene determination unit that performs scene determination based on a recognition result of the scene recognition unit; a stabilization unit that stabilizes the determination result of the scene determination unit in terms of time; as well as A setting unit changes the setting of the digital camera according to the determination result of the scene determination unit.

Images