Controller and method for providing operation settings of an imaging device

Abstract

A controller (104) for providing operating settings of an imaging device (102) is disclosed. The controller is configured to provide a user with preferred operating settings for acquiring an image. The controller is further configured to receive a user input and generate response information based on the user input, the response information indicating whether an image generated by the imaging device is accepted by the user. The controller is further configured to update the preferred operating settings using a machine learning algorithm based on the response information.

Description

Technical Field

[0001] The present invention relates to a controller and a method for providing operating settings of an imaging device. Furthermore, the present invention relates to an imaging system including an imaging device and a controller for providing its operating settings. Background Art

[0002] In the field of microscopes, finding the operating settings of a microscope to achieve optimal image quality is a crucial requirement. Generally, the operating settings include multiple setting parameters that must be optimized simultaneously before starting image acquisition. Optimizing the setting parameters is a cumbersome and time-consuming task for the user and must be repeated for each image acquisition. Therefore, in order to obtain the best image quality in a simple and reproducible manner, there is a great need to provide tools for the user to make it easier to select the optimal operating settings.

[0003] Recently, machine learning algorithms have been developed to help users find the appropriate operating settings without having to adjust a large number of setting parameters each time the microscope is used. An example of a microscope system using a machine learning algorithm to provide operating settings is disclosed in the document DE 10 2014 102 080 A1. The machine learning algorithm disclosed therein allows the microscope system to learn the rules based on which the setting parameters are adjusted. The learning of such a system occurs during the manufacturing or development cycle of the microscope. Summary of the Invention

[0004] An object of the present invention is to provide a controller and a method for enabling a user to quickly find the appropriate operating settings of an imaging device with minimal effort. Furthermore, an object of the present invention is to provide an imaging system including a controller of the above type.

[0005] To achieve the above object, a controller for providing operating settings of an imaging device is proposed. The controller is configured to provide a user with preferred operating settings for acquiring an image. The controller is further configured to receive user input and generate response information based on the user input, the response information indicating whether an image generated by the imaging device is accepted by the user. The controller is further configured to update the preferred operating settings using a machine learning algorithm based on the response information.

[0006] The operating settings of the imaging device may include a plurality of setting parameters suitable for operating the imaging device as needed. In particular, the operating settings may include parameters for determining the quality of an object image generated by the imaging device.

[0007] The response information may include information indicating whether an image generated by the imaging device is discarded by the user or whether the setting parameters are changed by the user.

[0008] Before acquiring an image, the controller provides the user with preferred operation settings. These preferred operation settings can be directly used for subsequent image acquisition, i.e., they do not need to be changed by the user before image acquisition. Alternatively, the preferred operation settings can be changed by the user before image acquisition. In this case, the preferred operation settings are overwritten by the changed operation settings, and these changed operation settings are subsequently used for image acquisition.

[0009] After acquiring an image based on the operation settings, which can be the above-mentioned preferred settings initially provided by the controller or the settings changed by the user and overwriting the preferred settings, the controller receives user input. Based on the user input, the controller can determine whether the user discards the image generated by the imaging device according to the selected operation settings. Subsequently, based on this response information obtained from the user input, the controller applies a machine learning algorithm to update the preferred operation settings.

[0010] The response information represents simple decision criteria that the machine learning algorithm can use to effectively update the preferred operation settings of the imaging device. In particular, since the response information is generated based on user input during the imaging operation performed by the imaging device, the process of updating the operation settings using the machine learning algorithm can be performed while the user is actually working with the imaging device. Therefore, it is not necessary to implement a separate learning process before the actual image acquisition when the user is working with the imaging device. In other words, the user does not have to manually vote on whether the acquired image is a good image as taught, for example, by the above-mentioned document DE 10 2014 102 080 A1. In contrast, the controller according to an embodiment of the present invention determines whether the acquired image is a good image based on the user's behavior. Based on this conclusion, the controller can provide the machine learning algorithm.

[0011] The proposed controller is able to reliably predict the preferred operation settings and immediately apply the predicted settings. Therefore, the workflow can be improved, and the user no longer even needs to adjust multiple setting parameters before starting image acquisition. This prediction is based on previous user input, which indicates whether the current image acquired during the ongoing imaging process is accepted or discarded by the user.

[0012] Preferably, the controller is further configured to determine that the image is a good image when the response information indicates that the image has not been discarded by the user, and to determine that the image is not a good image when the response information indicates that the image has been discarded by the user, and to provide information indicating whether the image is determined to be a good image to the machine learning algorithm accordingly. Hereinafter, a good image can be understood as an image that meets the user's expectations, especially in terms of image quality, such that the user input generated after image acquisition indicates that the image is accepted, i.e., not discarded by the user. Therefore, a good image can be an image that the user continues to work with in one way or another during the imaging process. An image that is not accepted by the user as a good image as defined above will hereinafter be referred to as a bad image.

[0013] According to a preferred embodiment, the controller is configured to further update the preferred operation settings based on the operation settings used for image acquisition. The latter settings for image acquisition may be the above-mentioned preferred operation settings initially provided by the controller, or settings intentionally adjusted by the user for image acquisition. The intentionally adjusted settings may be derived from the preferred operation settings.

[0014] In this embodiment, the update of the preferred operation settings, i.e., the prediction of the next settings to be used for image acquisition, is based on the previous settings. Thus, the controller can feed the current operation settings to a machine learning algorithm to decide on the operation settings that the user may need at a later stage.

[0015] Preferably, the controller is configured to update the preferred operation settings to strengthen the operation settings used for image acquisition when the image is determined to be a good image, and / or update the preferred operation settings to weaken the operation settings used for image acquisition when the image is determined not to be a good image. In such an embodiment, the reinforcement machine learning can be performed in such a way that a good image results in the reinforcement of the selected operation settings. It can be said that the update of the preferred operation settings integrates the settings used for image acquisition. In contrast, a bad image results in the weakening of the current operation settings, i.e., the update of the preferred operation settings deviates from the settings used for image acquisition. For example, assume that the value of the preferred setting parameter is A, and then the user overwrites A with the changed parameter value B and acquires an image based on the value B. If the acquired image is a good image, the subsequent preferred parameter may be B. On the other hand, if the acquired image is a bad image, the next preferred setting parameter may be A again.

[0016] According to a preferred embodiment, the controller is configured to generate response information indicating that the image is discarded if, after image acquisition, the user changes the setting parameters of a first set of parameter values of the operation settings compared to the previous operation settings used for image acquisition.

[0017] Preferably, the first set of parameter values includes at least one of histogram settings, exposure time settings, gain settings, contrast settings, illumination light settings, objective lens settings, position settings, time settings, repeat settings, merge settings, HDR settings, digital fusion settings, color / black and white settings, auto exposure on / off settings, and confocal microscope settings.

[0018] The illumination light settings may include settings for each light type, including on / off settings for the ring light for the combined light segment, on / off settings for the coaxial light, and on / off settings for the transmitted light. The objective lens settings may include settings for selecting the objective lens for the overview image, z-stack, and z-stack height. The position settings may include settings for rows, columns, and their combinations. The time settings may include settings for time periods, time intervals, etc. The confocal microscope settings may include average number settings, speed settings, format settings, bi-directional scan on / off settings, pinhole size settings, motorized correction ring settings, automatic gain on / off settings, emission settings, excitation light settings, and dye settings.

[0019] Preferably, the controller is configured to generate response information indicating that the image is not discarded if, after acquiring the image, the set parameters of the second set of parameters of the operation settings are changed by the user compared to the previous operation settings.

[0020] Preferably, the second set of parameters includes at least one of the zoom setting and the stage setting.

[0021] In a further preferred embodiment, the controller is configured to generate response information indicating that the image is not discarded if the image is stored or further processed, even with further manipulation. The further processing may include annotation, measurement, cropping, etc.

[0022] According to another aspect, there is provided an imaging system including an imaging device and a controller for providing operation settings of the imaging device as described above.

[0023] The controller may be configured to automatically control the imaging device based on the updated operation settings.

[0024] In a preferred embodiment, the imaging device is a microscope system, such as a confocal microscope system.

[0025] According to another aspect, there is provided a method for providing operation settings of an imaging device, which includes the following steps: providing preferred operation settings for acquiring an image to a user; receiving user input and generating response information based on the user input, the response information indicating whether an image generated by the imaging device is discarded by the user; and updating the preferred operation settings using a machine learning algorithm based on the response information.

[0026] According to another aspect, there is provided a computer program having program code for performing the method when the computer program runs on a processor. Description of the Drawings

[0027] The preferred embodiments will be described below in conjunction with the drawings, where:

[0028] Figure 1is a block diagram showing an imaging system including a controller according to an embodiment; and

[0029] Figure 2 is a flowchart showing a workflow for providing operating settings of an imaging device. DETAILED DESCRIPTION

[0030] An imaging system 100 according to an embodiment is shown in the block diagram of Figure 1 . In the present embodiment, the imaging system 100 is formed by a microscope system.

[0031] The imaging system 100 includes an imaging device 102 such as a microscope and a controller 104. In the present invention, the controller 104 may include an input module 106 and a learning module 108. Both modules 106, 108 can be implemented by hardware and / or software. Further, in the present embodiment, the controller 104 is configured to control the overall operation of the imaging system 100.

[0032] In particular, when the imaging device 102 is activated, the controller 104 provides the user with preferred operating settings. These preferred operating settings may include a plurality of setting parameters for controlling the imaging device 102 to acquire an image. The preferred operating settings may have been pre-stored in a memory device included in the controller 104. To notify the user of the preferred operating settings, the imaging system 100 may include, for example Figure 1 a monitor not shown in, on which the preferred operating settings are displayed.

[0033] Optionally, the preferred operating information may be adjusted or selected according to information about the sample to be imaged. The user interface may be configured to receive sample information input by the user. Alternatively, information about the sample is collected by measurement, and the result of this measurement is transmitted to the user interface in the form of sample information. Such a measurement may provide information about specific sample characteristics without the user having to input such information into the user interface. Two ways of receiving such information, namely by user input and by sample measurement, are also possible. According to the information about the sample attributes received through the user interface, the controller 104 may automatically adjust one or more microscope components of the microscope.

[0034] After the preferred operating settings have been selected and / or adjusted, the imaging device 102 is operated to generate an image. To this end, the imaging device 102 is controlled based on the preferred operating settings or based on the operating settings that have been changed by the user before acquiring the image. In the latter case, the preferred operating settings may be overwritten in the storage device of the controller 104 in response to a corresponding user input.

[0035] After the image has been captured by the imaging device, the input module 106 of the controller 104 receives a user input made by the user through a suitable input device included in the imaging system 100. Such an input device can be a control panel, a keyboard, etc. configured to transmit the user input to the input module 106. The controller 104 is configured to generate response information based on the user input, that is, the response information is derived from the user input, such that the response information indicates whether the image generated by the imaging device 102 is accepted or discarded by the user, or whether the set parameters are changed by the user.

[0036] Subsequently, based on the response information derived from the user input, the controller 104 updates the preferred operation settings using the learning module 108. For this purpose, the learning module 108 is configured to apply a machine learning algorithm to the input data including the response information. This algorithm is capable of predicting the operation settings that are assumed to be preferred by the user when operating the imaging device for the next image acquisition. In this regard, it is worth noting that the controller 104 is configured to update the operation settings through the learning module 108 while the user is actually using the imaging device 102 for work. In other words, the processes of image acquisition and updating the operation settings are performed simultaneously. Therefore, the controller 104 can automatically control the imaging device 102 based on the updated operation settings.

[0037] In addition to the response information indicating whether the image generated by the imaging device 102 has been discarded by the user, the learning module 108 of the controller 104 can further be provided with the current operation settings used for image acquisition to predict the next update of the settings. In the case of adjusting or selecting the preferred operation information according to the information about the sample, the learning module 106 of the controller 104 can also be provided with the operation settings.

[0038] Figure 2 is a flowchart showing an exemplary workflow for updating the operation settings of the imaging device 102 included in Figure 1 the imaging system 100. For simplicity, Figure 2 is a hybrid diagram combining elements of a block diagram and a flowchart. In particular, Figure 2 shows the interaction between the learning module 108 and the rest of the imaging system 100.

[0039] In Figure 2In step S1 of the workflow shown, the controller 104 determines whether the image generated by the imaging device 102 is discarded by the user. To this end, the controller 104 analyzes response information derived from user input, which is input by the user in response to an image presented to him or her, for example, by being displayed on a monitor. If the controller 104 determines in S1 that the image is discarded, the controller 104 concludes in step S2 that the image is a good image. On the other hand, if the controller determines in S1 that the image is discarded by the user, the controller concludes in step S3 that the image is a bad image.

[0040] The conclusion of whether to discard the image, that is, whether the image is a good image or a bad image, can be obtained based on whether the user has changed one or more specific setting parameters of the operation settings after the image has been acquired. For example, if the setting parameters of the first set of parameters of the operation settings are changed by the user compared to the previous operation settings used to acquire the image after the image has been acquired, the image is determined to be a bad image, that is, discarded (step S3). The aforementioned first set of parameters may include at least one of histogram settings, exposure time settings, gain settings, contrast settings, illumination light settings, objective lens settings, position settings, time settings, repeat settings, merge settings, HDR settings, digital fusion settings, color / black and white settings, auto exposure on / off settings, and confocal microscope settings.

[0041] On the other hand, for example, if the setting parameters of the second set of parameters of the operation settings are changed by the user compared to the previous operation settings used to acquire the image after the image has been acquired, the image is determined to be a good image, that is, accepted by the user or not discarded (step S2). For example, the aforementioned second set of parameters may include at least one of zoom settings and stage settings. In other words, if only the zoom and / or stage settings are changed, the image will not be discarded. In addition, if the image has been stored or otherwise further processed, it can be determined that the image is a good image, that is, not discarded. Therefore, this further processing of the image indicates that the user is willing to continue using the image and has thus accepted the image.

[0042] In Figure 2 In step S4 of the workflow shown, the learning module 108 of the controller 102 is fed with response information indicating whether the current image is determined to be a good image or a bad image. In addition, in this embodiment, the learning module 108 is further fed with the operation settings for acquiring the current image. Applying a learning algorithm that takes into account the aforementioned information, the learning module 108 makes a prediction, based on which the preferred operation settings to be used in the next image acquisition are updated. As a result, the learning module 108 provides the imaging system 100 with the updated preferred operation settings to be used next.

[0043] When using the operation settings for obtaining the current image, the learning module can update the preferred operation settings in the case of affirmative or negative operation settings. Thus, the learning module 108 can be configured to apply reinforcement machine learning in such a way that good images lead to the reinforcement of the selected operation settings, while bad images lead to the weakening of the current operation settings.

[0044] The image can be a single acquired image, a sequence or set of acquired images, or can be a "live" image, i.e., a sequence of different images or frames continuously acquired, displayed, and updated by a microscope.

[0045] Although some aspects have been described in the context of a device, it is clear that these aspects also represent a description of the corresponding method, where blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent a description of the corresponding blocks or items or features of the corresponding device. Some or all method steps can be performed by (or using) hardware devices, such as a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some one or more of the most important method steps can be performed by such a device.

[0046] According to certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. For example, an implementation can be performed using a non-transitory storage medium such as a digital storage medium, such as a floppy disk, a DVD, a Blu-ray disc, a CD, a ROM, a PROM, and an EPROM, an EEPROM, or a flash memory, on which electronic readable control signals are stored, which cooperate with (or are capable of cooperating with) a programmable computer system in order to perform the corresponding method. Thus, the digital storage medium can be computer-readable.

[0047] Some embodiments according to the present invention include a data carrier having an electronically readable control signal that is capable of cooperating with a programmable computer system so as to perform one of the methods described herein.

[0048] Generally, embodiments of the present invention can be implemented as a computer program product having program code that is operable to perform one of the methods when the computer program product runs on a computer. For example, the program code can be stored on a machine-readable carrier.

[0049] Other embodiments include a computer program stored on a machine-readable carrier for performing one of the methods described herein.

[0050] In other words, embodiments of the present invention are thus a computer program having program code that, when the computer program runs on a computer, is for performing one of the methods described herein.

[0051] Accordingly, a further embodiment of the present invention is a storage medium (or data carrier, or computer-readable medium) having stored thereon a computer program for performing one of the methods described herein when executed by a processor. The data carrier, digital storage medium or recording medium is generally tangible and / or non-transitory. A further embodiment of the present invention is an apparatus as described herein, comprising a processor and a storage medium.

[0052] Accordingly, a further embodiment of the present invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence can be configured, for example, to be transmitted via a data communication connection, such as via the Internet.

[0053] A further embodiment includes a processing device, such as a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0054] A further embodiment includes a computer having installed thereon a computer program for performing one of the methods described herein.

[0055] A further embodiment according to the present invention includes an apparatus or system configured to transmit (e.g., electronically or optically) to a receiver a computer program for performing one of the methods described herein. For example, the receiver can be a computer, a mobile device, a memory device, etc. For example, the apparatus or system can include a file server for transmitting the computer program to the receiver.

[0056] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0057] List of reference numerals

[0058] 100 Imaging system

[0059] 102 Imaging device

[0060] 104 Controller

[0061] 106 Input module

[0062] 108 Learning module

[0063] Method steps S1 to S4

Claims

1. An imaging system, comprising an imaging device and a controller, wherein the controller is configured to: Provide a user with preferred operation settings of the imaging device for acquiring an image; Control the imaging device to acquire the image using operation settings based on the preferred operation settings, where the preferred operation settings are changed or not changed by the user; After the image has been acquired, receive a user input, which is the behavior of the user when using the imaging device, and the behavior of the user indicates whether the user changes the setting parameters of the operation settings for the next image acquisition; Generate response information based on the user input, and the response information indicates whether the image is accepted by the user based on whether the user changes the setting parameters of the operation settings for the next image acquisition; And Update the preferred operation settings using a machine learning algorithm based on the response information; Wherein the controller is configured to: If, after the image is acquired, the setting parameters of a first set of parameters of the operation settings are changed by the user compared with the previous operation settings used to acquire the image, generate the response information indicating that the image is discarded without the user manually inputting whether the image is a good image.

2. The imaging system according to claim 1, wherein the controller is further configured to: When the response information indicates that the image is not discarded by the user, determine that the image is a good image, When the response information indicates that the image is discarded by the user, determine that the image is not a good image, and Accordingly, provide information indicating whether the image is determined to be a good image to the machine learning algorithm.

3. The imaging system according to claim 1, wherein the controller is further configured to update the preferred operation settings based on the operation settings used to acquire the image.

4. The imaging system according to claim 3, wherein the controller is further configured to: When the image is determined to be a good image, update the preferred operation settings to strengthen the operation settings used to acquire the image, and / or When the image is determined not to be a good image, update the preferred operation settings to weaken the operation settings used to acquire the image.

5. The imaging system according to claim 1, wherein the first set of parameters includes at least one of histogram settings, exposure time settings, gain settings, contrast settings, illumination light settings, objective settings, position settings, time settings, repeat settings, merge settings, HDR settings, digital fusion settings, color / black and white settings, auto exposure on / off settings, and confocal microscope settings.

6. The imaging system according to claim 1, wherein the controller is configured to: If, after the image is acquired, the setting parameters of a second set of parameters of the operation settings are changed by the user compared with the previous operation settings used to acquire the image, generate the response information indicating that the image is not discarded.

7. The imaging system according to claim 6, wherein the second set of parameters includes at least one of zoom settings and stage settings.

8. The imaging system according to claim 1, wherein the controller is configured to: generate the response information indicating that the image is not discarded if the image is stored or further processed.

9. The imaging system according to claim 1, wherein the controller is configured to automatically control the imaging device based on updated operation settings.

10. The imaging system according to any one of claims 1-9, wherein the imaging system is a microscope system.

11. A method for providing operation settings of an imaging device, comprising the steps of: providing a user with preferred operation settings of the imaging device for acquiring an image; causing the imaging device to acquire the image based on the preferred operation settings, where the preferred operation settings are changed or not changed by the user; after the image has been acquired, receiving a user input, the user input being the user's behavior when using the imaging device, the user's behavior indicating whether the user changes setting parameters of the operation settings for the next image acquisition; generating response information based on the user input, the response information indicating whether the image is discarded by the user based on whether the user changes the setting parameters of the operation settings for the next image acquisition; and updating the preferred operation settings using a machine learning algorithm based on the response information; further comprising: generating the response information indicating that the image is discarded without the user manually inputting whether the image is a good image if, after acquiring the image, the setting parameters of a first set of parameters of the operation settings are changed by the user compared to the previous operation settings used to acquire the image.

12. A computer-readable storage medium having a computer program stored thereon, the computer program having program code that, when the computer program runs on a processor, is used to execute the method according to claim 11.

Images