Microscope device, image acquisition system, and image acquisition method

The microscope device with dual image sensors and feature-based control enhances the efficiency and accuracy of partial image acquisition from biological tissues, addressing inefficiencies in existing methods.

JP7586167B2Active Publication Date: 2024-11-19SONY GROUP CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022505092
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-03-05
Filing Date
2021-02-15
Publication Date
2024-11-19
Estimated Expiration
2041-02-15

AI Technical Summary

Technical Problem

Existing methods for acquiring partial images from biological tissues using microscope devices are inefficient and could be improved for better user convenience and information provision.

Method used

A microscope device equipped with two image sensors, a first sensor for capturing entire images and a second sensor for capturing regions at higher magnification, with a determination unit to analyze features and control the second sensor based on these features, allowing for efficient and appropriate partial image acquisition.

Benefits of technology

This approach reduces the load on information processing devices and enables more efficient and accurate acquisition of partial images, improving user convenience and providing valuable information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586167000001
    Figure 0007586167000001
  • Figure 0007586167000002
    Figure 0007586167000002
  • Figure 0007586167000003
    Figure 0007586167000003
Patent Text Reader

Abstract

Provided is a microscope device that makes it possible to acquire an image of a specific area of living tissue efficiently or in an appropriate manner. The present invention provides a microscope device comprising a first imaging element for capturing an image of a subject including living tissue and obtaining image data and a second imaging element for capturing an image of the subject at a rate of magnification differing from that of the first imaging element and acquiring image data, wherein the first imaging element comprises a determination unit for determining a feature relating to the subject on the basis of the image data, and the second imaging element is controlled on the basis of the determination result. The present invention also provides an image acquisition system including the microscope device. The present invention additionally provides an image acquisition method carried out in the microscope device.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present technology relates to a microscope apparatus, an image acquisition system, and an image acquisition method. More specifically, the present technology relates to a microscope apparatus for capturing an image of biological tissue, an image acquisition system including the microscope apparatus, and an image acquisition method of biological tissue. [Background technology]

[0002] For pathological diagnosis, biological tissue images obtained by a microscope device may be used. In recent years, digital images of biological tissue may be obtained, and pathological diagnosis may be performed based on the digital images. Several techniques for obtaining such digital images have been proposed.

[0003] For example, Patent Document 1 listed below discloses an information processing device that includes an image analysis unit that divides an overall image of an entire observation area into a plurality of areas and determines whether or not an object to be observed is present in each area, and an imaging control unit that controls the imaging of a partial image corresponding to each area at a higher magnification than the overall image based on the determination result of the image analysis unit, in which the imaging control unit controls the imaging so as to capture a first partial image corresponding to an area where the object to be observed may be present and adjacent to the area where the object to be observed may be present, and then capture a second partial image corresponding to an area where the object to be observed may be present and adjacent to the area where the object to be observed is not present. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2013 / 179723 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to perform a pathological diagnosis, a digital image of a biological tissue may be used, and the digital image may be acquired, for example, using a microscope device. In acquiring the digital image using the microscope device, for example, an entire image of a subject may be acquired first, and then a partial image of a region of interest in the entire image may be acquired. As described in the above-mentioned Patent Document 1, the entire image may be used to control the capture of the partial image.

[0006] If the above-described partial image acquisition process could be performed more efficiently or more appropriately, it would be possible to improve convenience for users or provide useful information to users. Therefore, a primary objective of the present technology is to provide a technology for acquiring partial images more efficiently or more appropriately. [Means for solving the problem]

[0007] The present technology includes a first imaging element for capturing an image of a target including biological tissue and acquiring image data; a second imaging element that captures an image of the object at a magnification different from that of the first imaging element to acquire image data; Equipped with The first imaging element includes a discrimination unit that discriminates features related to the object based on the image data, The second imaging element is controlled based on a result of the determination. A microscope apparatus is provided. The second imaging element can divide the area where the image has been captured into a plurality of areas, and capture an image of at least one of the plurality of areas to obtain image data. The first imaging element may be configured as an imaging element in which the discrimination section and an imaging section that performs imaging are arranged on a single chip. The second imaging element can capture images of each of the multiple regions at a higher magnification than the first imaging element. The image data can be bright field image data and / or fluorescent image data. The image data may be an image signal comprising brightness values ​​and / or frequencies. The features related to the object may include features related to attributes of the object. The features relating to the attributes of the object may include one or more selected from features relating to the attributes of the biological tissue, features relating to autofluorescence components, and features relating to lesions in the biological tissue. The features relating to the object may include features relating to a region of the object. The features relating to the region of the object may include one or more selected from features relating to a diseased region in the biological tissue and features relating to a foreign body in the object. The discrimination unit may perform the discrimination using a trained model. The microscope device may further include a control unit that controls imaging of the object by the second imaging element or controls processing of the image obtained by the second imaging element based on a discrimination result by the discrimination unit. The control unit may further include an imaging sequence control unit that controls an imaging order based on characteristics related to the region of interest. The imaging sequence control unit, Imaging may be controlled so that seams of the plurality of regions do not overlap a region of interest in the object, or Imaging may be controlled so that a region of interest in the object is imaged multiple times. The control unit may further include an exposure control unit that controls gain and / or exposure time. The exposure controller may control gain and / or exposure time on a pixel-by-pixel and / or wavelength-by-wavelength basis. The control unit may further include a light amount control unit that controls the amount of light from the light source. Processing of the images may include fluorescence separation processing.

[0008] The present technology also provides a microscope device including a first imaging element that images an object including biological tissue to obtain image data, and a second imaging element that images the object at a magnification different from that of the first imaging element to obtain image data, the first imaging element further including a discrimination unit that discriminates features related to the object based on the image data, and the second imaging element is controlled based on a result of the discrimination; a control unit that controls imaging of the object by the second imaging element or controls processing of an image obtained by the second imaging element based on a result of the determination; An image acquisition system is provided, comprising:

[0009] The present technology also provides a first imaging step of imaging an object including biological tissue with a first imaging element to obtain image data; a discrimination step of discriminating features of the object based on the image data by a discrimination unit provided in the first imaging element; a second imaging step of imaging the object at a magnification different from that of the first imaging element to acquire image data; Including, The second imaging element is controlled based on a result of the determination. A method for acquiring an image is also provided. [Brief description of the drawings]

[0010] [Figure 1A] 1 is a schematic diagram for explaining a configuration example of a microscope apparatus according to the present technology. [Figure 1B] 1 is a schematic diagram for explaining a configuration example of a microscope apparatus according to the present technology. [Diagram 2] FIG. 2 is a schematic diagram showing an example of the configuration of a first imaging element. [Diagram 3] 2 is a perspective view showing an outline of an example of the external configuration of a first imaging element; FIG. [Figure 4] FIG. 1 is a diagram illustrating an example of the configuration of an image acquisition device that captures images using a line scan method. [Diagram 5] FIG. 1 is a diagram illustrating an example of the configuration of an optical system of an image acquisition device that captures images using a line scan method. [Figure 6] FIG. 2 is a diagram for explaining an example of an imaging target; [Figure 7] FIG. 4 is an enlarged view of a part of an image capture target for explaining an illumination area and an image capture area. [Figure 8] 1A and 1B are diagrams illustrating a method of acquiring spectral data when an imaging element is configured with a single image sensor. [Figure 9]FIG. 9 is a diagram showing wavelength characteristics of the spectroscopic data acquired in FIG. 8. [Figure 10] 1A and 1B are diagrams illustrating a method of acquiring spectral data when an imaging element is composed of a plurality of image sensors. [Figure 11] 1 is a conceptual diagram illustrating a scanning method of a line illumination that is irradiated onto a target. [Figure 12] FIG. 1 is a conceptual diagram illustrating three-dimensional data (X, Y, λ) acquired with multiple line illuminations. [Figure 13] FIG. 13 is a diagram illustrating an example of a wavelength configuration of an irradiating unit. [Figure 14] FIG. 2 is a block diagram showing an example of a control unit; [Figure 15] 1 is a diagram illustrating an example of a process flow of an imaging process performed by a microscope apparatus according to the present technology. [Figure 16] FIG. 13 is a diagram for explaining a method for setting an imaging area. [Figure 17] FIG. 13 is a diagram for explaining a method for setting an imaging area. [Figure 18] FIG. 13 is a diagram for explaining a method for setting an imaging area. [Figure 19] FIG. 13 is a diagram for explaining a method for setting an exposure time. [Figure 20] FIG. 13 is a diagram for explaining a method for setting an exposure time. [Figure 21] 13A and 13B are diagrams for explaining a method of imaging control when the target includes a region that is not the target of analysis. [Figure 22] 13A and 13B are diagrams for explaining a method of imaging control when the target includes a region that is not the target of analysis. [Figure 23] 13A and 13B are diagrams for explaining a method of imaging control when the target includes a region that is not the target of analysis. [Figure 24] FIG. 13 is a diagram for explaining removal of an autofluorescence signal. [Diagram 25] FIG. 13 is a diagram for explaining removal of an autofluorescence signal. [Figure 26] FIG. 1 illustrates an example of the configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Preferred embodiments for carrying out the present technology will be described below. Note that the embodiments described below are representative embodiments of the present technology, and the scope of the present technology is not limited to these embodiments. Note that the present technology will be described in the following order. 1. First embodiment (microscope device) (1) Description of the First Embodiment (2) First Example of the First Embodiment (2-1) First imaging unit (2-1-1) Discrimination by the first imaging element (2-1-2) Example of the configuration of the first imaging element (2-2) Second imaging unit (2-2-1) Example of the second imaging unit (2-3) Control section (2-4) Stage (2-5) First example of imaging control (fluorescence separation processing) (2-5-1) Processing flow (2-5-2) Fluorescence separation processing (2-6) Second example of imaging control (imaging sequence control) (2-7) Third example of imaging control (imaging parameter control) (2-8) Fourth Example of Imaging Control (Imaging Control of a Target Including an Unnecessary Area) (2-8-1) Imaging control of subjects including artifacts (2-8-2) Imaging control of subjects including biological tissues not subject to analysis (3) Second Example of the First Embodiment 2. Second embodiment (image acquisition system) 3. Second embodiment (image acquisition method)

[0012] 1. First embodiment (microscope device)

[0013] (1) Description of the First Embodiment

[0014] It is possible to use an entire image of a target including biological tissue for controlling imaging to obtain a more detailed image of the biological tissue. In this case, it is possible to output the entire image data of the target from an imaging element that has obtained the image of the entire target to an information processing device connected to the imaging element, and the information processing device uses the entire image data for controlling a process to obtain a more detailed image.

[0015] However, for example, when the amount of overall image data is large, it may take time for the information processing device to receive the overall image data from the imaging element, and this time may be limited by, for example, the output interface of the imaging element or the data communication speed, etc. In addition, since the information processing device processes the overall image data, a load is imposed on the information processing device.

[0016] The microscope device according to the present technology includes a first imaging element that images an entire object including a biological tissue to obtain whole image data, and a second imaging element that images the object at a magnification different from that of the first imaging element to obtain image data. The first imaging element includes a discrimination unit that discriminates features of the object based on the image data, and the second imaging element is controlled based on the result of the discrimination. Since the first imaging element includes the discrimination unit, the first imaging element can output, for example, data related to the features of the object, rather than the image data itself, and the second imaging element is controlled based on the data related to the features. This reduces the effect of the rate limiting factor described above. Furthermore, data acquisition for controlling the second imaging element can be performed efficiently, and a better partial image can be obtained. In this way, the microscope device according to the present technology can obtain partial images more efficiently or more appropriately.

[0017] (2) First Example of the First Embodiment

[0018] An example of a microscope apparatus according to the present technology and an example of an image acquisition process using the microscope apparatus will be described below with reference to FIG. 1A.

[0019] 1A is a schematic diagram for explaining a configuration example of a microscope apparatus according to the present technology. The microscope apparatus 100 shown in FIG. 1A includes a first imaging unit 110, a second imaging unit 120, a control unit 130, and a stage 140.

[0020] (2-1) First imaging unit

[0021] 1A, the first imaging unit 110 includes, for example, a first imaging element 111, a first observation optical system 112, and a first illumination optical system 113. The first imaging unit 110 is configured to be able to capture an image of a target S including biological tissue.

[0022] The first imaging element 111 captures an image of a target S (particularly, the entire target S) including biological tissue to obtain at least one image data (particularly, the entire image data of the target S). The at least one image data may be, for example, bright field image data and / or fluorescence image data. The image data may also be an image signal including a luminance value and / or a frequency.

[0023] The first imaging element 111 includes an imaging unit 114 that acquires image data of the target S, as well as a discrimination unit 115 that discriminates features of the target S based on the image data. An example configuration of the first imaging element 111, including the imaging unit 114 and the discrimination unit 115, will be described in detail separately below.

[0024] The first observation optical system 112 may be configured so that the first imaging element 111 can magnify and image the target S. The first observation optical system 112 may include, for example, an objective lens. The objective lens may have a magnification such that the first imaging element 111 can image the entire target S. The first observation optical system 112 may also include a relay lens for relaying the image magnified by the objective lens to the first imaging element 111. The configuration of the first observation optical system 112 may be appropriately selected by a person skilled in the art depending on, for example, the target S or the type of image data to be acquired, and may include optical components other than the objective lens and the relay lens.

[0025] The first illumination optical system 113 is an optical system for illuminating the object S in the imaging by the first imaging element 111. The first illumination optical system 113 includes a light source for the illumination, and may irradiate the object S with, for example, visible light or ultraviolet light. The light source included in the first illumination optical system 113 may be appropriately selected by a person skilled in the art according to the type of image data to be acquired by the first imaging element 111, and may include, for example, at least one selected from a halogen lamp, an LED lamp, a mercury lamp, and a xenon lamp. For example, when the image data is bright field image data, the first illumination optical system 113 may include, for example, an LED lamp or a halogen lamp. When the image data is fluorescence image data, the first illumination optical system 113 may include, for example, an LED lamp, a mercury lamp, or a xenon lamp. The wavelength of the irradiated light or the type of lamp may be selected according to the type of fluorescent material that emits fluorescence.

[0026] (2-1-1) Discrimination by the first imaging element

[0027] The first imaging element 111 includes a discrimination unit 115 that discriminates features of the target S based on the image data. The first imaging element 111 may acquire the features of the target S as data through the discrimination. The second imaging element 121 is controlled based on the result of the discrimination. The features of the target S may include, for example, features of biological tissue contained in the target S.

[0028] The features related to the object S may include, for example, features related to attributes of the object S. More specifically, the features related to the attributes of the object S may include one or more selected from features related to attributes of the biological tissue, features related to autofluorescence components, and features related to lesions in the biological tissue.

[0029] The features related to the attributes of the biological tissue may include, for example, features related to the type of organism from which the biological tissue originates, the type of the biological tissue, the type of cells contained in the biological tissue, or features related to a disease that the biological tissue may have. The type of organism from which the biological tissue is derived may be, for example, a taxonomic type of organism (e.g., human), a gender type (e.g., male or female), or an age type (e.g., age, month, or day). The type of biological tissue may be, for example, the type of organ from which the biological tissue originates (e.g., stomach), the type of organ component (e.g., mucosal or muscular layer of the stomach), or the type of bodily fluid (e.g., blood). The types of cells contained in the biological tissue may be based on classification from the perspective of one or more of morphology, function, and cellular components, or may be based on a label (e.g., a fluorescent substance or an antibody) attached to the cells. The characteristic related to a disease that the biological tissue may have may be, for example, whether or not the biological tissue has a disease, the possibility that the biological tissue has a disease, or the type of disease (for example, the name of the disease or the degree of progression of the disease).

[0030] The features related to the subject S may include, for example, features related to a region of the subject S. The features related to the region of the subject S may include, for example, one or more features selected from features related to a diseased region in the biological tissue and features related to a foreign body in the subject S. Furthermore, the features related to the region of the subject S may be a region of cells, a non-cellular region, or a region of a cell organelle (e.g., a cell nucleus, a nucleolus, an endoplasmic reticulum, or a mitochondria, etc.) in the biological tissue. The foreign body in the subject may be, for example, dust or artifacts in the subject S.

[0031] The feature related to the object S may include, for example, a feature related to the arrangement of the object S. The feature related to the arrangement of the object S may be, for example, the inclination of the object S, and more specifically, the inclination of the slide glass.

[0032] In a preferred embodiment of the present technology, the discrimination unit 115 performs the discrimination using a trained model. The trained model may be selected according to the discrimination result to be obtained. For example, the trained model may be a trained model that has been machine-learned using at least one, preferably a plurality of, teacher data including an image of a target including biological tissue and features associated with the image (particularly features related to the target).

[0033] The trained model may be a trained model generated by, for example, deep learning. For example, the trained model may be a multi-layer neural network, for example, a deep neural network (DNN), and more specifically, a convolutional neural network (CNN). The multi-layer neural network may have an input layer that inputs image data acquired by a first image sensor, an output layer that outputs image data used to extract features related to the target or features related to the target, and at least one intermediate layer, for example, 2 to 10 intermediate layers, provided between the input layer and the output layer.

[0034] In a preferred embodiment of the present technology, the discrimination unit 115 discriminates features related to the object using a trained model, and more specifically, discriminates attributes of the object and / or discriminates an area of ​​the object. By performing such discrimination by the discrimination unit 115 provided in the first imaging element 111, it is possible to more efficiently or more appropriately control the imaging by the second imaging element 121 or control the processing of the captured image.

[0035] (2-1-2) Example of the configuration of the first imaging element

[0036] Fig. 2 shows an example of the configuration of the first imaging element 111. As shown in Fig. 2, the first imaging element 111 has an imaging block 20 and a signal processing block 30. The imaging block 20 and the signal processing block 30 are electrically connected by connection lines (internal buses) CL1, CL2, and CL3.

[0037] The imaging block 20 includes an imaging section 21, an imaging processing section 22, an output control section 23, an output I / F 24, and an imaging control section 25. The imaging section 21 corresponds to the imaging section 114 described above. The signal processing block 30 may include a CPU (Central Processing Unit) 31, a DSP (Digital Signal Processor) 32, and a memory 33. The signal processing block 30 may further include a communication I / F 34, an image compression unit 35, and an input I / F 36. The signal processing block 30 performs predetermined signal processing using image data obtained by the imaging block 20 (particularly the imaging unit 21 and the imaging processing unit 22). The signal processing block 30 realizes the discrimination process by the discrimination unit described above in "(2-1-1) Discrimination performed by the first imaging element." These components included in the first imaging element 111 will be described below.

[0038] The imaging unit 21 captures an image of a target S including biological tissue. The imaging unit 21 can perform the imaging by being driven by, for example, an imaging processing unit 22. The imaging unit 21 may include, for example, a plurality of pixels arranged two-dimensionally. Each pixel included in the imaging unit 21 receives light, performs photoelectric conversion, and outputs an analog image signal based on the received light.

[0039] The size of the image (signal) output by the imaging unit 21 can be selected from a number of sizes, such as 12M (3968×2976) pixels or VGA (Video Graphics Array) size (640×480 pixels). The image output by the imaging unit 21 may be a color image or a black and white image. A color image can be represented, for example, by RGB (red, green, blue). A black and white image can be represented, for example, by brightness only. These selections can be made as a type of shooting mode setting.

[0040] The imaging processing unit 22 can perform imaging processing related to the imaging of the imaging unit 21. For example, under the control of the imaging control unit 25, the imaging processing unit 22 performs operations such as driving the imaging unit 21, AD (Analog to Digital) conversion of an analog image signal output by the imaging unit 21, and imaging signal processing. Imaging processing can be performed.

[0041] More specifically, the imaging signal processing may be, for example, a process of calculating the brightness of each small region of the image output by the imaging unit 21 by calculating the average pixel value for each predetermined small region, a process of converting the image output by the imaging unit 21 into an HDR (High Dynamic Range) image, defect correction, or development.

[0042] The imaging processing unit 22 can output a digital image signal (for example, an image of 12M pixels or VGA size) obtained by AD conversion or the like of the analog image signal output by the imaging unit 21 as a captured image.

[0043] The captured image output by the imaging processing section 22 can be supplied to the output control section 23. The captured image output by the imaging processing section 22 can be supplied to the signal processing block 30 (particularly, the image compression section 35) via a connection line CL2.

[0044] The output control unit 23 can receive a captured image from the imaging processing unit 22. In addition, the output control unit 23 can receive a determination result using, for example, captured image data from the signal processing block 30 via a connection line CL3.

[0045] The output control unit 23 performs output control to selectively output the captured image supplied from the imaging processing unit 22 and the determination result by the signal processing block 30 from (one) output I / F 24 to the outside of the first imaging element 111. The imaging element included in the microscope device of the present technology preferably includes an output control unit that performs the output control.

[0046] That is, the output control unit 23 selects the captured image from the imaging processing unit 22 or the discrimination result from the signal processing block 30, and supplies it to the output I / F 24.

[0047] The output I / F 24 is an I / F that outputs the captured image and the discrimination result supplied from the output control unit 23 to the outside. For example, a relatively high-speed parallel I / F such as MIPI (Mobile Industry Processor Interface) can be adopted as the output I / F 24. The output I / F 24 outputs the captured image from the imaging processing unit 22 or the discrimination result from the signal processing block 30 to the outside in accordance with the output control by the output control unit 23. Therefore, for example, when only the discrimination result from the signal processing block 30 is required outside and the captured image itself is not required, it is possible to output only the discrimination result, and the amount of data output from the output I / F 24 to the outside can be reduced. Furthermore, the signal processing block 30 performs discrimination processing to obtain a discrimination result used in components (e.g., the second imaging element 121 and / or the control unit 130) external to the first imaging element 111, and the discrimination result is output from the output I / F 24. This eliminates the need to perform signal processing externally, and can reduce the load on external blocks.

[0048] The imaging control unit 25 controls the imaging processing unit 22 in accordance with the imaging information stored in the register group 27, and can thereby control imaging by the imaging unit .

[0049] The register group 27 can store imaging information, the results of imaging signal processing in the imaging processing unit 22, and output control information related to output control in the output control unit 23. The output control unit 23 can perform output control to selectively output the captured image and the discrimination result in accordance with the output control information stored in the register group 27.

[0050] The imaging control unit 25 and a CPU included in the signal processing block 30 may be connected via a connection line CL1. The CPU can read and write information from and to the register group 27 via the connection line. That is, reading and writing of information from and to the register group 27 may be performed from the communication I / F 26 or from the CPU.

[0051] The signal processing block 30 determines characteristics related to the target S, for example, based on at least one image data (particularly the overall image data of the target S). The CPU 31, DSP 32, memory 33, communication I / F 34, and input I / F 36 that constitute the signal processing block 30 are connected to each other via a bus, and can exchange information as necessary. The determination by the signal processing block 30 can be performed, for example, by the signal processing block 30 performing a predetermined signal processing using image data obtained by the imaging unit. Components that can be included in the signal processing block 30 will be described below.

[0052] The CPU 31 executes a program stored in the memory 33 to perform various processes such as controlling the signal processing block 30 or reading and writing information to the register group 27 of the imaging control unit 25. For example, the CPU 31 executes a program to function as an imaging information calculation unit that calculates imaging information using a signal processing result obtained by signal processing in the DSP 32, and feeds back new imaging information calculated using the signal processing result to the register group 27 of the imaging control unit 25 via the connection line CL1 to store it. Therefore, the CPU 31 can control imaging by the imaging unit 21 and / or imaging signal processing by the imaging processing unit 22 according to the signal processing result of the captured image. In addition, the imaging information stored in the register group 27 by the CPU 31 can be provided (output) to the outside from the communication I / F 26. For example, focus information of the imaging information stored in the register group 27 can be provided from the communication I / F 26 to a focus driver (not shown) that controls the focus.

[0053] By executing a program stored in memory 33, DSP 32 functions as a signal processing unit that performs signal processing (e.g., discrimination processing) using image data supplied to the signal processing block 30 from the imaging processing unit 22 via connection line CL2 and information received from the outside by input I / F 36.

[0054] The memory 33 can be configured with a static random access memory (SRAM), a dynamic RAM (DRAM), etc. The memory 33 stores various data such as data used for processing by the signal processing block 30, for example.

[0055] For example, the memory 33 stores a program received from the outside via the communication I / F 34, captured image data compressed by the image compression unit 35, particularly captured image data used in signal processing by the DSP 32, signal processing results (e.g., discrimination results) of the signal processing performed by the DSP 32, or information received by the input I / F 36.

[0056] The communication I / F 34 is a second communication I / F, such as a serial communication I / F such as SPI (Serial Peripheral Interface), and exchanges necessary information, such as programs executed by the CPU 31 or DSP 32, with external components (such as a memory or an information processing device external to the first imaging element 111).

[0057] For example, the communication I / F 34 downloads a program to be executed by the CPU 31 or the DSP 32 from the outside, and supplies it to the memory 33 for storage. Therefore, the program downloaded by the communication I / F 34 allows the CPU 31 or the DSP 32 to execute various processes. The communication I / F 34 can exchange not only programs but also any data with the outside. For example, the communication I / F 34 can output to the outside the signal processing result obtained by the signal processing in the DSP 32. The communication I / F 34 can also output information according to the instruction of the CPU 31 to an external device, thereby controlling the external device according to the instruction of the CPU 31.

[0058] Here, the signal processing results obtained by the signal processing in the DSP 32 can be outputted to the outside from the communication I / F 34, and can also be written by the CPU 31 to the register group 27 of the imaging control unit 25. The signal processing results written to the register group 27 can be outputted to the outside from the communication I / F 26. The same applies to the processing results performed by the CPU 31.

[0059] The image compression section 35 receives a captured image via a connection line CL2 from the imaging processing section 22. The image compression section 35 performs a compression process to compress the captured image, and generates a compressed image having a smaller amount of data than the captured image. The compressed image generated by the image compression unit 35 is supplied to the memory 33 via the bus and stored therein.

[0060] Here, the signal processing in the DSP 32 can be performed using the captured image itself, as well as using a compressed image generated from the captured image by the image compression unit 35. Since the compressed image has a smaller amount of data than the captured image, it is possible to reduce the signal processing load in the DSP 32 and save the storage capacity of the memory 33 that stores the compressed image.

[0061] The compression process in the image compression unit 35 may involve, for example, scaling down a captured image of 12M (3968×2976) pixels to a VGA-sized image. In addition, when the signal processing in the DSP 32 is performed on luminance and the captured image is an RGB image, the compression process may involve YUV conversion to convert the RGB image into a YUV image. The image compression unit 35 can be realized by software or by dedicated hardware. The input I / F 36 is an I / F that receives information from the outside. For example, the input I / F 36 receives an output of an external sensor (external sensor output) from the external sensor, and supplies the output to the memory 33 via a bus for storage. As the input I / F 36, similar to the output I / F 24, for example, a parallel I / F such as MIPI (Mobile Industry Processor Interface) can be adopted. In addition, as an external sensor, for example, a distance sensor that senses information regarding distance can be used.Furthermore, as an external sensor, for example, an image sensor that senses light and outputs an image corresponding to that light, that is, an image sensor other than the first imaging element 111 can be used.

[0062] In addition to using the captured image (the compressed image generated from the captured image), the DSP 32 can perform signal processing using external sensor output received by the input I / F 36 from an external sensor such as those described above and stored in the memory 33.

[0063] In the one-chip first imaging element 111 configured as above, signal processing (e.g., discrimination processing) using a captured image (or a compressed image generated from the captured image) obtained by imaging in the imaging unit 21 is performed in the DSP 32, and the signal processing result of the signal processing and the captured image are selectively output from the output I / F 24. Therefore, it is possible to configure a compact imaging device that outputs information required by a user.

[0064] Here, in the case where the first imaging element 111 does not perform signal processing by the DSP 32, and therefore does not output the signal processing results from the first imaging element 111 but outputs an image, that is, where the first imaging element 111 is configured as an image sensor that simply captures and outputs an image, the first imaging element 111 can be configured only with an imaging block 20 that does not have an output control unit 23.

[0065] FIG. 3 is a perspective view showing an outline of an example of the external configuration of the first imaging element 111 in FIG.

[0066] The first imaging element 111 can be configured as a one-chip semiconductor device having a stacked structure in which a plurality of dies are stacked, for example, as shown in Fig. 3. That is, in the present technology, the first imaging element 111 may be configured as an imaging element in which the discrimination unit and an imaging unit that performs imaging are arranged in a single chip.

[0067] In FIG. 3, the first imaging element 111 is configured by stacking two dies, dies 51 and 52.

[0068] 3, the upper die 51 is equipped with the imaging unit 21, and the lower die 52 is equipped with the imaging processing unit 22 through the imaging control unit 25, and the CPU 31 through the input I / F 36. The upper die 51 and the lower die 52 are electrically connected to each other, for example, by forming a through hole that penetrates the die 51 and reaches the die 52, or by performing Cu-Cu bonding that directly connects a Cu wiring exposed on the lower surface side of the die 51 to a Cu wiring exposed on the upper surface side of the die 52.

[0069] Here, in the imaging processing section 22, as a method for performing AD conversion on the image signal output from the imaging section 21, for example, a column parallel AD method or an area AD method can be adopted.

[0070] In the column-parallel AD method, for example, an ADC (AD Converter) is provided for each column of pixels constituting the imaging unit 21, and the ADC of each column is responsible for AD conversion of the pixel signals of the pixels of that column, thereby performing AD conversion of the image signals of the pixels of each column in one row in parallel. When the column-parallel AD method is adopted, a part of the imaging processing unit 22 that performs AD conversion of the column-parallel AD method may be mounted on the upper die 51.

[0071] In the area AD method, the pixels constituting the imaging unit 21 are divided into a plurality of blocks, and an ADC is provided for each block. Then, the ADC of each block is responsible for AD conversion of the pixel signals of the pixels of that block, and thus AD conversion of the image signals of the pixels of the plurality of blocks is performed in parallel. In the area AD method, the block is the smallest unit, and AD conversion (reading and AD conversion) of the image signals can be performed only on necessary pixels among the pixels constituting the imaging unit 21.

[0072] If it is acceptable for the area of ​​the first imaging element 111 to be large, the first imaging element 111 can be configured as a single die.

[0073] 3, two dies 51 and 52 are stacked to form the one-chip first imaging element 111, but the one-chip first imaging element 111 can be formed by stacking three or more dies. When three dies are stacked to form the one-chip first imaging element 111, for example, the memory 33 in FIG. 3 can be mounted on a separate die.

[0074] (2-2) Second imaging unit

[0075] A schematic configuration example of the second imaging unit 120 is shown in FIG. 1A. The second imaging unit 120 may be configured to capture an image of the object at a magnification different from that of the first imaging unit 110 (more specifically, the first imaging element 111) to obtain image data. For example, the second imaging unit 120 may be configured to capture an area captured by the first imaging unit 110 (more specifically, the first imaging element 111) by dividing the area into a plurality of areas. The second imaging unit 120 includes a second imaging element 121, a second observation optical system 122, and a second illumination optical system 123.

[0076] The second imaging element 121 may be configured to divide the area imaged by the first imaging element 111 into a plurality of areas and image at least one of the plurality of areas to obtain image data. For example, the second imaging element 121 may image each of the plurality of areas at a higher magnification or resolution than the first imaging element 111.

[0077] The second imaging element 121 divides the area (hereinafter also referred to as the "whole area") imaged by the first imaging element into a plurality of areas and images the divided areas (hereinafter also referred to as the "divided areas"). Examples of the division method include, but are not limited to, a line scan method or a tiling method. In the line scan method, the entire area can be divided into a plurality of strip-shaped areas, and the second imaging element 121 can capture an image of each of the plurality of strip-shaped divided areas. In the tiling method, the entire area can be divided into a plurality of tile-shaped areas, and the second imaging element 121 can capture an image of each of the plurality of tile-shaped divided areas.

[0078] The second observation optical system 122 is an optical system configured to enable the second imaging element 121 to capture an enlarged image of each of the divided regions. The second observation optical system 122 may include, for example, an objective lens. The objective lens may have a magnification that enables the second imaging element 121 to capture an image of each of the divided regions. For example, the objective lens included in the second observation optical system 122 may have a higher magnification than the objective lens included in the first observation optical system 112. The second observation optical system 122 may also include a relay lens for relaying the image enlarged by the objective lens to the second imaging element 121.

[0079] The second illumination optical system 123 is an optical system for illuminating the object S. The second illumination optical system 123 may irradiate the object S with, for example, visible light or ultraviolet light. The second illumination optical system 123 may be appropriately selected by a person skilled in the art depending on the type of image data to be acquired, and may include, for example, at least one selected from a halogen lamp, an LED lamp, a mercury lamp, and a xenon lamp. The image data is, for example, fluorescence image data, and in this case, the second illumination optical system 123 may include, for example, an LED lamp, a mercury lamp, or a xenon lamp. The type of lamp may be selected depending on the type of phosphor that emits fluorescence.

[0080] When the object S is imaged by the second imaging unit 120, the stage 140 on which the object S is placed can be moved to a position where the object S can be imaged via the second observation optical system 122, as shown in Fig. 1B, for example. For this movement, the microscope device 100 may include a stage control unit (not shown) that moves the stage 140 between a position where the object S can be imaged by the first imaging unit 110 and a position where the object S can be imaged by the second imaging unit 120. The stage control unit can be controlled by the control unit 130, for example.

[0081] (2-2-1) Example of the second imaging unit

[0082] A configuration example in which the second imaging unit 120 is an imaging unit that captures images using a line scan method will be described below with reference to FIGS.

[0083] The second imaging unit 120 has a spectral imaging section 150 that acquires a fluorescence spectrum (spectral data) of linearly excited biological tissue, a second observation optical system 122, and a second illumination optical system 123 that irradiates the target S with a plurality of line illuminations having different wavelengths arranged parallel to different axes. As shown in Fig. 5, the spectral imaging section 150 includes a second imaging element 121. After imaging by the first imaging element 111, the stage 140 can be moved to a position where imaging can be performed by the second imaging unit 120.

[0084] Here, "different axes and parallel" refers to multiple line illuminations that are different axes and parallel. Different axes refers to not being on the same axis, and the distance between the axes is not particularly limited. Parallel does not necessarily mean parallel in the strict sense, but also includes a state of being approximately parallel. For example, deviation from the parallel state due to distortion from an optical system such as a lens or manufacturing tolerances is acceptable, and in this case too it is considered to be parallel.

[0085] A second illumination optical system 123 and a spectral imaging unit 150 are connected to the stage 140 via a second observation optical system 122 including an objective lens 44 and the like. The second observation optical system 122 has a function of tracking an optimal focus by a focus mechanism 60. A non-fluorescence observation unit 70 that performs dark-field observation or bright-field observation or the like may be connected to the second observation optical system 122.

[0086] The second illumination optical system 123 includes a plurality of light sources L1, L2, ..., and Ln (n is, for example, 1 to 10, particularly 1 to 8) capable of outputting light of a plurality of excitation wavelengths Ex1, Ex2, ..., and Exn (n is, for example, 1 to 10, particularly 1 to 8). The plurality of light sources are typically constituted by light-emitting diodes (LEDs), laser diodes (LDs), mercury lamps, or the like, and each light is converted into line illumination and irradiated onto the target S held on the stage 140.

[0087] The object S is typically composed of a slide including biological tissue Sa such as a tissue section as shown in FIG. 6, but may be other than that. The object S may be stained with multiple fluorescent dyes. The second imaging unit 120 observes the object S at a desired magnification. When the portion A in FIG. 6 is enlarged, the second illumination optical system 123 has multiple line illuminations (two (Ex1 and Ex2) in FIG. 7) arranged as shown in FIG. 7), and the imaging areas R1 and R2 of the spectral imaging section 150 are arranged so as to overlap with the respective illumination areas. The two line illuminations Ex1 and Ex2 are parallel to each other along the X-axis direction and are arranged at a predetermined distance (Δy) apart in the Y-axis direction.

[0088] The photographing areas R1 and R2 correspond to each slit section of the observation slit 51 (FIG. 5) in the spectral imaging section 150. That is, the same number of slit sections as the line illuminations are arranged in the spectral imaging section 150. In FIG. 7, the illumination line width is wider than the slit width, but the size relationship between them may be either. If the illumination line width is wider than the slit width, the positioning margin of the second illumination optical system 123 with respect to the spectral imaging section 150 can be increased.

[0089] The wavelengths constituting the first line illumination Ex1 and the wavelengths constituting the second line illumination Ex2 are different from each other. Line-shaped fluorescence excited by these line illuminations Ex1 and Ex2 is received by the spectral imaging unit 150 via the second observation optical system 122.

[0090] The spectral imaging section 150 has an observation slit 51 having a plurality of slit portions through which the fluorescence excited by the plurality of line illuminations can pass, and at least one second image sensor 121 capable of individually receiving the fluorescence that has passed through the observation slit 51. A two-dimensional imager such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) is used for the second image sensor 121. By arranging the observation slit 51 on the optical path, it is possible to detect the fluorescence spectra excited by each line without overlapping.

[0091] The spectral imaging unit 150 acquires fluorescence spectral data (x, λ) from each of the line illuminations Ex1 and Ex2, using the pixel array in one direction (e.g., the vertical direction) of the image sensor 121 as a wavelength channel. The acquired spectral data (x, λ) is recorded, for example, in a control unit, in a state where it is linked to which excitation wavelength each piece of spectral data was excited from.

[0092] 5, the second imaging unit 120 is configured such that a dichroic mirror 42 and a bandpass filter 45 are inserted in the optical path so that the excitation light (Ex1, Ex2) does not reach the second imaging element 121. In this case, intermittent portions occur in the fluorescence spectrum imaged on the second imaging element 121 (see FIGS. 8 and 9). By excluding the intermittent portions from the readout region, the frame rate can be improved.

[0093] 5, the second imaging element 121 may include a plurality of imaging elements 121a and 121b each capable of receiving fluorescence that has passed through the observation slit 51. In this case, the fluorescence spectra Fs1 and Fs2 excited by the respective line illuminations Ex1 and Ex2 are acquired on the imaging elements 121a and 121b as shown in FIG. 10, and are stored in a storage unit (not shown) in association with the excitation light.

[0094] The line illuminations Ex1 and Ex2 are not limited to being composed of a single wavelength, and each may be composed of multiple wavelengths. When the line illuminations Ex1 and Ex2 are each composed of multiple wavelengths, the fluorescence excited by these also includes multiple spectra. In this case, the spectral imaging unit 150 has a wavelength dispersion element for separating the fluorescence into spectra derived from the excitation wavelengths. The wavelength dispersion element is composed of, for example, a diffraction grating or a prism, and is typically disposed on the optical path between the observation slit 51 and the image sensor 121.

[0095] The second imaging unit 120 further includes a scanning mechanism (not shown) that scans the stage 140 with a plurality of line illuminators Ex1 and Ex2 in the Y-axis direction, that is, in the arrangement direction of each of the line illuminators Ex1 and Ex2. By using the scanning mechanism, dye spectra (fluorescence spectra) excited with different excitation wavelengths, which are spatially separated by Δy on the sample S (observation target Sa), can be continuously recorded in the Y-axis direction. In this case, for example, as shown in FIG. 11, the photographing region Rs is divided into a plurality of parts in the X-axis direction, and the sample S is scanned in the Y-axis direction, and then moved in the X-axis direction, and the operation of scanning in the Y-axis direction is repeated. Spectral images derived from a sample excited by several kinds of excitation wavelengths can be photographed in one scan.

[0096] Typically, the scanning mechanism scans the stage 140 in the Y-axis direction, but a galvanometer mirror disposed midway through the optical system may scan the multiple line illuminators Ex1 and Ex2 in the Y-axis direction. Finally, three-dimensional data of (X, Y, λ) as shown in Fig. 12 is acquired for each of the multiple line illuminators Ex1 and Ex2. The three-dimensional data derived from each of the line illuminators Ex1 and Ex2 is data whose coordinates are shifted by Δy about the Y-axis, and is therefore corrected and output based on Δy recorded in advance or a value of Δy calculated from the output of the second image sensor 121.

[0097] In the examples so far, the line illumination as the excitation light is configured with two lines, but this is not limited thereto, and it may be three, four, five or more lines. Also, each line illumination may include a plurality of excitation wavelengths selected so as not to deteriorate the color separation performance as much as possible. Even if there is only one line illumination, if the excitation light source is configured with a plurality of excitation wavelengths, and each excitation wavelength is combined and recorded with the row data acquired by the image sensor, a multicolor spectrum can be obtained, although the separation ability is not as good as that of different parallel axes. For example, a configuration as shown in FIG. 13 may be adopted.

[0098] Next, the second imaging unit 120 will be described in detail with reference to Fig. 5. Here, an example in which the second imaging unit 120 is configured in the configuration example 2 in Fig. 13 will be described.

[0099] The second illumination optical system 123 has a plurality of (four in this example) excitation light sources L1, L2, L3, and L4. Each of the excitation light sources L1 to L4 is formed of a laser light source that outputs laser light having a wavelength of 405 nm, 488 nm, 561 nm, and 645 nm, respectively.

[0100] The second illumination optical system 123 further includes a plurality of collimator lenses 11 and laser line filters 12 corresponding to the respective excitation light sources L1 to L4, dichroic mirrors 13A, 13B, and 13c, a homogenizer 14, a condenser lens 15, and an entrance slit 16.

[0101] The laser light emitted from the excitation light source L1 and the laser light emitted from the excitation light source L3 are each collimated by a collimator lens 11, then transmitted through a laser line filter 12 for cutting the base of each wavelength band, and made coaxial by a dichroic mirror 13A. The two coaxial laser lights are further beam-shaped by a homogenizer 14, such as a fly-eye lens, and a condenser lens 15 to become the line illumination Ex1.

[0102] The laser light emitted from the excitation light source L2 and the laser light emitted from the excitation light source L4 are similarly made coaxial by the dichroic mirrors 13B and 13c, and are converted into line illumination Ex2 having a different axis from the line illumination Ex1. The line illuminations Ex1 and Ex2 form different-axis line illuminations (primary images) separated by Δy in an entrance slit 16 (slit conjugate) having multiple slit portions through which each of the line illuminations Ex1 and Ex2 can pass.

[0103] This primary image is irradiated onto the target S on the stage 140 via the second observation optical system 122. The second observation optical system 122 has a condenser lens 41, dichroic mirrors 42 and 43, an objective lens 44, a bandpass filter 45, and a condenser lens 46. The line illuminations Ex1 and Ex2 are collimated by the condenser lens 41 paired with the objective lens 44, reflected by the dichroic mirrors 42 and 43, transmitted through the objective lens 44, and irradiated onto the target S.

[0104] 7 is formed on the target S surface. Fluorescence excited by these illuminations is collected by the objective lens 44, reflected by the dichroic mirror 43, transmitted through the dichroic mirror 42 and a bandpass filter 45 that cuts the excitation light, collected again by the condenser lens 46, and enters the spectral imaging unit 150.

[0105] The spectral imaging section 150 includes an observation slit 51, a second imaging element 121 (121a, 121b), a first prism 53, a mirror , a diffraction grating 55 (wavelength dispersion element), and a second prism .

[0106] The observation slit 51 is disposed at the focusing point of the condenser lens 46, and has the same number of slit sections as the number of excitation lines. The fluorescence spectra originating from the two excitation lines that have passed through the observation slit 51 are separated by the first prism 53, and are further separated into fluorescence spectra of each excitation wavelength by being reflected by the grating surface of the diffraction grating 55 via the mirror 54. The four fluorescence spectra thus separated are incident on the image pickup elements 121a and 121b via the mirror 54 and the second prism 56, and are expanded into (x, λ) information as spectroscopic data.

[0107] The pixel size (nm / Pixel) of the second imaging elements 121a and 121b is set to, for example, 2 nm or more and 20 nm or less, but is not limited to this. This dispersion value may be realized optically by the pitch of the diffraction grating 35, or may be realized by using hardware binning of the imaging elements 121a and 121b.

[0108] The stage 140 and the scanning mechanism 50 constitute an XY stage, and the object S is moved in the X-axis direction and the Y-axis direction to acquire a fluorescent image of the object S. In WSI (Whole Slide Imaging), the object S is scanned in the Y-axis direction, then moved in the X-axis direction, and then scanned again in the Y-axis direction, and this operation is repeated (see FIG. 11).

[0109] The non-fluorescence observation section 70 is composed of a light source 71, a dichroic mirror 43, an objective lens 44, a condenser lens 72, an image pickup element 73, etc. In the non-fluorescence observation system, an observation system using dark-field illumination is shown in FIG.

[0110] The light source 71 is disposed below the stage 20, and irradiates the target S on the stage 140 with illumination light from the opposite side to the line illuminations Ex1 and Ex2. In the case of dark-field illumination, the light source 71 illuminates from outside the NA (numerical aperture) of the objective lens 44, and the light diffracted by the target S (dark-field image) is captured by the image sensor 73 via the objective lens 44, the dichroic mirror 43, and the condenser lens 72. By using dark-field illumination, even a seemingly transparent sample such as a fluorescently stained sample can be observed with contrast.

[0111] The dark-field image may be observed simultaneously with the fluorescence and used for real-time focusing. In this case, the illumination wavelength may be selected so as not to affect the fluorescence observation. The non-fluorescence observation unit 70 is not limited to an observation system that acquires a dark-field image, but may be configured with an observation system that can acquire non-fluorescence images such as bright-field images, phase-contrast images, phase images, and in-line hologram images. For example, various observation methods such as the Schlieren method, phase-contrast method, polarized observation method, and epi-illumination method can be adopted as a method for acquiring a non-fluorescence image. The position of the illumination light source is not limited to below the stage, and may be above the stage or around the objective lens. In addition to a method of performing focus control in real time, other methods such as a pre-focus map method in which focus coordinates (Z coordinates) are recorded in advance may be adopted.

[0112] (2-3) Control section

[0113] The control unit 130 may, for example, control the imaging of the target S by the second imaging element 121 based on the determination result by the determination unit 115, or control the processing of the image obtained by the second imaging element 121. The control unit 130 performs these controls based on the determination result without transmitting image data from the first imaging element 111 to the control unit 130, thereby reducing the effect of the rate limiting factor described above. In addition, the control unit 130 may selectively control the second imaging element 121 based on the image data itself acquired by the first imaging element 111 and control the second imaging element 121 based on the discrimination result by the discrimination unit 115 .

[0114] The control unit 130 may adjust the positional relationship between the stage 140 and the second imaging element 121, for example, based on the discrimination result by the discrimination unit 115, in order to control the imaging of the target S by the second imaging element 121. For this adjustment, for example, the control unit 130 may move the stage 140 or the second imaging element 121, or may move both the stage 140 and the second imaging element 121.

[0115] The control unit 130 may adjust, for example, the second observation optical system 122 and / or the second illumination optical system 123 for imaging control of the target S by the second imaging element 121. For example, the control unit 130 may switch the objective lens included in the second observation optical system 122 to an objective lens with a different magnification for the imaging control. Furthermore, the control unit 130 may control a light source included in the second illumination optical system 123 to change the illumination light for the imaging control.

[0116] Fig. 14 shows an example of a block diagram of the control unit 130. As shown in Fig. 14, the control unit 130 may further include at least one selected from an imaging sequence control unit 131, an exposure control unit 132, a light amount control unit 133, an image processing unit 134, and a calculation parameter control unit 135.

[0117] The control unit 130 may further include an imaging sequence control unit 131 that controls the imaging order based on, for example, features related to the region of the target S. The imaging sequence control unit 131 enables multiple parts of the target S to be imaged more efficiently or more appropriately.

[0118] The imaging sequence control unit 131 can control imaging so that the seams of the multiple regions do not overlap the region of interest in the target. Since the seams of the multiple regions may deteriorate image quality, the imaging control is performed as described above to prevent deterioration of image quality in the region of interest.

[0119] The imaging sequence control unit 131 may control imaging so that the region of interest in the target is imaged multiple times. The imaging regions in the multiple imagings may preferably be shifted from each other, but may be set to include the region of interest. More preferably, the imaging regions in the multiple imagings may be set so that the region of interest does not overlap the edge of the imaging region in each imaging.

[0120] The controller 130 may further include an exposure controller 132 for controlling gain and / or exposure time. The exposure controller 132 may control the gain and / or exposure time based on characteristics related to the region of interest. The exposure controller may, for example, control the gain and / or exposure time on a pixel-by-pixel and / or wavelength-by-wavelength basis.

[0121] The exposure control unit 132 may control the gain and / or the exposure time, for example, by controlling any one of the second imaging element 121, the second observation optical system 122, or the second illumination optical system 123. For example, the exposure control unit 132 may control the exposure time by controlling a shutter (which may be included in any one of the second imaging element 121, the second observation optical system 122, or the second illumination optical system 123). In addition, the exposure control unit 132 may control the gain by controlling the second imaging element 121 (for example, an AD converter included in the second imaging element 121).

[0122] The control unit 130 may further include a light amount control unit 133 that controls the amount of light from the light source. The light amount control unit 133 may control the amount of light from the second illumination optical system 123 of the second imaging unit 120, for example.

[0123] The control unit 130 can include an image processing unit 134 that processes the image obtained by the second imaging element 121, and a calculation parameter control unit 135 used in the image processing.

[0124] The image processing by the image processing unit 134 may include, for example, a fluorescence separation process. That is, the image processing unit 134 may perform, for example, a fluorescence separation process on an image obtained by the second imaging element 121. The fluorescence separation process may be, for example, a separation process for separating a fluorescent signal derived from a fluorophore labeling a biological tissue and an autofluorescence signal (for example, an autofluorescence signal derived from a biological tissue, an autofluorescence signal derived from an objective lens, and an autofluorescence signal derived from an immersion oil), or a separation process for separating fluorescent signals derived from each of a plurality of fluorophores labeling a biological tissue from each other. The details of the fluorescence separation process by the image processing unit 134 will be described later.

[0125] The calculation parameter control unit 135 adjusts the calculation parameters used in the image processing by the image processing unit 134. Preferably, the calculation parameter control unit 135 performs the adjustment based on the result of the discrimination by the discrimination unit 115.

[0126] The calculation parameter control unit 135 can adjust, for example, image quality correction parameters. Examples of image quality correction parameters include white balance adjustment parameters, saturation correction parameters, and contrast correction parameters.

[0127] Furthermore, the calculation parameter control unit 135 can adjust parameters used in the fluorescence separation process. For example, the calculation parameter control unit can adjust a calculation matrix for the fluorescence separation process.

[0128] The functions of the control unit 130 (for example, the functions of the imaging sequence control unit, exposure control unit, light amount control unit, image processing unit, and calculation parameter control unit described above) may be realized by an information processing device. An example of the hardware configuration of the information processing device for realizing the functions of the control unit 130 will be described below with reference to FIG. 26. Note that the configuration of the information processing device is not limited to the following.

[0129] 26 includes a CPU (Central Processing Unit) 1001, a RAM 1002, and a ROM 1003. The CPU 1001, the RAM 1002, and the ROM 1003 are connected to one another via a bus 1004. An input / output interface 1005 is further connected to the bus 1004.

[0130] To the input / output interface 1005, a communication device 1006, a storage unit 1007, a drive 1008, an output unit 1009, and an input unit 1010 are connected.

[0131] The communication device 1006 connects the information processing device 1000 to a network 1011 by wire or wirelessly. The communication device 1006 enables the information processing device 1000 to acquire various data (e.g., image data, etc.) via the network 1011. The acquired data can be stored in, for example, a disk 1007. The type of the communication device 1006 may be appropriately selected by one skilled in the art.

[0132] The memory unit 1007 can store an operating system (e.g., WINDOWS (registered trademark), UNIX (registered trademark), LINUX (registered trademark), ANDROID (registered trademark), or iOS (registered trademark), etc.), a program for executing the image acquisition method according to the present technology in a microscope device, and various other programs, as well as various data (e.g., image data and feature data, etc.).

[0133] The drive 1008 can read data (e.g., image data and feature data) or programs recorded on a recording medium and output the data to the RAM 1002. The recording medium is, for example, a microSD memory card, an SD memory card, or a flash memory, but is not limited to these.

[0134] The output unit 1009 causes the image display unit to output image display light based on the image data. The input unit 1010 accepts operations of the microscope device by, for example, a user.

[0135] (2-4) Stage

[0136] The stage 140 holds a subject S including a biological tissue. For example, the subject S may be placed on the stage 140, or the subject S may be attached to the stage 140. The stage 140 may be configured to be movable. The movement of the stage 140 may be controlled by the control unit 130, for example.

[0137] (2-5) First example of imaging control (fluorescence separation processing)

[0138] In the fluorescent observation of biological tissue, a fluorescent separation process may be performed. The microscope device of the present technology is suitable for use in fluorescent observation in which a fluorescent separation process is performed. Below, an example of imaging of fluorescently labeled biological tissue by the microscope device of the present technology will be described with reference to Figs. 1A and 1B and Fig. 15. Figs. 1A and 1B are as described above, and Fig. 15 is an example of a processing flow diagram of the imaging process.

[0139] (2-5-1) Processing flow

[0140] 15, an imaging process using the microscope apparatus of the present technology is started. Prior to the start of the imaging process, as shown in FIG 1A, a target S including a biological tissue is placed on a stage 140. In addition, the stage 140 is placed at a position where imaging by the first imaging unit 110 is possible.

[0141] In a first imaging process in step S102, an image of the target S is captured by the first imaging unit 110. The imaging can be performed, for example, as follows.

[0142] In step S102, the first illumination optical system 113 irradiates light onto the target S on the stage 140. The light may be, for example, excitation light that excites a phosphor contained in the target S, in particular a phosphor that marks the observation target region. In step S102, the control unit 130 or a user may control the first illumination optical system 113 to irradiate the light.

[0143] In step S102, the first imaging element 111 captures an image of the target S via the first observation optical system 112 while the target S is irradiated with the light, thereby acquiring image data. The image data may be image data of the entire target S or a part of the target S. The image data may be the image data itself obtained by imaging, or compressed image data obtained by compressing the image data obtained by imaging, or both the image data and the compressed image data. The imaging may be performed in particular such that the first imaging element 111 captures an image of at least a part of a biological tissue included in the target S. The imaging in step S102 may be performed by the control unit 130 or a user controlling the first imaging element 111.

[0144] In the discrimination process of step S103, the first imaging element 111 (particularly, the discrimination unit 115 provided in the first imaging element 111) discriminates features related to the target S based on the image data acquired in step S102. For example, the discrimination unit 115 can discriminate the attributes of the target S or the area of ​​the target S, or both.

[0145] For example, the discrimination unit 115 may determine, based on the image data, which organ or part the biological tissue contained in the target S originates from, or which tissue the biological tissue belongs to. For example, the discrimination unit 115 may determine whether the biological tissue is brain, oral cavity, lung, trachea, heart, esophagus, stomach, intestine, kidney, liver, pancreas, mammary gland, or skin tissue. The discrimination unit 115 may also determine whether the biological tissue belongs to epithelial tissue, connective tissue, muscle tissue, or nerve tissue.

[0146] The discrimination unit 115 may determine the size and / or number of components in the biological tissue based on the image data. The components may be, for example, epithelial tissue, connective tissue, muscle tissue, or nerve tissue, or may be cells or cell organelles. More specifically, the discrimination unit 115 may determine the size of adipose tissue in the biological tissue, or the size and / or number of red blood cells in the biological tissue.

[0147] The discrimination unit 115 may discriminate whether the biological tissue has a lesion, discriminate which disease the biological tissue has, or discriminate the progress of the disease, based on the image data.

[0148] The discrimination unit 115 may discriminate the autofluorescence level based on the image data. The autofluorescence may be, for example, autofluorescence derived from the biological tissue, autofluorescence derived from an encapsulant for encapsulating the biological tissue, or autofluorescence derived from the imaging environment (for example, autofluorescence derived from the objective lens of the first observation optical system 112 or autofluorescence of immersion oil). The encapsulant may have a different spectrum or brightness depending on, for example, a medium or a drug used when encapsulating the tissue between a slide glass and a cover glass.

[0149] The discrimination unit 115 discriminates the features of the target S, preferably using a trained model. The trained model may be generated, for example, by using one or more combinations of image data of the target including biological tissue and features of the target as teacher data. With regard to the trained model, the explanation in "(2-1-1) Discrimination Performed by the First Imaging Element" above also applies to this example.

[0150] In step S103, the first imaging element 111 transmits the discrimination result by the discrimination unit 115 to the control unit 130. In step S103, for example, in addition to the discrimination result, image data and / or compressed image data of the target S may be transmitted to the control unit 130, but preferably the image data and / or compressed image data of the target S is not transmitted to the control unit 130. In this way, the first imaging element 111 outputs only the discrimination result to the control unit 130, thereby reducing the influence of rate limitations due to the output interface of the first imaging element 111, etc. Furthermore, the control unit 130 receives only the discrimination result, thereby reducing the processing load on the control unit 130.

[0151] In the control process of step S104, the control unit 130 receives the determination result transmitted from the first imaging element 111. Based on the determination result, the control unit 130 can generate imaging control data for controlling imaging by the second imaging unit 120, image processing control data for controlling processing of image data obtained by imaging by the second imaging unit 120, or both of these data.

[0152] The imaging control data includes, for example, imaging control data related to the manner of divided imaging by the second imaging element 121, and more specifically, may include, for example, data related to the manner of division, the order of imaging of the divided regions, or imaging conditions (for example, magnification, gain, exposure time, and light amount, etc.). Preferably, the control unit 130 may set the manner of division so that a region of interest in the subject S (for example, a disease region, particularly a tumor region) does not overlap with a seam of the divided regions. Although image quality may be degraded at the seam of the divided regions, the control unit can prevent image quality degradation in the region of interest by setting the manner of division in this manner. The imaging control data can be generated by, for example, the imaging sequence control unit 131, the exposure control unit 132, or the light amount control unit 133 described above.

[0153] The image processing control data may include, for example, data for controlling image processing performed by the control unit 130 in step S106 described below. More specifically, the image processing control data may be data on how to join the multiple image data acquired in step S105, data on how to process each of the multiple image data, and data on the fluorescence separation process in step S106. The image processing control data can be generated, for example, by the image processing unit 134 or the calculation parameter control unit described above.

[0154] In the second imaging step of step S105, the control unit 130 may control the second imaging unit 120 based on the imaging control data generated in step S104 to cause the second imaging unit 120 to image the target S. The imaging will be described in more detail below. Prior to the imaging, the control unit 130 moves the stage 140 on which the object S is placed to a position where the object S can be imaged by the second imaging unit 120. An example of the state after the movement is shown in Fig. 1B. Alternatively, the control unit 130 may drive a device that moves or operates the object S, and the device may move the object S from the stage 140 to a stage separately provided in the second imaging unit 120.

[0155] For example, in step S105, the second illumination optical system 123 irradiates the target S on the stage 140 with excitation light of a phosphor contained in the target S. The excitation light may be the same as or different from the light irradiated in step S102. In step S105, the control unit 130 or a user may control the second illumination optical system 123 to irradiate the excitation light.

[0156] In step S105, while the light is irradiated onto the object S, the second imaging element 121 may image the object S at a magnification different from that of the first imaging element 111, based on the imaging control data. For example, the second imaging element 121 may divide an area imaged by the first imaging element 111 into multiple areas and image the area, based on the imaging control data. For example, imaging by the second imaging element 121 is performed multiple times, thereby obtaining multiple image data (hereinafter also referred to as "divided image data"). The divided imaging may be performed, for example, by a tiling method or a line scan method.

[0157] In step S105, the second imaging element 121 receives the fluorescence generated from the target S by irradiating it with excitation light via the second observation optical system 122. The second observation optical system 122 may include a wavelength separation element (e.g., a filter or a spectroscope) to separate and receive a desired fluorescence.

[0158] The second imaging element 121 transmits to the control unit 130 the image data (one or a plurality of divided image data, particularly a plurality of divided image data) acquired by imaging in step S105.

[0159] In the image processing step of step S106, the control unit 130 receives the image data (one or a plurality of divided image data, particularly a plurality of divided image data) acquired in step S105, and performs image processing using the image data. The control unit 130 can perform the image processing based on, for example, the image processing control data.

[0160] For example, in step S106, the control unit 130 may use two or more pieces of the divided image data to generate one piece of image data related to the area of ​​the two or more pieces of divided image data. For example, the control unit 130 may use the multiple pieces of image data to generate image data of the entire area captured in step S102.

[0161] Furthermore, in step S106, the control unit 130 may perform fluorescence separation processing on the image data (one or multiple divided image data, particularly multiple divided image data). In step S106, the control unit 130 may perform fluorescence separation processing on the image data of the entire area imaged in step S102. The fluorescence separation processing may be, for example, a separation processing that separates a fluorescent signal derived from a fluorophore that has labeled the biological tissue from an autofluorescence signal, or a separation processing that separates fluorescent signals derived from each of multiple fluorophores that have labeled the biological tissue from each other. Details of the fluorescence separation processing will be described separately below in "(2-5-2) Fluorescence Separation Processing".

[0162] The image data generated by the image processing in step S106 may be stored, for example, in a storage medium provided in the microscope device 100, or may be output to an external storage medium connected to the microscope device 100 via a wired or wireless connection.

[0163] In step S107, the microscope apparatus 100 ends the imaging process. By the above image acquisition process, the second imaging element 121 is controlled without transmitting image data from the first imaging element 111 to the control unit 130. This enables efficient or appropriate image acquisition.

[0164] (2-5-2) Fluorescence separation processing

[0165] In step S106, the image processing unit 134 included in the control unit 130 may perform, for example, a fluorescence separation process to separate an autofluorescence signal of the biological tissue and a fluorescence signal of at least one phosphor from image data of the biological tissue labeled with at least one phosphor. Also, the image processing unit 134 may perform a fluorescence separation process to separate the fluorescence signals of each of the two or more phosphors from image data of a biological dye labeled with two or more phosphors. These fluorescence separation processes may be performed based on, for example, information on the biological tissue and information on the phosphors.

[0166] Here, the term "information on biological tissue" (hereinafter, for convenience, referred to as "biological tissue information") refers to information including measurement channel and spectrum information specific to the autofluorescence components contained in biological tissue. The "measurement channel" is a concept indicating one or more elements that constitute an autofluorescence signal. More specifically, an autofluorescence spectrum is a mixture of the fluorescence spectra of one or more autofluorescence components. For example, FIG. 24 shows fluorescence spectra including autofluorescence when Alexa Fluor 488 (denoted as "Alexa 488" in the figure), Alexa Fluor 555 (denoted as "Alexa 555" in the figure), and Alexa Fluor 647 (denoted as "Alexa 647" in the figure) are used as fluorescent reagents. The autofluorescence components include NADH (nicotinamide adenine dinucleotide reduced form), FAD (flavin adenine dinucleotide), and porphin, and the mixture of the fluorescence spectra of these autofluorescence components is included in FIG. 24 as the autofluorescence spectrum. In the example of FIG. 24, the measurement channel according to the present technology is a concept that refers to NADH, FAD, and Porphin that constitute the autofluorescence spectrum, and a total of three channels are used (in other words, the measurement channel according to the present technology is information that indicates the autofluorescence components contained in the biological tissue). Since the number of autofluorescence components varies depending on the biological tissue, the measurement channel is linked to each biological tissue and managed as biological tissue information. In addition, the "spectral information" included in the biological tissue information is information regarding the autofluorescence spectrum of each of the autofluorescence components contained in each biological tissue. Note that the content of the biological tissue information is not limited to the above. The biological tissue information will be explained in more detail later.

[0167] "Information about phosphors" (hereinafter referred to as "phosphor information" for convenience) is information that includes spectral information of the fluorescent components contained in the phosphors. The "spectral information" included in the phosphor information is information about the fluorescent spectrum of each of the fluorescent components contained in each phosphor. Note that the content of the phosphor information is not limited to the above. The phosphor information will be explained in more detail later.

[0168] The image processing unit 134 estimates the autofluorescence signal derived from the biological tissue based on the measurement channel and spectrum information included in the biological tissue information and the spectrum information included in the phosphor information, and can extract the fluorescent signal of the phosphor by removing the autofluorescence signal from the image data. For example, the image processing unit 134 can extract the fluorescent signals of Alexa Fluor 488, Alexa Fluor 555, and Alexa Fluor 647 as shown in FIG. 25 by removing the autofluorescence signals due to the autofluorescence components NADH, FAD, and porphin from the image information.

[0169] The above-mentioned processing by the image processing unit 134 reduces the burden on the user because it is not necessary to measure the fluorescent signal of unlabeled biological tissue in order to perform the subtraction processing of the autofluorescence signal. Furthermore, the image processing unit 134 can separate the autofluorescence signal and the fluorescent signal with higher accuracy by using both biological tissue information and fluorescent material information, rather than using only either biological tissue information or fluorescent material information.

[0170] In addition, in the past, in order to obtain the fluorescence signal to be measured, a method of amplifying the fluorescence signal to be measured to a level where the autofluorescence signal derived from the biological tissue can be ignored has been widely and generally performed, but it is difficult to obtain the autofluorescence signal derived from the biological tissue by the above-mentioned fluorescence separation processing. On the other hand, the image processing unit 134 can extract the autofluorescence signal derived from the biological tissue from the image data. Therefore, the image processing unit 134 can perform segmentation (or area division) to recognize the area of ​​the object (e.g., cell, intracellular structure (cytoplasm, cell membrane, nucleus, etc.), or tissue (tumorous part, non-tumorous part, connective tissue, blood vessel, vascular wall, lymphatic vessel, fibrotic structure, necrosis, etc.)) included in the image data based on the distribution of the autofluorescence signal and the fluorescence signal in the image data.

[0171] Furthermore, since the autofluorescence signal of the entire biological tissue changes depending on the fixation state of the biological tissue (e.g., fixation method, etc.), the image processing unit 134 can analyze (evaluate) the fixation state of the biological tissue based on the extracted autofluorescence signal. More specifically, the image processing unit 134 can analyze the fixation state of the biological tissue based on the component ratio of the autofluorescence signals of two or more components that emit autofluorescence.

[0172] More specifically, the image processing unit 134 recognizes one or more elements that constitute the autofluorescence signal based on the measurement channels included in the biological tissue information. For example, the image processing unit 134 recognizes one or more autofluorescence components that constitute the autofluorescence signal. Then, the image processing unit 134 predicts or estimates the autofluorescence signal included in the image data using the spectral information of these autofluorescence components included in the biological tissue information. Then, the image processing unit 134 separates the autofluorescence signal and the fluorescent signal from the image data based on the spectral information of the fluorescent components of the phosphors included in the phosphor information and the predicted or estimated autofluorescence signal.

[0173] Here, when the biological tissue is stained with two or more phosphors, the image processing unit 134 separates the fluorescent signals of each of these two or more phosphors from the image data (or the fluorescent signal after separation from the autofluorescence signal) based on the biological tissue information and the phosphor information. For example, the image processing unit 134 separates the fluorescent signals of each phosphor from the entire fluorescent signal after separation from the autofluorescence signal, using spectral information of the fluorescent components of each phosphor included in the phosphor information.

[0174] Furthermore, when the autofluorescence signal is composed of two or more autofluorescence components, the image processing unit 134 separates the autofluorescence signal of each autofluorescence component from the image data (or the autofluorescence signal after separation from the fluorescent signal) based on the biological tissue information and the phosphor information. For example, the image processing unit 134 separates the autofluorescence signal of each autofluorescence component from the entire autofluorescence signal after separation from the fluorescent signal, using spectral information of each autofluorescence component included in the biological tissue information.

[0175] The image processing unit 134, which has separated the fluorescent signal and the autofluorescent signal, performs various processes using these signals. For example, the image processing unit 134 may extract the fluorescent signal from the image data of the other biological tissue by performing subtraction processing (also referred to as "background subtraction processing") on the image data of the other biological tissue using the separated autofluorescent signal. When there are multiple biological tissues that are identical or similar in terms of the tissue used for the biological tissue, the type of disease to be treated, the attributes of the subject, and the lifestyle of the subject, the autofluorescent signals of these biological tissues are likely to be similar. The similar specimens referred to here include, for example, a tissue section before staining of a stained tissue section (hereinafter referred to as a section), a section adjacent to the stained section, a section different from the stained section in the same block (sampled from the same place as the stained section), or a section in a different block (sampled from a different place from the stained section) in the same tissue, a section taken from a different patient, etc. Therefore, when the image processing unit 134 can extract an autofluorescence signal from a certain biological tissue, the image processing unit 134 may extract a fluorescence signal from the image data of the other biological tissue by removing the autofluorescence signal from the image data of the other biological tissue. Furthermore, when calculating the S / N value using the image data of the other biological tissue, the image processing unit 134 can improve the S / N value by using the background after removing the autofluorescence signal.

[0176] In addition to the background subtraction process, the image processing unit 134 can perform various processes using the separated fluorescent signal or autofluorescent signal. For example, the image processing unit 134 can use these signals to analyze the fixation state of the biological tissue, or perform segmentation (or area division) to recognize the area of ​​objects (e.g., cells, intracellular structures (cytoplasm, cell membrane, nucleus, etc.), or tissues (tumorous area, non-tumorous area, connective tissue, blood vessels, blood vessel walls, lymphatic vessels, fibrotic structures, necrosis, etc.)) included in the image data. The analysis and segmentation of the fixation state of the biological tissue will be described in detail later.

[0177] The image processing unit 134 may generate (reconstruct) image data based on the fluorescent signal or the autofluorescent signal separated by the above-mentioned fluorescence separation process. For example, the image processing unit 134 may generate image data including only the fluorescent signal, or generate image data including only the autofluorescent signal. In this case, when the fluorescent signal is composed of a plurality of fluorescent components, or the autofluorescent signal is composed of a plurality of autofluorescent components, the image processing unit 134 may generate image data for each component. Furthermore, when the image processing unit 134 performs various processes (e.g., analysis of the fixation state of the biological tissue, segmentation, or calculation of the S / N value) using the fluorescent signal or the autofluorescent signal after separation, the image processing unit 134 may generate image data showing the results of those processes. The image generation by the image processing unit 134 visualizes distribution information of the fluorophores that label the target molecules, that is, the two-dimensional spread, intensity, and wavelength of the fluorescence, and the respective positional relationships, and can improve visibility for users such as doctors and researchers, especially in the tissue image analysis area where the information on the target substance is complex.

[0178] The image processing unit 134 may also control to distinguish the fluorescent signal from the autofluorescent signal based on the fluorescent signal or the autofluorescent signal separated by the above-mentioned fluorescence separation process, and generate image data. Specifically, the image data may be generated by controlling to improve the brightness of the fluorescent spectrum of the fluorescent substance that labels the target molecule, to extract and discolor only the fluorescent spectrum of the labeled fluorescent substance, to extract the fluorescent spectra of two or more fluorescent substances from a biological tissue labeled with two or more fluorescent substances and discolor each of them to a different color, to extract only the autofluorescent spectrum of the biological tissue and divide or subtract it, to improve the dynamic range, etc. This allows the user to clearly distinguish the color information derived from the fluorescent substance bound to the target substance of interest, and improves the user's visibility.

[0179] (2-6) Second example of imaging control (imaging sequence control)

[0180] When observing biological tissue under a microscope, for example, the amount of data generated can be reduced by capturing an image of the target biological tissue with high image quality and capturing an image of other regions with low image quality. The microscope device of the present technology is suitable for selectively capturing an image of the target biological tissue with high image quality. For example, the microscope device of the present technology makes it possible to capture an image of a region including a target biological tissue region (hereinafter also referred to as a "region of interest") with high image quality and capture an image of a region not including the region of interest with low image quality. In this way, the microscope device of the present technology can change the imaging conditions according to the imaging region, thereby reducing the amount of image data generated. An example of such imaging processing will be described below.

[0181] An example of an imaging process in which a microscope device of the present technology extracts a region of interest and controls an imaging sequence of the microscope device based on the result of the extraction will be described with reference to FIGS. 1A and 1B and FIG. 15.

[0182] Steps S101 and S102 are as explained in "(2-5) First example of imaging control (fluorescence separation processing)" above, and the explanation also applies to this example.

[0183] In the discrimination process of step S103, the first imaging element 111 (particularly the discrimination unit 115) discriminates features related to the target S based on the image data acquired in step S102. For example, the discrimination unit 115 may discriminate regarding the area of ​​the target S. Furthermore, the discrimination unit 115 may discriminate regarding the attributes of the target S in addition to discriminating regarding the area of ​​the target S.

[0184] For example, the discrimination unit 115 may perform discrimination regarding a region of the target S based on the image data. The discrimination unit 115 may extract a region of interest in the target S, for example, through the discrimination. The region of interest may be, for example, a region of a specific cell or tissue, or a lesion region. The discrimination unit 115 may also extract a region of an unintended object through the discrimination. The unintended object may be, for example, dust or an artifact.

[0185] The discrimination unit 115 may generate image data in which the area captured by the first imaging element 111 is separated into an area of ​​interest and other areas based on the result of discrimination regarding the area of ​​the target S, for example. For this discrimination, for example, a predetermined class may be assigned to the area of ​​interest, and a class different from the above class may be assigned to the other areas. These classes may be previously associated with the characteristics of each of these areas.

[0186] The discrimination unit 115 discriminates the features of the target S, preferably using a trained model. The trained model may be generated, for example, by using one or more combinations of image data of the target including biological tissue and features of the target as teacher data. With regard to the trained model, the explanation in "(2-1-1) Discrimination Performed by the First Imaging Element" above also applies to this example.

[0187] In step S103, the first imaging element 111 transmits the result of the discrimination by the discriminator 115 to the controller .

[0188] In the control process of step S104, the control unit 130 receives the determination result transmitted from the first imaging element 111. Based on the determination result, the control unit 130 can generate imaging control data for controlling imaging by the second imaging unit 120, image processing control data for controlling processing of image data obtained by imaging by the second imaging unit 120, or both of these data.

[0189] The control unit 130 may generate, as the imaging control data, imaging control data related to a method of dividing the imaging area by the second imaging element 121. The imaging control data may include data related to a method of setting the imaging area, a method of dividing the area, an order of imaging the divided areas, imaging conditions (e.g., magnification, gain, exposure time, and light amount), and the like.

[0190] The imaging control data generated by the control unit 130 may include, for example, imaging conditions for an area including the area of ​​interest and imaging conditions for an area other than the area of ​​interest. For example, these imaging conditions may be set so that the resolution or magnification in imaging the area including the area of ​​interest is higher than the resolution or magnification in imaging the area other than the area of ​​interest.

[0191] The imaging control data generated by the control unit 130 may include, for example, data regarding a division method. For example, the control unit 130 may divide the area imaged by the first imaging element 111 into an area including the area of ​​interest and an area not including the area of ​​interest based on the determination result in step S203.

[0192] For example, as shown in FIG. 16, the control unit 130 can divide the region R imaged by the first imaging element 111 into a region R1 that includes the region of interest ROI and a region R2 that does not include the region of interest ROI.

[0193] For example, as shown in Fig. 16, the regions R1 and R2 may be set so that there is a seam region Rs where a part of the region R1 and a part of the region R2 overlap. In this manner, the control unit 130 may preferably set the seams of the divided regions to overlap. The seam region allows for more accurate reconstruction when reconstructing the entire image by stitching together the images of the multiple regions.

[0194] For example, as shown in Fig. 16, the regions R1 and R2 may be set so that the region of interest ROI is not included in the seam region Rs. In this manner, the control unit 130 may preferably set the multiple regions such that the regions of interest do not overlap with the seam regions between the multiple divided regions. Note that the image quality of the seam region Rs may deteriorate. Therefore, as shown in FIG. 17, when the seam region Rs includes the region of interest ROI, the image quality of the region of interest ROI may deteriorate.

[0195] Further, the control unit 130 may set a plurality of imaging regions R3 for one region of interest ROI, as shown in FIG. 18. In FIG. 18, imaging regions R3-1, R3-2, and R3-3 are set for the region of interest ROI-1, that is, all of the imaging regions R3-1, R3-2, and R3-3 include the region of interest ROI-1. This allows the image quality of the region of interest ROI-1 to be improved. In this manner, the control unit 130 may preferably set imaging conditions so that imaging is performed multiple times for one region of interest. For a region that does not include a region of interest ROI, one imaging region R4 may be set as shown in FIG.

[0196] Furthermore, when imaging is performed multiple times for one region of interest ROI-1, the control unit 130 may set imaging regions R3-1, R3-2, and R3-3 for the multiple imaging times to be shifted from each other, as shown in Fig. 18. This allows the image quality of the region of interest ROI-1 to be improved. In this manner, the control unit 130 may preferably set multiple imaging regions for one region of interest that include the region of interest and are shifted from each other.

[0197] In step S104, the control unit 130 transmits the generated imaging control data to the second imaging unit 120 (for example, the second imaging element 121, etc.).

[0198] In a second imaging step of step S105, the second imaging unit 120 receives the imaging control data generated in step S104, and images the target S based on the imaging control data. Prior to the imaging, the control unit 130 moves the stage 140 on which the target S is placed to a position where the second imaging unit 120 can image the target S.

[0199] For example, in step S105, the second illumination optical system 123 irradiates the target S on the stage 140 with excitation light of a phosphor contained in the target S. The excitation light may be the same as or different from the light irradiated in step S102. In step S105, the control unit 130 or a user may control the second illumination optical system 123 to irradiate the excitation light.

[0200] In step S105, in a state in which the light is irradiated onto the object S, the second imaging element 121 may, for example, image the object S at a magnification different from that of the first imaging element 111 and acquire image data based on the imaging control data. For example, the second imaging element 121 may, based on the imaging control data, divide the area imaged by the first imaging element 111 into a plurality of areas and image at least one of the plurality of areas to acquire image data. That is, the area imaged by the first imaging element 111 may be divided into a plurality of areas and imaged. In this way, imaging by the second imaging element 121 may be performed a plurality of times, thereby acquiring a plurality of image data.

[0201] In step S105, the second imaging element 121 receives the fluorescence generated from the target S by irradiating it with excitation light via the second observation optical system 122. The second observation optical system 122 may include a wavelength separation element (e.g., a filter or a spectroscope) to separate and receive a desired fluorescence.

[0202] The second imaging element 121 transmits to the control unit 130 image data (one or multiple image data, particularly multiple image data) acquired by imaging in step S105.

[0203] In the image processing step of step S106, the control unit 130 receives the image data (one or multiple image data, particularly multiple image data) acquired in step S105, and performs image processing using the multiple image data.

[0204] For example, in step S106, the control unit 130 may generate one image data set using two or more image data sets among the plurality of image data sets. For example, the control unit 130 may generate image data sets of the entire area captured in step S102 using the plurality of image data sets.

[0205] The image data generated by the image processing in step S106 may be stored, for example, in a storage medium provided in the microscope device 100, or may be stored in an external storage medium connected to the microscope device 100 via a wired or wireless connection.

[0206] In step S107, the microscope apparatus 100 ends the imaging process.

[0207] (2-7) Third example of imaging control (imaging parameter control)

[0208] If an image is captured without setting an appropriate exposure time for the signal strength, the captured image may have blown-out highlights or blocked-up shadows. Therefore, it is desirable to adjust the exposure time. The microscope device of the present technology makes it possible to perform such an adjustment.

[0209] An example of an imaging process in which a microscope device of the present technology extracts a region of interest and controls imaging parameters (e.g., exposure time and / or gain) of the microscope device based on the results of the extraction is described with reference to Figures 1A and B and Figure 15.

[0210] Steps S101 and S102 in FIG. 15 are as explained in "(2-5) First example of imaging control (fluorescence separation processing)" above, and the explanation also applies to this example.

[0211] In the discrimination process of step S103, the first imaging element 111 (particularly the discrimination unit 115) discriminates features related to the target S based on the image data acquired in step S102. For example, the discrimination unit 115 may discriminate regarding the area of ​​the target S. Furthermore, the discrimination unit 115 may discriminate regarding the attributes of the target S in addition to discriminating regarding the area of ​​the target S.

[0212] For example, the discrimination unit 115 can make a discrimination regarding the area of ​​the target S based on the image data. The discrimination unit 115 can extract high signal intensity regions and low signal intensity regions in the object S, for example, by the discrimination. A high signal intensity region is a region of the region imaged by the first imaging element 111 in which the signal intensity is equal to or greater than a predetermined value. A low signal intensity region is a region of the region imaged by the first imaging element 111 in which the signal intensity is less than a predetermined value. The predetermined value may be set in advance. For example, the region of interest is labeled with a fluorescent material, and therefore has a higher signal intensity than other regions. Furthermore, the discrimination unit 115 may divide the area captured by the first imaging element 111 into three or more areas according to the signal intensity values, instead of the above two areas. A different signal intensity numerical range may be assigned in advance to each of the three or more areas.

[0213] The discrimination unit 115 discriminates the features of the target S, preferably using a trained model. The trained model may be generated, for example, by using one or more combinations of image data of the target including biological tissue and features of the target as teacher data. With regard to the trained model, the explanation in "(2-1-1) Discrimination Performed by the First Imaging Element" above also applies to this example.

[0214] In step S103, the first imaging element 111 transmits the result of the discrimination by the discriminator 115 to the controller .

[0215] In the control process of step S104, the control unit 130 receives the determination result transmitted from the first imaging element 111. Based on the determination result, the control unit 130 can generate imaging control data for controlling imaging by the second imaging unit 120, image processing control data for controlling processing of image data obtained by imaging by the second imaging unit 120, or both of these data.

[0216] The control unit 130 may generate, as the imaging control data, imaging control data related to a method of dividing the imaging area by the second imaging element 121. The imaging control data may include data related to a method of setting the imaging area, a method of dividing the area, an order of imaging the divided areas, imaging conditions (e.g., magnification, gain, exposure time, and light amount), and the like.

[0217] The imaging control data generated by the control unit 130 may include, for example, imaging conditions for a high signal region and imaging conditions for a low signal region. For example, the imaging conditions may be set so that the exposure time for imaging the high signal region is shorter than the exposure time for imaging the low signal region. By performing imaging using the second imaging element 121 according to these imaging conditions, it is possible to prevent whiteout or blackout.

[0218] Furthermore, for example, when fluorescence of a plurality of wavelengths is detected, for example, when the signal intensity differs depending on the wavelength, the control unit 130 can set a different exposure time for each wavelength.

[0219] For example, as shown in FIG. 19A, assume that a region R imaged by the first imaging element 111 includes a plurality of regions of interest R5 and other regions R6. The region of interest R5 is a region in which cells labeled with a fluorescent material exist, and is brighter than the other regions R6. The image data of FIG. 19A is discriminated by the discrimination unit 115, and as shown in FIG. 19B, the discrimination unit 115 extracts the region of interest R5 and divides the region R as shown by the broken lines. As shown in FIG. 19C, the control unit 130 sets the exposure time for each divided region. Specifically, the control unit 130 sets the exposure time t-Δt for the divided region including the region of interest R5, and sets the exposure time t for the other divided regions. That is, the exposure time for the divided region including the region of interest is shorter than the exposure time for the other divided regions. In this way, the control unit 130 can generate imaging control data including the division method and the exposure time assigned to each divided region.

[0220] In Fig. 19, the region R includes two regions R5 and R6, but may include three or more regions. For example, three or more types of exposure times may be set according to the signal strength of the region of interest. An example in which three or more types of exposure times are set will be described with reference to Fig. 20.

[0221] For example, as shown in FIG. 20A, assume that a region R imaged by the first imaging element 111 includes a plurality of regions of interest R7-1, R7-2, and R7-3 and another region R8. The signal intensities of the regions of interest R7-1, R7-2, and R7-3 are different from each other. The regions of interest R7-1, R7-2, and R7-3 are regions in which cells labeled with different fluorescent materials exist, and are brighter than the other region R8. The image data of FIG. 20A is discriminated by the discrimination unit 115, and as shown in FIG. 20B, the discrimination unit 115 extracts the regions of interest R7-1, R7-2, and R7-3, and divides the region R as shown by the dashed lines. As shown in FIG. 20C, the control unit 130 sets the exposure time for each divided region. Specifically, the control unit 130 sets exposure times t-Δt0, t-Δt1, and t-Δt2 for the divided regions of interest R7-1, R7-2, and R7-3, respectively, and sets an exposure time t for the other divided regions. That is, the exposure time of the divided region including the region of interest is shorter than the exposure time of the other divided regions, and the exposure time is set according to the signal strength of the region of interest. For example, a shorter exposure time may be set for a divided region including a region of interest with a stronger signal strength, and a longer exposure time may be set for a divided region including a region of interest with a weaker signal strength. The control unit 130 may generate imaging control data including the division method and the exposure time allocated to each divided region as described above.

[0222] As described above, the control unit 130 divides the area imaged by the first imaging element 111 into a region of interest and other regions, and the control unit 130 can generate imaging control data that controls the second imaging element 121 so that the region of interest is imaged with a shorter exposure time than the other regions.

[0223] In step S104, the control unit 130 transmits the generated imaging control data to the second imaging unit 120 (for example, the second imaging element 121, etc.).

[0224] In a second imaging step of step S105, the second imaging unit 120 receives the imaging control data generated in step S104, and images the target S based on the imaging control data. Prior to the imaging, the control unit 130 moves the stage 140 on which the target S is placed to a position where the second imaging unit 120 can image the target S.

[0225] For example, in step S105, the second illumination optical system 123 irradiates the target S on the stage 140 with excitation light of a phosphor contained in the target S. The excitation light may be the same as the light irradiated in step S102. In step S105, the control unit 130 or a user may control the second illumination optical system 123 to irradiate the excitation light.

[0226] In step S105, in a state in which the light is irradiated onto the object S, the second imaging element 121 may, for example, image the object S at a magnification different from that of the first imaging element 111 based on the imaging control data. For example, the second imaging element 121 may, based on the imaging control data, divide the area imaged by the first imaging element 111 into multiple areas and image the divided areas. For example, imaging by the second imaging element 121 may be performed multiple times, thereby obtaining multiple image data. For example, in the example shown in FIG. 19, the second imaging element 121 may image the region of interest R5 with an exposure time t-Δt, and may image the other region R6 with an exposure time t.

[0227] In step S105, the second imaging element 121 receives the fluorescence generated from the target S by irradiating it with excitation light via the second observation optical system 122. The second observation optical system 122 may include a wavelength separation element (e.g., a filter or a spectroscope) to separate and receive a desired fluorescence.

[0228] The second imaging element 121 transmits to the control unit 130 the image data (one or a plurality of divided image data, particularly a plurality of divided image data) acquired by imaging in step S105.

[0229] In the image processing step of step S106, the control unit 130 receives the image data (one or a plurality of divided image data, particularly a plurality of divided image data) acquired in step S105, and performs image processing using the image data.

[0230] For example, in step S106, the control unit 130 may generate one image data set using two or more image data sets among the plurality of image data sets. For example, the control unit 130 may generate image data sets of the entire area captured in step S102 using the plurality of image data sets.

[0231] The image data generated by the image processing in step S106 may be stored, for example, in a storage medium provided in the microscope device 100, or may be stored in an external storage medium connected to the microscope device 100 via a wired or wireless connection.

[0232] In step S107, the microscope apparatus 100 ends the imaging process.

[0233] (2-8) Fourth Example of Imaging Control (Imaging Control of a Target Including an Unnecessary Area)

[0234] When observing biological tissue under a microscope, artifacts may be present in the field of view, such as glass chipped areas, glass edges, dirt, air bubbles, embedding medium, and markers (e.g., printed or handwritten marks), which may adversely affect the imaging of the region of interest. In addition, when observing biological tissues under a microscope, biological tissues other than the target of analysis may be present in the field of view. The biological tissues other than the target of analysis may also adversely affect the imaging of the region of interest. In addition, detailed images of the biological tissues other than the target of analysis may not be required. The microscope device of the present technology can image a region of interest without imaging unnecessary regions such as artifacts and biological tissues not to be analyzed, or can image a region of interest such that the amount of data on the unnecessary region is reduced. This can prevent imaging failure. It can also reduce the effect of the unnecessary region on the focus. An example of such an imaging process will be described below.

[0235] (2-8-1) Imaging control of subjects including artifacts

[0236] An example of an imaging process in which artifacts are extracted as unnecessary regions by the microscope device of the present technology, and the imaging sequence of the microscope device is controlled based on the results of the extraction will be described with reference to Figures 1A and 1B and Figure 15.

[0237] Steps S101 and S102 in FIG. 15 are as explained in "(2-5) First example of imaging control (fluorescence separation processing)" above, and the explanation also applies to this example.

[0238] In the discrimination process of step S103, the first imaging element 111 (particularly the discrimination unit 115) discriminates features related to the target S based on the image data acquired in step S102.

[0239] In step S103, for example, the discrimination unit 115 may perform discrimination regarding the region of the target S based on the image data. For example, the discrimination unit 115 may extract an unnecessary region in the target S by the discrimination. The unnecessary region may be, for example, an artifact region or a biological tissue region not to be analyzed. For example, in step S103, the discrimination unit 115 can extract a biological tissue region and an object region other than biological tissue (eg, a label on a slide, a cover glass, a slide glass, etc.) in the image data. In step S103, the discrimination unit 115 may further extract artifacts in the image data. The artifacts may be, for example, glass defects, dust, air bubbles, embedding medium, or marker (printing with a marker). The artifacts may be, for example, present within the biological tissue region or present outside the biological tissue region.

[0240] Furthermore, in step S103, the discrimination unit 115 may discriminate on the attributes of the target S, in addition to discriminating on the region of the target S. For example, discrimination on the attributes of living tissue may be performed.

[0241] The discrimination unit 115 discriminates the features of the target S, preferably using a trained model. The trained model may be generated, for example, by using one or more combinations of image data of the target including biological tissue and features of the target as teacher data. With regard to the trained model, the explanation in "(2-1-1) Discrimination Performed by the First Imaging Element" above also applies to this example.

[0242] In step S103, the first imaging element 111 transmits the result of the discrimination by the discriminator 115 to the controller .

[0243] In the control process of step S104, the control unit 130 receives the determination result transmitted from the first imaging element 111. Based on the determination result, the control unit 130 can generate imaging control data for controlling imaging by the second imaging unit 120, image processing control data for controlling processing of image data obtained by imaging by the second imaging unit 120, or both of these data.

[0244] The control unit 130 may generate, as the imaging control data, imaging control data related to a method of dividing the imaging area by the second imaging element 121. The imaging control data may include data related to a method of setting the imaging area, a method of dividing the area, an order of imaging the divided areas, imaging conditions (e.g., magnification, gain, exposure time, and light amount), and the like.

[0245] The imaging control data generated by the control unit 130 may include, for example, data on the division method. In accordance with the data on the division method, the region of the image data acquired in step S102 is divided into a plurality of divided regions.

[0246] The imaging control data generated by the control unit 130 may include, for example, imaging conditions for a divided area including an unnecessary area and imaging conditions for a divided area not including an unnecessary area. For example, imaging conditions for these divided areas may be set so that the resolution or magnification in imaging the divided area including the unnecessary area is lower than the resolution or magnification in imaging the divided area not including the unnecessary area. Furthermore, the control unit 130 may set an imaging condition such that an image of a divided area including an unnecessary area is not captured.

[0247] Also, for example, there are cases where a region of interest in a pathological specimen overlaps with an artifact. In such a case, the control unit 130 extracts the artifact as an unnecessary region, but the unnecessary region is included in the region of interest. In such a case where the region of interest includes an unnecessary region, the divided region including the unnecessary region may be set as an imaging target (particularly an imaging target with high resolution or high magnification) in the second imaging step. In this case, the control unit 130 may preferably set the imaging sequence so that the divided area including the unnecessary area is imaged after the surrounding divided areas not including the unnecessary area. The divided area including the unnecessary area may fail autofocus due to the presence of artifacts. By setting the imaging sequence as described above, it is possible to use the autofocus results of the surrounding divided areas not including the unnecessary area, thereby avoiding autofocus failure.

[0248] As described above, the control unit 130 can divide the area imaged by the first imaging element 111 into a divided area including an unnecessary area and a divided area not including an unnecessary area, and set imaging conditions for each divided area. The control unit 130 can then generate imaging control data for controlling the second imaging element 121 so that the divided area not including the unnecessary area is imaged with higher image quality than the divided area including the unnecessary area.

[0249] In step S104, the control unit 130 transmits the generated imaging control data to the second imaging unit 120 (for example, the second imaging element 121, etc.).

[0250] In a second imaging step of step S105, the second imaging unit 120 receives the imaging control data generated in step S104, and images the target S based on the imaging control data. Prior to the imaging, the control unit 130 moves the stage 140 on which the target S is placed to a position where the second imaging unit 120 can image the target S.

[0251] For example, in step S105, the second illumination optical system 123 irradiates the target S on the stage 140 with excitation light of a phosphor contained in the target S. The excitation light may be the same as or different from the light irradiated in step S102. In step S105, the control unit 130 or a user may control the second illumination optical system 123 to irradiate the excitation light.

[0252] In step S105, in a state in which the light is irradiated onto the object S, the second imaging element 121 may, for example, image the object S at a magnification different from that of the first imaging element 111 based on the imaging control data. For example, the second imaging element 121 may, based on the imaging control data, divide the area imaged by the first imaging element 111 into a plurality of divided areas and image the divided areas.

[0253] In step S105, the second imaging element 121 receives the fluorescence generated from the target S by irradiating it with excitation light via the second observation optical system 122. The second observation optical system 122 may include a wavelength separation element (e.g., a filter or a spectroscope) to separate and receive a desired fluorescence.

[0254] The second imaging element 121 transmits to the control unit 130 the image data (one or a plurality of divided image data, particularly a plurality of divided image data) acquired by imaging in step S105.

[0255] In the image processing step of step S106, the control unit 130 receives the image data (one or a plurality of divided image data, particularly a plurality of divided image data) acquired in step S105, and performs image processing using the image data.

[0256] For example, in step S106, the control unit 130 may generate one image data set using two or more image data sets among the plurality of image data sets. For example, the control unit 130 may generate image data sets of the entire area captured in step S102 using the plurality of image data sets.

[0257] The image data generated by the image processing in step S106 may be stored, for example, in a storage medium provided in the microscope device 100, or may be stored in an external storage medium connected to the microscope device 100 via a wired or wireless connection.

[0258] In step S107, the microscope apparatus 100 ends the imaging process.

[0259] (2-8-2) Imaging control of subjects including biological tissues not subject to analysis

[0260] An example of imaging processing in which the microscope device of the present technology extracts biological tissue areas that are not to be analyzed as unnecessary areas, and the imaging sequence or image processing of the microscope device is controlled based on the results of the extraction will be described with reference to Figures 1A and B and Figure 15.

[0261] Steps S101 and S102 in Fig. 15 are as explained in "(2-5) First Example of Imaging Control (Fluorescence Separation Processing)" above, and the explanation also applies to this example. In this example, it is assumed that the image data shown in Fig. 21 is acquired. Of the biological tissues in the image data, it is assumed that the shaded region Rh shown in Fig. 22 is biological tissue that is not the subject of analysis.

[0262] In the discrimination process of step S103, the first imaging element 111 (particularly the discrimination unit 115) discriminates features related to the target S based on the image data acquired in step S102.

[0263] In step S103, for example, the discrimination unit 115 may discriminate the region of the target S based on the image data. For example, the discrimination unit 115 may discriminate the hatched region Rh shown in Fig. 22 as a region not to be analyzed by the discrimination, and the other biological tissue regions Rs as regions to be analyzed.

[0264] The discrimination unit 115 discriminates the features of the target S, preferably using a trained model. The trained model may be generated, for example, by using one or more combinations of image data of the target including biological tissue and features of the target as teacher data. With regard to the trained model, the explanation in "(2-1-1) Discrimination Performed by the First Imaging Element" above also applies to this example.

[0265] In step S103, the first imaging element 111 transmits the result of the discrimination by the discriminator 115 to the controller .

[0266] In the control process of step S104, the control unit 130 receives the determination result transmitted from the first imaging element 111. Based on the determination result, the control unit 130 can generate imaging control data for controlling imaging by the second imaging unit 120, image processing control data for controlling processing of image data obtained by imaging by the second imaging unit 120, or both of these data.

[0267] The control unit 130 can generate, for example, imaging control data relating to a manner of performing divided imaging by the second imaging element 121 as the imaging control data. The imaging control data generated by the control unit 130 includes, for example, imaging control data that does not image the hatched region Rg by the second imaging element 121, but images the other biological tissue region Rs by the second imaging element 121.

[0268] In step S104, the control unit 130 transmits the generated imaging control data to the second imaging unit 120 (for example, the second imaging element 121, etc.).

[0269] In a second imaging step of step S105, the second imaging unit 120 receives the imaging control data generated in step S104, and images the target S based on the imaging control data. Prior to the imaging, the control unit 130 moves the stage 140 on which the target S is placed to a position where the second imaging unit 120 can image the target S.

[0270] For example, in step S105, the second illumination optical system 123 irradiates the target S on the stage 140 with excitation light of a phosphor contained in the target S. The excitation light may be the same as or different from the light irradiated in step S102. In step S105, the control unit 130 or a user may control the second illumination optical system 123 to irradiate the excitation light.

[0271] In step S105, in a state in which the light is irradiated onto the target S, the second imaging element 121 can image the target S based on the imaging control data, for example, at a magnification different from that of the first imaging element 111. For example, the second imaging element 121 images the other biological tissue region out of the region imaged by the first imaging element 111 based on the imaging control data.

[0272] The second imaging element 121 transmits to the control unit 130 image data of the other biological tissue regions acquired by imaging in step S105.

[0273] In the image processing step of step S106, the control unit 130 receives the image data acquired in step S105, and performs image processing using the image data. For example, fluorescence separation processing can be performed on the image data.

[0274] Furthermore, in step S106, image synthesis processing may be performed by the control unit 130. For example, the control unit 130 may synthesize an image of the hatched region Rh (region not to be analyzed) in the image data acquired in step S102 and an image of the other biological tissue region Rs (region to be analyzed) acquired in step S105 to generate a synthetic image of two images captured under different imaging conditions, as shown in FIG. In this manner, the control unit of the microscope device according to the present technology can generate a composite image by combining two or more partial images captured under different imaging conditions.

[0275] The image data generated by the image processing in step S106 may be stored, for example, in a storage medium provided in the microscope device 100, or may be stored in an external storage medium connected to the microscope device 100 via a wired or wireless connection.

[0276] In step S107, the microscope apparatus 100 ends the imaging process.

[0277] (3) Second Example of the First Embodiment

[0278] The present technology also provides a microscope device including a first imaging element that images an object including biological tissue and acquires image data, a second imaging element that images the object at a magnification different from that of the first imaging element and acquires image data, and a discrimination unit 115 that discriminates features of the object using a trained model based on the image data, and the second imaging element is controlled based on the result of the discrimination. In the microscope device, the second imaging element is controlled based on the discrimination result by the discrimination unit 115 that uses the trained model. Therefore, an image of a specific region (e.g., a region of interest) in biological tissue can be efficiently or appropriately acquired.

[0279] The microscope device in this second example is the same as the microscope device in the first example described above, except that it is essential that the discrimination by the discrimination unit 115 is performed using a trained model, and that the discrimination unit 115 does not have to be provided in the first imaging element.

[0280] The discrimination using the trained model may be performed as described in the first example above. The discrimination enables efficient or appropriate image acquisition.

[0281] For example, the discrimination unit 115 may be provided outside the first imaging element 111. In one embodiment, the discrimination unit 115 may be provided inside the microscope device 100 and outside the first imaging element 111, and may be included in the control unit 130, for example.

[0282] 2. Second embodiment (image acquisition system)

[0283] The present technology also provides an image acquisition system including the microscope device described in 1 above, and a control unit that controls the imaging of the object by the second imaging element or the processing of the image obtained by the second imaging element based on the result of discrimination by a discrimination unit 115 provided in the microscope device.

[0284] In this embodiment, the control unit may be provided within the microscope device or may be provided outside the microscope device. For example, the control unit may be configured as an information processing device connected to the microscope device by wire or wirelessly, or the control unit may be configured as a server connected to the microscope device via a network.

[0285] 3. Second embodiment (image acquisition method)

[0286] An image acquisition method according to the present technology includes a first imaging step of acquiring image data by imaging a target including biological tissue with a first imaging element, a discrimination step of discriminating features of the target based on the image data by a discrimination unit provided in the first imaging element, and a second imaging step of acquiring image data by imaging the target at a magnification different from that of the first imaging element, and the second imaging element is controlled based on the result of the discrimination.

[0287] The first imaging step is the first imaging step of step S102 in FIG. 15, and the contents relating to step S102 described in 1 above also apply to this embodiment. The determination step is the determination step of step S103 in FIG. 15, and the contents relating to step S103 explained in 1 above also apply to this embodiment. The second imaging step is the second imaging step of step S105 in FIG. 15, and the contents relating to step S105 described in 1 above also apply to this embodiment.

[0288] The image acquiring method may also include other steps (control step S104, image processing step S106) in FIG. 15 described in 1 above.

[0289] Furthermore, the image acquisition method may be performed using, for example, the microscope device 100 described in 1 above, but may also be performed by other microscope devices.

[0290] The present technology can also be configured as follows. [1] a first imaging element for capturing an image of a target including biological tissue and acquiring image data; a second imaging element that captures an image of the object at a magnification different from that of the first imaging element to acquire image data; Equipped with The first imaging element includes a discrimination unit that discriminates features related to the object based on the image data, The second imaging element is controlled based on a result of the determination. Microscope equipment. [2] The microscope device described in [1], characterized in that the second imaging element further divides the area where the image is captured into a plurality of areas, captures an image of at least one of the plurality of areas, and obtains image data. [3] The microscope device according to claim 1 or 2, wherein the first imaging element is an imaging element in which the discrimination unit and an imaging unit that performs imaging are arranged within a single chip. [4] The microscope device according to any one of [1] to [3], wherein the second imaging element images each of the plurality of regions at a higher magnification than the first imaging element. [5] The microscope apparatus according to any one of [1] to [4], wherein the image data is bright-field image data and / or fluorescent image data. [6] The microscope device according to any one of [1] to [5], wherein the image data is an image signal including a luminance value and / or a frequency. [7] The microscope device according to any one of [1] to [6], wherein the features related to the object include features related to attributes of the object. [8] The microscope device according to claim 7, wherein the features relating to the attributes of the object include one or more selected from features relating to the attributes of the biological tissue, features relating to autofluorescence components, and features relating to lesions in the biological tissue. [9] The microscope apparatus according to any one of [1] to [6], wherein the features relating to the object include features relating to a region of the object.

[10] The microscope apparatus of claim 9, wherein the features relating to the region of the object include one or more selected from features relating to a diseased region in the biological tissue and features relating to a foreign object in the object.

[11] The microscope device according to any one of [1] to

[10] , wherein the discrimination unit performs the discrimination using a trained model.

[12] The microscope device described in any one of [1] to

[11] , further comprising a control unit that controls the imaging of the object by the second imaging element or the processing of the image obtained by the second imaging element based on the discrimination result by the discrimination unit.

[13] The microscope device according to claim 12, wherein the control unit further includes an imaging sequence control unit that controls an imaging order based on characteristics related to the region of the target.

[14] The imaging sequence control unit, Controlling the imaging so that the seams of the plurality of regions do not overlap the region of interest in the object; or The microscope apparatus according to claim 13, wherein the imaging sequence control unit controls imaging so that a region of interest in the object is imaged multiple times.

[15] The microscope apparatus according to any one of

[12] to

[14] , wherein the control unit further includes an exposure control unit that controls gain and / or exposure time.

[16] The microscope apparatus according to claim 15, wherein the exposure control unit controls gain and / or exposure time on a pixel-by-pixel basis and / or on a wavelength-by-wavelength basis.

[17] The microscope device according to any one of

[12] to

[16] , wherein the control unit further includes a light amount control unit that controls the light amount of the light source.

[18] The microscope apparatus according to any one of

[12] to

[17] , wherein the image processing includes a fluorescence separation process.

[19] a microscope apparatus comprising: a first imaging element that images an object including biological tissue to obtain image data; and a second imaging element that divides the imaged area into a plurality of areas and images the divided areas, the first imaging element further comprising a discrimination unit that discriminates features of the object based on the image data, and the second imaging element is controlled based on a result of the discrimination; a control unit that controls imaging of the object by the second imaging element or controls processing of an image obtained by the second imaging element based on a result of the determination; 1. An image acquisition system comprising:

[20] a first imaging step of imaging an object including biological tissue with a first imaging element to acquire image data; a discrimination step of discriminating features of the object based on the image data by a discrimination unit provided in the first imaging element; a second imaging step of dividing the imaged area into a plurality of areas and imaging the divided areas by a second imaging element; Including, The second imaging element is controlled based on a result of the determination. Image acquisition method. [Explanation of symbols]

[0291] 100 Microscope Equipment 110 First imaging unit 111 First imaging element 112 First observation optical system 113 First illumination optical system 120 Second imaging unit 121 Second imaging element 122 Second Observation Optical System 123 Second illumination optical system 130 Control section 140 Stages

Claims

1. a first imaging element for capturing an image of a target including biological tissue and acquiring image data; A second imaging element that captures an image of the object at a magnification different from that of the first imaging element to obtain image data; Equipped with The first imaging element includes a discrimination unit that discriminates features related to the object based on the image data, the second imaging element is controlled based on data relating to the characteristic; Microscope equipment.

2. 2. The microscope apparatus according to claim 1, wherein the second imaging element further divides the area where the image has been captured into a plurality of areas, and captures an image of at least one of the plurality of areas to obtain image data.

3. The microscope apparatus according to claim 1 , wherein the first imaging element is an imaging element in which the discrimination unit and an imaging unit that performs imaging are disposed on a single chip.

4. The microscope apparatus according to claim 2 , wherein the second image sensor captures images of the plurality of regions at a magnification higher than that of the first image sensor.

5. The microscope apparatus according to claim 1 , wherein the image data is bright-field image data and / or fluorescent image data.

6. The microscope apparatus according to claim 1 , wherein the image data is an image signal including a luminance value and / or a frequency.

7. The microscope apparatus according to claim 1 , wherein the features related to the object include features related to attributes of the object.

8. The microscope apparatus according to claim 7 , wherein the features relating to attributes of the object include one or more selected from features relating to attributes of the biological tissue, features relating to autofluorescence components, and features relating to lesions in the biological tissue.

9. The microscope apparatus of claim 1 , wherein the features related to the object include features related to a region of the object.

10. The microscope apparatus according to claim 9 , wherein the features relating to the region of the object include one or more selected from the group consisting of features relating to a diseased region in the biological tissue and features relating to a foreign object in the object.

11. The microscope apparatus according to claim 1 , wherein the discrimination unit performs the discrimination using a trained model.

12. The microscope device according to claim 1 , further comprising a control unit that controls the imaging of the object by the second imaging element or the processing of the image obtained by the second imaging element based on the discrimination result by the discrimination unit.

13. The microscope apparatus according to claim 12 , wherein the control unit further includes an imaging sequence control unit that controls an imaging order based on features related to the region of the target.

14. The second imaging element further divides the area where the imaging has been performed into a plurality of areas, images at least one of the plurality of areas, and acquires image data; The imaging sequence control unit, Controlling the imaging so that the seams of the plurality of regions do not overlap the region of interest in the object; or controlling imaging so that a region of interest in the object is imaged multiple times; The microscope apparatus according to claim 13.

15. The microscope apparatus according to claim 12 , wherein the control unit further includes an exposure control unit that controls a gain and / or an exposure time.

16. The microscope apparatus according to claim 15 , wherein the exposure control unit controls gain and / or exposure time on a pixel-by-pixel basis and / or on a wavelength-by-wavelength basis.

17. The microscope apparatus according to claim 12 , wherein the control unit further includes a light amount control unit that controls the amount of light from the light source.

18. The microscope apparatus according to claim 12 , wherein the processing of the image includes fluorescence separation processing.

19. a microscope device comprising: a first imaging element that images an object including biological tissue to obtain image data; and a second imaging element that images the object at a magnification different from that of the first imaging element to obtain image data, the first imaging element further comprising a discrimination unit that discriminates features of the object based on the image data, and the second imaging element is controlled based on data relating to the features; a control unit that controls imaging of the object by the second imaging element or controls processing of an image obtained by the second imaging element based on a result of the determination; 1. An image acquisition system comprising:

20. a first imaging step of imaging an object including biological tissue with a first imaging element to acquire image data; a discrimination step of discriminating features of the object based on the image data by a discrimination unit provided in the first imaging element; a second imaging step of imaging the object with a second imaging element at a magnification different from that of the first imaging element to obtain image data; Including, the second imaging element is controlled based on data relating to the characteristic; Image acquisition method.

Citation Information

Patent Citations

  • Information processing device, information processing method, program and microscope system

    WO2013179723A1

  • Information processing device, information processing method, solid-state image capture element, solid-state image capture element operating method, program, and electronic device

    WO2017022462A1

  • Learning device, method, and program for determiner, and determiner

    WO2019039035A1

  • Imaging control device, method, and program

    WO2019044424A1