Microscope system, microscope autofocusing method, and medical device

Abstract

The application is a divisional application of 202010284514.X. A microscope system, a microscope automatic focusing method and a medical device are provided, the microscope system comprising: a camera assembly and a focusing device arranged in a first light path; an auxiliary focusing device arranged in a second light path, the auxiliary focusing device comprising: an auxiliary focusing light source and an auxiliary focusing camera. The application can realize automatic focusing of the microscope system through the auxiliary focusing device, form and output an image clearly focused through the first light path, save the focusing time of the microscope system, and improve the focusing accuracy.

Description

[0001] This application is a divisional application of application number 202010284514.X, filed on April 13, 2020, entitled "Automatic Focusing Method for Microscope, Microscope System, Medical Device, and Storage Medium". Technical Field

[0002] This invention relates to medical image processing technology, and more particularly to microscope autofocus methods, microscope systems, medical devices, and storage media. Background Technology

[0003] With the research and advancement of artificial intelligence (AI) technology, AI has been studied and applied in multiple fields. For example, in recent years, augmented reality (AR) and AI have been proposed for use in traditional optical microscope systems. This involves using a camera to acquire images of the sample under observation on a traditional optical microscope, and then combining these images with machine learning algorithms to analyze them in real time.

[0004] The ability of the camera to capture high-quality images is crucial for ensuring the accuracy of the augmented reality microscope's algorithms. Images of out-of-focus samples lose much important optical information; therefore, ensuring the camera can capture accurately focused images of the sample is paramount. Inaccurately focused microscope images negatively impact the model's output. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a microscope autofocus method, a microscope system, a medical device, and a storage medium. The technical solution of the embodiments of the present invention is implemented as follows:

[0006] This invention provides a microscope system, the microscope system comprising:

[0007] The objective lens is used to capture the light from the sample to be observed entering the first optical path, and the light generated by the image projection module enters the first optical path through the lens assembly and then merges at the beam splitter.

[0008] The beam splitter assembly includes at least one beam splitter for separating and projecting light rays in different optical paths respectively;

[0009] The lens assembly includes at least one lens for projecting light generated by the observed sample during observation into different optical paths after the light enters through the objective lens, so as to achieve the propagation of the light along different optical paths;

[0010] An image projection component is disposed in the corresponding optical path of the light rays projected by the lens component, and is used to perform image enhancement processing on the image of the sample to be observed;

[0011] A camera assembly, wherein the camera assembly is disposed in a first optical path, the camera assembly includes a camera for taking pictures of the sample to be observed in the field of view of the microscope, so as to form and output a clear and focused image captured through the first optical path;

[0012] An auxiliary focusing device, including an auxiliary focusing light source and an auxiliary focusing camera, is disposed in a second optical path and is used to determine a focal length that matches the camera assembly;

[0013] A focusing device configured to adjust the focal length of the image light entering the camera assembly based on the defocus amount of the sample image to be tested determined by the auxiliary focusing device.

[0014] In the above scheme, the microscope system further includes:

[0015] An eyepiece and a trinocular tube, wherein the eyepiece and the trinocular tube are fitted together for observing the sample to be observed through the objective lens;

[0016] The trinocular tube is located at the end of the beam splitter away from the objective lens. The trinocular tube includes channels and a tube lens. The channels include at least two channels and are located at the end away from the beam splitter. One of the channels communicates with the eyepiece, and the tube lens is located at the end close to the beam splitter.

[0017] In the above scheme, the focusing device includes a shift drive component and a variable focal length lens to enable the photographing of the sample to be observed in the microscope field of view at different focal lengths.

[0018] In the above scheme,

[0019] The beam splitter assembly is connected to the objective lens and the tube lens of the trinocular lens tube, respectively, and the camera assembly is disposed in one of the channels of the trinocular lens tube;

[0020] The beam splitter assembly includes a beam splitter, and the lens assembly includes a lens disposed between the beam splitter and the image projection assembly;

[0021] The focusing device is located between the beam splitter and the camera assembly, and is used to adjust the focal length of the image light entering the camera assembly based on the defocus amount of the sample image to be tested determined by the auxiliary focusing device.

[0022] In the above scheme,

[0023] The image projection component further includes a first polarizer, which is located between the lens component and the beam splitter, and is used to polarize the corresponding light rays in the first optical path.

[0024] The camera assembly further includes a second polarizer located between the focusing device and the beam splitter, for polarizing the corresponding light collected by the camera assembly.

[0025] In the above scheme,

[0026] The auxiliary focusing light source is located in the Fourier back focal plane corresponding to the condenser lens assembly of the microscope system, and is used to emit auxiliary focusing light to form the second optical path;

[0027] The beam splitter assembly includes a beam splitter disposed between the focusing device and the camera assembly for reflecting light in the second optical path to the auxiliary focusing camera;

[0028] The auxiliary focusing camera is positioned at an axially offset position on the conjugate plane of the camera assembly, and is used to capture an overlapping image that matches the sample to be observed in the microscope field of view based on the light in the second optical path.

[0029] In the above scheme,

[0030] The auxiliary focusing light source is located in the Fourier back focal plane corresponding to the condenser lens assembly of the microscope system, and is used to emit auxiliary focusing light to form the second optical path;

[0031] The auxiliary focusing camera and the image projection component are arranged opposite each other along the beam splitter component, and are used to capture an overlapping image that matches the sample to be observed in the microscope field of view based on the light in the second optical path.

[0032] In the above scheme, the image projection component and the camera component operate using a time-division multiplexing mechanism.

[0033] In the above scheme, the microscope system further includes:

[0034] At least one output interface device, coupled to the data processing unit of the microscope system, is provided to output a clearly focused image captured through a first optical path and an image of the sample to be observed after image enhancement processing.

[0035] In the above scheme, the objective lens includes at least one of the following:

[0036] Achromatic objectives, plan achromatic objectives, plan semi-apochromatic objectives, or plan apochromatic objectives;

[0037] The beam splitter includes at least one of the following:

[0038] Cuboid beam splitter, flat beam splitter, or thin-film beam splitter.

[0039] This invention also provides a method for automatic focusing of a microscope, the method comprising:

[0040] To obtain the measurement sample captured by the auxiliary focusing camera in the second optical path of the microscope;

[0041] Based on the measurement samples captured by the camera with assisted focusing and the corresponding image evaluation criteria, calculate the corresponding image evaluation parameters;

[0042] Based on the image evaluation parameters, the relationship between the image evaluation parameters and the defocus amount is found in the pre-stored calibration curves, thereby determining the required defocus amount.

[0043] Based on the determined defocus amount, the focal length of the image light entering the camera assembly is adjusted so that the camera assembly can capture a clearly focused image through the first optical path.

[0044] In the above scheme, acquiring the measurement sample captured by the auxiliary focusing camera in the second optical path of the microscope includes:

[0045] The auxiliary focusing camera captures light from the second optical path;

[0046] The light in the second optical path is processed according to the type of the auxiliary focusing camera to achieve the capture of an overlapping image that matches the sample to be observed in the microscope's field of view.

[0047] The method in the above scheme further includes:

[0048] Based on the focal length adjustment, the sample to be observed in the microscope's field of view is photographed by the light in the first optical path, forming and outputting a clear and focused image captured by the first optical path.

[0049] This invention also provides a medical device, the medical device comprising:

[0050] The system includes a microscope system, a memory, and a processor. The microscope system is the microscope system provided in the foregoing embodiments. The processor performs the following steps:

[0051] Memory, used to store executable instructions;

[0052] A processor, used to execute executable instructions stored in the memory, implements an autofocusing method for a microscope prior to this.

[0053] This invention also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the aforementioned microscope autofocus method.

[0054] The embodiments of the present invention have the following beneficial effects:

[0055] In this embodiment of the invention, an objective lens is used to capture light rays from the sample being observed entering a first optical path. These light rays, after passing through a lens assembly and entering the first optical path, converge at a beam splitter. The beam splitter assembly includes at least one beam splitter for separating and projecting light rays from different optical paths. The lens assembly includes at least one lens for projecting light rays generated by the sample during observation through the objective lens into different optical paths, thus enabling the light rays to propagate along different optical paths. An image projection assembly is disposed in the corresponding optical path of the light rays projected by the lens assembly, for projecting the light rays... The system includes: an image enhancement process for the sample to be observed; a camera assembly, disposed in the first optical path, comprising a camera for photographing the sample in the microscope's field of view to form and output a clearly focused image captured through the first optical path; an auxiliary focusing device, comprising an auxiliary focusing light source and an auxiliary focusing camera, disposed in the second optical path for determining a focal length matching the camera assembly; and a focusing device configured to adjust the focal length of the image light entering the camera assembly based on the defocusing amount of the sample image determined by the auxiliary focusing device. This allows the focusing device to automatically focus on the camera assembly of the microscope system, forming and outputting a clearly focused image captured through the first optical path, saving focusing time and improving focusing accuracy. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the application environment of the microscope autofocus method provided in the embodiments of the present invention;

[0057] Figure 2 This is a schematic diagram of the composition structure of a medical device provided in an embodiment of the present invention;

[0058] Figure 3 This is an optional structure of the microscope system in an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of an optional process of the microscope autofocusing method provided in an embodiment of the present invention;

[0060] Figure 5 This is an optional structural schematic diagram of the microscope system provided in an embodiment of the present invention;

[0061] Figure 6 This is an optional structural schematic diagram of the microscope system provided in an embodiment of the present invention;

[0062] Figure 7 This is an optional structural schematic diagram of the microscope system provided in an embodiment of the present invention;

[0063] Figure 8 This is a schematic diagram illustrating the relationship between defocusing amount and ghost distance in an embodiment of the present invention;

[0064] Figure 9 This is a schematic diagram showing the fitting of the relationship between defocus amount and ghost distance in this invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0067] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0068] 1) In response to, used to indicate the conditions or states on which the operation performed depends. When the conditions or states on which it depends are met, one or more operations performed may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0069] 2) Terminals, including but not limited to: ordinary terminals and dedicated terminals, wherein the ordinary terminals maintain a long connection and / or a short connection with the transmission channel, and the dedicated terminals maintain a long connection with the transmission channel.

[0070] 3) Client: The carrier that implements specific functions in the terminal. For example, a mobile client (APP) is a carrier of specific functions in a mobile terminal, such as performing payment and consumption functions or purchasing financial products.

[0071] 4) Lens assembly, a device comprising at least one lens assembly, which may be provided with a lens tube for observing magnified optical images of objects such as cells.

[0072] 5) Field of view: The range that can be observed when viewing a magnified image of cells in a smear through a lens assembly.

[0073] 6) Computer-aided Diagnosis (CAD): CAD is used to assist in the detection of lesions and improve the accuracy of diagnosis by combining imaging, medical image processing technology, and other possible physiological and biochemical methods with computer analysis and calculation.

[0074] The following example illustrates the microscope autofocus method provided by this invention using microscopic observation of corresponding lesion cell sections. (Refer to...) Figure 1 , Figure 1 This is a schematic diagram illustrating a usage scenario of the microscope autofocus method provided in this embodiment of the invention. (See attached diagram.) Figure 1 The terminals (including terminals 10-1 and 10-2) are equipped with corresponding clients capable of performing different functions. These clients allow the terminals (including terminals 10-1 and 10-2) to retrieve and browse different slice images from the corresponding server 200 via a network 300. The terminals connect to the server 200 via the network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both, using a wireless link for data transmission. The types of slice images retrieved by the terminals (including terminals 10-1 and 10-2) from the corresponding server 200 via the network 300 can be the same or different. For example, the terminals (including terminals 10-1 and 10-2) can retrieve pathological images or videos matching the target object from the corresponding server 200 via the network 300, or they can retrieve and browse only pathological slices matching the current target. The server 200 can store slice images corresponding to different target objects, as well as auxiliary analysis information matching the slice images of the target objects.

[0075] The server-deployed neural network model in the field of artificial intelligence can utilize images of the sample being observed acquired using a camera on a traditional optical microscope, and then analyze these images in real time using machine learning algorithms. Artificial Intelligence (AI) ， Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0076] Specifically, artificial intelligence (AI) is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0077] It should be noted that the patient lesions viewed under the microscope system 400 (a medical device in contact with the pathological cell section of the target object) can include a variety of different application scenarios, such as lung cancer cell screening, early cervical cancer screening, and screening of different cell sections. The microscope system image processing method based on this embodiment can be deployed to various application scenarios, thereby facilitating remote review and use by doctors.

[0078] Server 200 sends pathological information of the same target object to terminals (terminal 10-1 and / or terminal 10-2) via network 300 to enable users of terminals (terminal 10-1 and / or terminal 10-2) to analyze the pathological information of the target object. As an example, server 200 deploys a corresponding neural network model to analyze the clear image information output by the microscope system. The microscope system acquires images by: acquiring a measurement sample captured by the auxiliary focusing camera in the second optical path of the microscope; calculating corresponding image evaluation parameters based on the measurement sample captured by the auxiliary focusing camera and the corresponding image evaluation criteria; finding the relationship between the image evaluation parameters and the defocus amount in a pre-stored calibration curve based on the image evaluation parameters, thereby determining the required defocus amount; and adjusting the focal length of the image light entering the camera assembly based on the determined defocus amount, so that the camera assembly captures a clearly focused image through the first optical path.

[0079] Based on the result of the focal length adjustment, the sample to be observed in the microscope field of view is photographed, forming and outputting a clear and focused image captured through the first optical path.

[0080] The structure of the medical device according to an embodiment of the present invention will be described in detail below. The medical device can be implemented in various forms, such as a dedicated terminal with image processing function of a microscope system, or a medical device with image processing function of a microscope system, or a cloud server, as described above. Figure 1 Server 200 in the middle. Figure 2 This is a schematic diagram of the composition and structure of a medical device provided in an embodiment of the present invention. It can be understood that... Figure 2 Only an exemplary structure of the medical device is shown, not the entire structure; implementation is possible as needed. Figure 2The structure shown may be part or all of the structure.

[0081] The medical device provided in this embodiment of the invention includes at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. The various components in the medical device 20 are coupled together via a bus system 205. It is understood that the bus system 205 is used to implement communication between these components. In addition to a data bus, the bus system 205 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 205.

[0082] The user interface 203 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0083] It is understood that memory 202 can be volatile memory or non-volatile memory, or both. In this embodiment of the invention, memory 202 is capable of storing data to support the operation of the terminal (e.g., 10-1). Examples of this data include any computer programs used to operate on the terminal (e.g., 10-1), such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0084] In some embodiments, the microscope system provided by the present invention can be implemented using a combination of hardware and software. For example, the microscope system provided by the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method of the microscope system provided by the present invention. For instance, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0085] As an example of the combined hardware and software implementation of the microscope system provided in this embodiment of the invention, the microscope system provided in this embodiment of the invention can be directly embodied as a combination of software modules executed by processor 201. The software modules can be located in a storage medium, which is located in memory 202. Processor 201 reads the executable instructions included in the software modules in memory 202 and combines them with necessary hardware (e.g., including processor 201 and other components connected to bus 205) to complete the image processing method of the microscope system provided in this embodiment of the invention.

[0086] As an example, processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0087] As an example of the hardware implementation of the microscope system provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor 201 in the form of a hardware decoding processor. For example, it can be executed by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the image processing method of the microscope system provided in this embodiment of the invention.

[0088] In this embodiment of the invention, the memory 202 is used to store various types of data to support the operation of the medical device 20. Examples of such data include: any executable instructions for operation on the medical device 20, such as executable instructions that implement the image processing method from the microscope system according to this embodiment of the invention, which may be included in the executable instructions.

[0089] In other embodiments, the microscope system provided by the present invention can be implemented in software. Figure 2 A microscope system 2020 stored in memory 202 is shown. This system may be software in the form of programs and plugins, and includes a series of modules. As an example of a program stored in memory 202, it may include the microscope system 2020. The microscope system 2020 includes the following software modules:

[0090] The information processing module 2081 is used to acquire the measurement sample captured by the auxiliary focusing camera in the second optical path of the microscope; calculate the corresponding image evaluation parameters based on the measurement sample captured by the auxiliary focusing camera and the corresponding image evaluation criteria; find the relationship between the image evaluation parameters and the defocus amount in the pre-stored calibration curve based on the image evaluation parameters, and then determine the required defocus amount; adjust the focal length of the image light entering the camera assembly based on the determined defocus amount, so that the camera assembly can capture a clear and focused image through the first optical path.

[0091] Before introducing the microscope autofocus method provided by this invention, the microscope focusing process in related technologies will first be described, wherein, referring to Figure 3 , Figure 3 As an optional structure of the microscope system in the related technology of the present invention, a microscope 300 is provided in a related embodiment. The microscope 300 has a microscope body 301, a microscope body stage focusing knob 302, a microscope body stage 303, a sample to be observed 304, a microscope body objective lens 305, a trinocular tube 306, a camera 307, and an eyepiece 308. The microscope body stage 303 is disposed above the microscope body 301, and the sample to be observed 304 is placed on the microscope body stage 303. The microscope body stage focusing knob 302 is located on both sides of the microscope body 301. The microscope body objective lens 305 is located above the microscope body stage 303. The trinocular tube 306 is also disposed above the microscope body objective lens 305, and the head of the trinocular tube 306 is connected to the camera 307 and the eyepiece 308 respectively. Adjusting the focusing knob 302 on the microscope stage allows the microscope stage 303 to rise or fall vertically, thereby changing the distance between the microscope stage 303 and the microscope objective lens 305 to achieve focusing. Alternatively, the microscope objective lens 305 can be moved, thus changing the distance between the microscope stage 303 and the microscope objective lens 305 to achieve focusing.

[0092] The aforementioned focusing of the microscope 300 is based on the assumption that the eyepiece 308 and the camera 307 of the trinocular tube 306 of the microscope 300 are in focus. However, due to limitations in focusing technology, the image from the camera 307 may be out of focus with the image from the eyepiece 308. For example, different magnification objectives may not be properly aligned, different users of the microscope 300 may have different refractive powers, and when switching users, the new user may not be aware of adjusting the diopter knob on the eyepiece 308 and instead directly adjust the stage to refocus the sample. These factors can all lead to the image from the eyepiece 308 and the camera 307 being out of focus. As a result, while the human eye sees a clear image, the camera 307 captures an out-of-focus image, thus compromising the accuracy of the image analysis results. In the field of automated microscope image analysis, the camera's ability to capture high-quality images is crucial for the accuracy of augmented reality microscope algorithms. Out-of-focus images of the sample lose much important optical information, which even powerful post-processing algorithms cannot compensate for. Therefore, it is especially important to ensure that the camera can capture an image of the sample that is accurately focused.

[0093] To overcome the above-mentioned shortcomings, refer to Figure 4 , Figure 4 This is an optional process diagram of the microscope autofocusing method provided in an embodiment of the present invention. The microscope autofocusing method provided by the present invention includes the following steps:

[0094] Step 401: Obtain the measurement sample captured by the camera that assists in focusing in the second optical path of the microscope;

[0095] Step 402: Calculate the corresponding image evaluation parameters based on the measurement samples captured by the camera with assisted focusing and the corresponding image evaluation criteria.

[0096] Among them, the image evaluation criterion can be the relative pixel offset of the ghosting in the image acquired by the auxiliary focusing camera.

[0097] Step 403: Based on the image evaluation parameters, find the relationship between the image evaluation parameters and the defocus amount in the pre-stored calibration curves, and then determine the required defocus amount;

[0098] The calibration curve can be pre-saved in the corresponding storage medium, which can be used to realize the autofocus function of the augmented reality microscope. The pre-saved calibration curve is a curve determined based on the images corresponding to different defocus amounts and the corresponding image evaluation criteria (such as the relative pixel offset of the ghosting in the image acquired by the auxiliary focusing camera). Therefore, by looking up the calibration curve, the relationship between the defocus amount and the image evaluation parameters of different defocus degrees can be determined.

[0099] Step 404: Adjust the focal length of the image light entering the camera assembly according to the determined defocus amount, so that the camera assembly can capture a clear and focused image through the first optical path.

[0100] In this embodiment, the autofocus method adjusts the focal length of the image light entering the camera assembly according to the determined defocus amount. Regardless of whether the image to be observed presented in the eyepiece is clear, the camera assembly can capture a clear and focused image through the first optical path, avoiding the shortcomings of slow speed and poor accuracy of manual focusing in the prior art.

[0101] The following description, using different types of microscope systems, illustrates the automatic focusing method for microscopes provided by this invention. For different types of microscope systems, their structures are similar to... Figure 3 The microscopes shown also differ. For example, Augmented Reality Microscopes (ARMs) can conveniently and accurately observe samples under a microscope while simultaneously acquiring other augmented information, thus helping observers quickly locate and quantify features of interest. Taking a medical diagnostic scenario as an example, when doctors use an ARM microscope to observe slides, they can simultaneously obtain diagnostic results based on those slides. That is, the ARM microscope can overlay the diagnostic results as augmented reality information onto the slide, making it convenient for doctors to read conclusions in real time within the field of view. At the same time, the neural network model running on the server can make auxiliary diagnostic judgments on the diseased area, helping doctors make correct judgments on the pathological information of the lesions.

[0102] The structure of the microscope system provided in this application will be further described below in conjunction with different microscope system configurations, with reference to... Figure 5 , Figure 5This is an optional structural schematic diagram of a microscope system provided in an embodiment of the present invention. Specifically, the microscope system 1100 includes an objective lens 115, a beam splitter 1112, an image projection assembly 1111, a camera assembly 117, and a trinocular tube 116. The objective lens 115 has a first end 10a and a second end 10b disposed opposite to each other, with the first end 10a facing the sample to be observed. The beam splitter 1112 is disposed at the second end 10b and communicates with both the objective lens 115 and the tube end 1118 of the trinocular tube. The camera assembly 117 is disposed in one of the channels of the trinocular tube. The image projection assembly 1111 projects a corresponding field of view using the light transmitted through the lens 1115. The image is projected into the image. The camera assembly 117 receives light transmitted from the tube lens 1118. The camera assembly 117 includes a camera and a corresponding image output device for transmitting the captured image in the corresponding field of view to a server for image processing or recognition. The trinocular tube 116 is located at the end of the beam splitter 1112 away from the objective lens 115. The trinocular tube 116 includes channels and a tube lens 1118. The channels include at least two and are located at the end away from the beam splitter 1112. The tube lens 1118 is located at the end close to the beam splitter 1112. The camera assembly 117 receives the light output from the beam splitter 1112 through the tube lens 1118 to complete the acquisition of the image in the corresponding field of view.

[0103] In this process, since the camera assembly 117 and the image projection assembly 1111 are located in different positions, in order to avoid the influence of light propagation, the image projection assembly 1111 also includes a first polarizer 1116. The first polarizer 1116 is located between the tube lens 1118 and the beam splitter 1112 and is used to polarize the corresponding light in the first optical path. The camera assembly also includes a second polarizer 1117, which is located between the tube lens 1118 and the camera assembly 117 and is used to polarize the corresponding light collected by the camera assembly 117.

[0104] Furthermore, the optical path of the microscope system 1100 is as follows: the light from the objective lens 115 is transmitted to the beam splitter 1112. The beam splitter 1112 reflects a portion of the light to the tube lens 1118 and transmits it to the photosensitive chip of the camera assembly 117 through the first polarizer 1116. At the same time, the beam splitter 1112 transmits a portion of the light to the tube lens 1118, passes through the tube lens 1118, and reflects the transmitted light through the tube lens 1118 to reach the trinocular tube 116. The trinocular tube 116 transmits the light to the eyepiece 118, through which the image of the sample 114 to be observed can be viewed. Meanwhile, the light generated by the image projection assembly 1111 travels along the lens 1115, is polarized by the second polarizer 1117, and is prevented from reaching the camera assembly 117 by the beam splitter 1112 and the first polarizer 1116, thus not affecting the imaging of the camera assembly 117.

[0105] However, during this process, when microscope users with different refractive powers switch between microscope systems, they need to repeat the tedious trinocular parfocal adjustment every time. The camera also cannot autofocus, making it impossible to capture clear images. Furthermore, while the microscope user sees a clear image through the eyepiece, the camera captures an out-of-focus image, thus compromising the accuracy of the image analysis results executed on the server.

[0106] To solve the above problems, further reference is made. Figure 6 , Figure 6This is an optional structural schematic diagram of a microscope system provided in an embodiment of the present invention. Specifically, the microscope system 1100 includes an objective lens 115, a beam splitter 1112, an image projection assembly 1111, a camera assembly 117, and a trinocular tube 116. The objective lens 115 has a first end 10a and a second end 10b disposed opposite to each other, with the first end 10a facing the sample to be observed. The beam splitter 1112 is disposed at the second end 10b and communicates with both the objective lens 115 and the tube end 1118 of the trinocular tube. The camera assembly 117 is disposed in one of the channels of the trinocular tube. The image projection assembly 1111 projects a corresponding field of view using the light transmitted through the lens 1115. The image is projected into the image. The camera assembly 117 receives light transmitted from the tube lens 1118. The camera assembly 117 includes a camera and a corresponding image output device for transmitting the captured image in the corresponding field of view to a server for image processing or recognition. The trinocular tube 116 is located at the end of the beam splitter 1112 away from the objective lens 115. The trinocular tube 116 includes channels and a tube lens 1118. The channels include at least two and are located at the end away from the beam splitter 1112. The tube lens 1118 is located at the end close to the beam splitter 1112. The camera assembly 117 receives the light output from the beam splitter 1112 through the tube lens 1118 to complete the acquisition of the image in the corresponding field of view. Since the camera assembly 117 and the image projection assembly 1111 are located in different positions, to avoid interference during light propagation, the image projection assembly 1111 further includes a second polarizer 1117. The second polarizer 1117 is located between the lens 1115 and the beam splitter 1112, and is used to polarize the corresponding light in the first optical path. The camera assembly also includes a first polarizer 1116, which is located between the tube lens 1118 and the camera assembly 117, and is used to polarize the corresponding light collected by the camera assembly 117. Specifically, since the microscope operator's naked eye wants to see through the eyepiece 11... The camera assembly 117 can observe both the image of the sample 114 to be observed and the image output from the image projection assembly 1111. At the same time, the camera assembly 117 only wants to capture the image of the sample 114 to be observed, while ignoring the image output from the image projection assembly 1111. The second polarizer 1117 can convert the light output from the image projection assembly 1111 into polarized light, which can be directly observed by the human eye through the eyepiece 118. However, since the polarization directions of the first polarizer 1116 and the second polarizer 1117 are perpendicular to each other, the first polarizer 1116 can eliminate the polarized light output from the second polarizer 1117, so that the camera assembly 117 can only capture the image of the sample 114 to be observed.

[0107] In some embodiments of the present invention, the objective lens includes at least one of the following:

[0108] The beam splitter includes at least one of the following: a cuboid beam splitter, a plan achromatic beam splitter, a plan semi-apochromatic beam splitter, or a plan apochromatic beam splitter. Specifically, considering the potential need for different magnifications when observing objects—for example, different magnifications may be required for the same object, such as the outline and nucleus of a cell, or for objects of different sizes—objective combinations with different magnifications can be provided for user selection. For example, objective combinations with magnifications of 4.0X, 10.0X, 20.0X, 60.0X, and 100.0X can be provided for user selection. Furthermore, the cuboid beam splitter, plan beam splitter, or thin-film beam splitter can be selected and adapted according to the type of augmented reality microscope to suit different usage environments.

[0109] Meanwhile, in order to achieve autofocus, the microscope system 1100 also includes: an auxiliary focusing light source 1140 disposed in the Fourier back focal plane corresponding to the condenser lens assembly 1141, for emitting auxiliary focusing light, wherein the auxiliary focusing light source 1140 can be two identical infrared LED light emitters, and the auxiliary focusing camera 1143 is disposed at an axially offset position on the conjugate plane of the camera assembly 117, wherein the second optical path formed by the light generated by the auxiliary focusing light source 1140 is shown in Figure 1144.

[0110] Furthermore, the microscope system 1100 optical path includes a first optical path and a second optical path. The first optical path is used to project the light generated by the observed sample after it enters the optical path through the objective lens, so that the camera assembly can take pictures of the observed sample in the microscope's field of view, thereby forming and outputting a clear and focused image captured by the first optical path. At the same time, it can also enable the image projection assembly to perform image enhancement processing on the image of the observed sample using the light in the second optical path.

[0111] The first optical path includes: light from the objective lens 115 is transmitted to the beam splitter 1112; the beam splitter 1112 reflects a portion of the light to the tube mirror 1118, passes through the first lens 1116, and is transmitted to the photosensitive chip of the camera assembly 117; simultaneously, the beam splitter 1112 transmits a portion of the light to the tube mirror 1118, passes through the tube mirror 1118, and reflects the transmitted light through the tube mirror 1118 to reach the trinocular tube 116; the trinocular tube 116 transmits the light to the eyepiece 118, through which observation can be performed. The image of the sample 114 to be observed is simultaneously captured by the image projection component 1111. The light generated by the image projection component 1111 travels along the lens 1115, is polarized by the second polarizer 1117, and then, through the beam splitter 1112 and the first polarizer 1116, is prevented from reaching the camera component 117. This ensures that the camera component 117 can only capture the image of the sample 114 to be observed, without affecting the camera's imaging capabilities. The polarization directions of the first polarizer 1116 and the second polarizer 1117 are perpendicular to each other. Before the camera component 117 acquires the image in the corresponding field of view, its focal length needs to be adjusted. Specifically, the defocusing parameters can be determined from the image in the second optical path.

[0112] The second optical path includes: light from the auxiliary focusing light source 1140 in the Fourier back focal plane passes through the objective lens 115 and reaches the beam splitter 1112. The beam splitter 1112 transmits the light to the tube lens 1118. Finally, the infrared light emitted by the infrared LED, which serves as the auxiliary focusing light source, is transmitted through the tube lens 1118 to the auxiliary focusing camera 1143, where it forms an image (partially overlapping image).

[0113] Furthermore, in some embodiments of the present invention, the focusing device 1121 is located between the beam splitter 1112 and the camera assembly 117, and is used to drive the first lens to adjust its focal length based on the focal length determined by the defocusing amount of the overlapping image, thereby forming a new focal length. Specifically, the focusing device 1121 can be an electric motor, such as an ultrasonic drive motor, or other mechanical motor that can be used to drive the lens group accordingly; it can also be a liquid zoom lens that performs liquid zoom independently of the lens group to adapt to different usage environments.

[0114] In some embodiments of the present invention, a focusing device 1121 is located between the beam splitter 1112 and the camera assembly 117, and is used to adjust the focal length by means of a variable focal length determined by the defocusing amount of the overlapping image through a variable focal length lens to form the new focal length; the camera assembly 117 is used to take pictures of the sample to be observed in the microscope field of view based on the new focal length to form and output a clear and focused image taken through the first optical path.

[0115] In some embodiments of the present invention, considering that camera interfaces may not be standardized, in order to be compatible with multiple cameras, or to expand or narrow the field of view, the camera can also be used in conjunction with a camera adapter. The camera based on the photosensitive chip is connected to the camera interface at the top of the trinocular observation tube through the camera adapter, thereby realizing the connection between the camera and the trinocular observation tube. The camera adapter may also embed a polarizer, which can filter out light rays with polarization states perpendicular to the polarizer to avoid interference with imaging.

[0116] refer to Figure 7 , Figure 7This is an optional structural schematic diagram of a microscope system provided in an embodiment of the present invention. Specifically, the microscope system 1100 includes an objective lens 115, a beam splitter 1112, an image projection assembly 1111, a camera assembly 117, and a trinocular tube 116. The objective lens 115 has a first end 10a and a second end 10b disposed opposite to each other. The first end 10a faces the sample to be observed. The beam splitter 1112 is disposed at the second end 10b and communicates with both the objective lens 115 and the tube end 1118 of the trinocular tube. The camera assembly 117 is disposed in one of the channels of the trinocular tube. The image projection assembly 1111 projects light transmitted through the lens 1115 into a corresponding field of view. The image projection is obtained by the camera assembly 117 receiving light transmitted from the tube lens 1118. The camera assembly 117 includes a camera and a corresponding image output device for transmitting the captured image in the corresponding field of view to a server for image processing or recognition. The trinocular tube 116 is located at the end of the beam splitter 1112 away from the objective lens 115. The trinocular tube 116 includes channels and a tube lens 1118. The channels include at least two and are located at the end away from the beam splitter 1112. The tube lens 1118 is located at the end close to the beam splitter 1112. The camera assembly 117 receives the light output from the beam splitter 1112 through the tube lens 1118 to complete the acquisition of the image in the corresponding field of view. Since the camera assembly 117 and the image projection assembly 1111 are located in different positions, in order to avoid the influence of light propagation, the image projection assembly 1111 further includes a second polarizer 1117. The second polarizer 1117 is located between the lens 1115 and the beam splitter 1112, and is used to polarize the corresponding light in the first optical path. The camera assembly also includes a first polarizer 1116, which is located between the tube lens 1118 and the camera assembly 117, and is used to polarize the corresponding light collected by the camera assembly 117. Specifically, since the microscope operator wants to observe both the image of the sample 114 and the image output from the image projection assembly 1111 through the eyepiece 118, and the camera assembly 117 only wants to capture the image of the sample 114 and ignore the image output from the image projection assembly 1111, the second polarizer 1117 can convert the light output from the image projection assembly 1111 into polarized light, which can be directly observed by the human eye through the eyepiece 118. However, since the polarization directions of the first polarizer 1116 and the second polarizer 1117 are perpendicular to each other, the first polarizer 1116 can eliminate the polarized light output from the second polarizer 1117, so that the camera assembly 117 can only capture the image of the sample 114.

[0117] Meanwhile, in order to achieve automatic focusing, the microscope system 1100 also includes: an auxiliary focusing light source 1140 disposed in the Fourier back focal plane corresponding to the condenser lens group 1141, for emitting auxiliary focusing light. The auxiliary focusing light source 1140 can be two identical infrared LED light emitters. The auxiliary focusing camera 1143 is disposed at a horizontally symmetrical position of the image projection assembly 1111 and receives the light in the second optical path refracted by the beam splitter 112 through the lens 1119. The second optical path formed by the light generated by the auxiliary focusing light source 1140 is shown in Figure 1144.

[0118] Furthermore, the optical path of the microscope system 1100 includes a first optical path and a second optical path. The first optical path includes: light from the objective lens 115 is transmitted to a beam splitter 1112; the beam splitter 1112 reflects a portion of the light onto the tube mirror 1118, passes through the first lens 1116, and is transmitted to the photosensitive chip of the camera assembly 117; simultaneously, the beam splitter 1112 transmits a portion of the light to the tube mirror 1118, passes through the tube mirror 1118, and reflects the transmitted light through the tube mirror 1118 to reach the trinocular tube 116; the trinocular tube 116 transmits the light to the eyepiece 118, through which the image of the sample 114 can be observed. Simultaneously, the image projection assembly 1111 generates… The generated light rays travel along lens 1115, where the polarization directions of the first polarizer 1116 and the second polarizer 1117 are perpendicular to each other. After polarization processing by the second polarizer 1117, and through the beam splitter 1112 and the action of the first polarizer 1116, the light rays cannot reach the camera assembly 117. This ensures that the camera assembly 117 can only capture images of the sample 114 being observed, thus preventing the light output from the image projection assembly 1111 from affecting the camera assembly 117's image capture. Simultaneously, the microscope operator wants to be able to observe both the image of the sample 114 and the image output from the image projection assembly 1111 through the eyepiece 118. Before the camera assembly 57 acquires images in the corresponding field of view, the focal length of the camera assembly 117 needs to be adjusted. Specifically, the corresponding defocusing parameters can be determined by the image in the second optical path.

[0119] The second optical path includes: light from the auxiliary focusing light source 1140 in the Fourier back focal plane passes through the objective lens 115 and reaches the beam splitter 1112. The beam splitter 1112 transmits the light to the lens 1119, and finally the infrared light emitted by the infrared LED, which is the auxiliary focusing light source, is transmitted through the lens 1119 to the auxiliary focusing camera 1143, where it forms an image (partially overlapping image).

[0120] Furthermore, in some embodiments of the present invention, the focusing device 1121 is located between the beam splitter 1112 and the camera assembly 117, and is used to drive the first lens to adjust the focal length based on the focal length determined by the defocusing amount of the overlapping image, thereby forming a new focal length.

[0121] In some embodiments of the present invention, a focusing device 1121 is located between the beam splitter 1112 and the camera assembly 117, and is used to adjust the focal length by means of a variable focal length determined by the defocusing amount of the overlapping image through a variable focal length lens to form the new focal length; the camera assembly 117 is used to take pictures of the sample to be observed in the microscope field of view based on the new focal length to form and output a clear and focused image taken through the first optical path.

[0122] The following will continue to combine Figure 3 The method shown and Figures 5 to 7 The different microscope system configurations shown illustrate the microscope autofocus method provided by this invention, wherein, reference... Figure 8 , Figure 8 This is a schematic diagram illustrating the relationship between defocusing amount and ghosting distance in an embodiment of the present invention. Infrared light emitted by the infrared LED, serving as an auxiliary focusing light source, is transmitted to the auxiliary focusing camera 1143 and then imaged at the auxiliary focusing camera 1143 (partially overlapping images). The auxiliary focusing camera can be a standard industrial camera with the infrared filter removed, or a dedicated infrared camera capable of capturing infrared light. The camera in the photographic assembly is typically a color camera, and color cameras usually have an infrared cut-off filter in front of them; therefore, the infrared light generated by the infrared LED 40 is not captured by the camera. Of course, an additional infrared cut-off filter can be added in front of the camera to achieve a better effect of filtering out infrared light.

[0123] Among them, the peak position of the autocorrelation result obtained after performing autocorrelation calculation on the image acquired by the auxiliary focusing camera will change under different defocusing amounts.

[0124] The following theoretical derivation can be used:

[0125] Suppose the image acquired by the assisted focusing camera is z[x] = s[x] + s[x-x0], where s[x] and s[x-x0] are two ghost images with a distance of x0 between them. z[x] can also be expressed as z[x] = s[x] * h[x]. Here, '*' represents the convolution symbol, and h[x] = δ[x] + δ[x-x0].

[0126] By performing autocorrelation on z[x], we obtain R(z[x]) = R(s[x]) * R(h[x]) = R(s[x]) * (2δ[x] + δ[x-x0] + δ[x+x0]). Here, 'R()' represents the autocorrelation operator. 2δ[x] + δ[x-x0] + δ[x+x0] represents three delta functions. R(s[x]) * (2δ[x] + δ[x-x0] + δ[x+x0]) represents the convolution of R(s[x]) and the three delta functions. This means that the result of R(z[x]) will form three peaks through the algorithm. One peak is the highest and located in the middle, while the other two peaks are located on either side of this highest peak, each at a distance x0 from the peak. This also means that if we can determine the distance between any two of the three peaks formed by the algorithm, we can obtain x0, which is the distance between the two ghost images captured by the camera.

[0127] Continuing with the example of observing mouse kidney slices under a microscope, the automatic focusing method of the microscope provided by this invention will be explained, wherein, as Figure 5 The image shown is a sample ghosting formed by two infrared LEDs on the surface of an auxiliary focusing camera (the image shows a screenshot of a mouse kidney slice under a 20x objective lens).

[0128] Continue to refer to Figure 8 Where: (a1), (b1), and (c1) show the focused images of mouse kidney slices in the corresponding fields of view acquired by the auxiliary focusing camera at different defocusing levels. (a2), (b2), and (c2) are the autocorrelation results of each image in (a1), (b1), and (c1), respectively.

[0129] Figure 8 Images (a1)-(c1) represent infrared images containing two ghost images acquired by the auxiliary focusing camera in the microscope system of the preceding different embodiments. The distance between the two ghost images varies depending on the defocus level of the sample. By performing some calculations (including but not limited to autocorrelation calculations) on the images acquired by the auxiliary focusing camera, a curve relating the sample defocus level to the distance between the two ghost images in the image is obtained. This fitted curve is then used as a reference table for the subsequent focusing process to achieve automatic focusing.

[0130] Continue to refer to Figure 9 , Figure 9 This is a schematic diagram illustrating the fitting of the relationship between defocus amount and ghost distance in this invention. A schematic diagram of a curve fitted based on the relationship between defocus amount and ghost distance is shown below. Figure 9 The fitting curve shows the relationship between the sample defocus amount and the ghosting in the image acquired by the auxiliary focusing camera.

[0131] Figure 9The reason why the curve shows a monotonically increasing trend is that the auxiliary focusing camera is offset relative to the camera components. In this figure, the offset is 60 micrometers, as shown at the sixth point in the center of the figure. Figure 9 The display shows the sample defocus amount of -30 micrometers ( Figure 9 From 30 micrometers in the middle, i.e., the first point) to +30 micrometers ( Figure 7 The curve is fitted from 11 ghost images acquired at 90 micrometers (point 11). An offset is set because, theoretically, the fitted curve would be close to a "V" shape without it. However, near the point of focus, the distance between the two ghost images is very close. According to the autocorrelation calculation method described above, the distance between the three peaks would also be very close, causing the peak values ​​to be submerged or new, unexpected, and irrelevant peaks to appear. This makes it difficult to find the desired peak location. Therefore, an offset is introduced here to increase the distance between the three peaks.

[0132] It should be noted that by fitting different index values ​​to a curve, the vertex of the curve represents the position corresponding to zero defocus. The closer the defocus is to zero, the sharper the image, which is the position that the focusing device needs to adjust the camera to.

[0133] Beneficial technical effects:

[0134] This invention provides an embodiment of the invention that acquires measurement samples captured by an auxiliary focusing camera in the second optical path of a microscope; calculates corresponding image evaluation parameters based on the measurement samples captured by the auxiliary focusing camera and corresponding image evaluation standards; finds the relationship between the image evaluation parameters and the defocus amount in a pre-stored calibration curve based on the image evaluation parameters, thereby determining the required defocus amount; and adjusts the focal length of the image light entering the camera assembly based on the determined defocus amount, so that the camera assembly can capture a clearly focused image through the first optical path. Thus, the focusing device can automatically focus on the camera assembly of the microscope system, forming and outputting a clearly focused image captured through the first optical path, saving focusing time and improving focusing accuracy.

[0135] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A microscope system, characterized in that, The microscope system includes: A camera assembly and a focusing device are disposed in the first optical path. The first optical path also includes an objective lens, a beam splitter, and a trinocular tube. The beam splitter is disposed at the end of the objective lens away from the sample to be observed. The beam splitter is connected to the tube lens of the objective lens and the trinocular tube, respectively. The camera assembly is disposed in one of the channels of the trinocular tube. The focusing device is located between the beam splitter and the camera assembly. An auxiliary focusing device is provided in the second optical path, which further includes a condenser lens assembly, the objective lens, the beam splitter, and the trinocular tube. The condenser lens assembly is located at the end of the objective lens facing the sample to be observed. The auxiliary focusing device includes an auxiliary focusing light source and an auxiliary focusing camera. The auxiliary focusing light source is located on the Fourier back focal plane corresponding to the condenser lens assembly, and the auxiliary focusing camera is located at an axially offset position on the conjugate plane of the camera assembly. The auxiliary focusing light source includes: two identical infrared light emitters for emitting infrared light; and an auxiliary focusing camera for acquiring two infrared images of the sample to be observed after the infrared light is transmitted, wherein there is a ghosting between the two infrared images, and a defocus amount is determined based on the distance between the two infrared images, wherein the defocus amount is used to indicate the focal length of the image light entering the camera assembly.

2. The microscope system according to claim 1, characterized in that, The beam splitter is used to reflect or refract light from the second optical path into the auxiliary focusing camera.

3. The microscope system according to claim 1 or 2, characterized in that, The auxiliary focusing device is used to determine the defocus amount corresponding to the sample to be observed; The focusing device is used to adjust the focal length of the image light entering the camera assembly according to the defocus amount corresponding to the sample to be observed.

4. A method for automatic focusing of a microscope, characterized in that, The method includes: Two infrared images of the sample to be observed are obtained by transmitting infrared light emitted from two identical infrared emitters. There is a ghosting between the two infrared images. The defocusing amount is determined based on the distance between the two infrared images. Adjust the focal length of the image light entering the camera assembly according to the defocus amount corresponding to the sample to be observed; The microscope system includes: a camera assembly and a focusing device disposed in a first optical path, the first optical path further including an objective lens, a beam splitter, and a trinocular tube, the beam splitter being disposed at the end of the objective lens away from the sample to be observed, the beam splitter being connected to the tube ends of the objective lens and the trinocular tube respectively, the camera assembly being disposed in one of the channels of the trinocular tube, and the focusing device being located between the beam splitter and the camera assembly; and an auxiliary focusing device disposed in a second optical path, the second optical path further including a condenser lens assembly, the objective lens, the beam splitter, and the trinocular tube, the condenser lens assembly being disposed at the end of the objective lens facing the sample to be observed, the auxiliary focusing device including: an auxiliary focusing light source and an auxiliary focusing camera, the auxiliary focusing light source being disposed at the Fourier back focal plane corresponding to the condenser lens assembly, the auxiliary focusing camera being disposed at an axially offset position on the conjugate plane of the camera assembly, and the auxiliary focusing light source including: two identical infrared light emitters, the two identical infrared light emitters being used to emit infrared light.

5. A medical device, characterized in that, The medical device includes: The system includes a microscope system, a memory, and a processor, wherein the microscope system is the microscope system according to any one of claims 1 to 3, and the processor performs the following steps: Memory, used to store executable instructions; The processor, when executing the executable instructions stored in the memory, implements the microscope autofocus method of claim 4.

Images