Self-adaptive focusing microscope equipment

Abstract

The invention discloses self-adaptive focusing microscope equipment, which adopts a data processor of a ZYNQ heterogeneous multi-core architecture, realizes automatic focusing through steps S1 to S5, is based on a parallel processing architecture of a PL unit, namely an FPGA (Field Programmable Gate Array), and can execute image convolution operation of multiple paths of images in parallel in the step S3. In the step S1, the line-by-line scanning process of the microscopic image in the step S1 cannot be synchronously executed, so that most calculation of the image definition evaluation function is completed in the step S3 while the microscopic image scanning is completed in the step S1, and after the microscopic image scanning is completed, the image definition evaluation function is calculated in the step S3. The image definition evaluation function can calculate the definition evaluation value of the microscopic image in an extremely short time; therefore, a camera + ZYNQ heterogeneous multi-core architecture is used for replacing a serial execution architecture of an industrial camera + an upper computer in the prior art, and the definition evaluation value can be obtained only through scanning time + extremely short time of a microscopic image, so that the time for obtaining the definition evaluation value is greatly shortened, and the focusing speed is improved.

Description

Technical Field

[0001] The present invention relates to a microscope device, specifically an adaptive focusing microscope device. Background Art

[0002] The structure of the microscope device commonly used in the current market consists of a microscope body, a USB camera, a PC, a monitor, a chip on board, a motor driver, and a transmission mechanical structure. The microscope body includes a light source system, but generally has no intelligent control function, and the light source brightness is manually adjusted by humans. Generally, an industrial camera is installed at the top of the microscope. After scanning an image through the industrial camera, it is transmitted to the PC through a USB or Ethernet interface. The PC usually uses a QT upper computer or Labview to implement a visualization interface and issue control commands. At the same time, an embedded hardware operating system is built externally, and the control commands are sent to the embedded chip through a serial port or other means. The chip controls the motor driver to drive the motor to realize the up and down movement of the stage. There is a transmission structure between the motor and the stage, and the stage is driven to move up and down stably through the transmission structure. Generally, the microscope body is also equipped with manual adjustment buttons, and the up and down movement of the stage or the XY-axis plane movement can be controlled by a manual knob.

[0003] Currently, the focusing speed of similar microscope device products is not fast enough. Especially in an environment with multiple sample detections, the focusing process accumulated by multiple samples will take a lot of time. Therefore, focusing on a single sample that can ensure both stability and speed is the main development direction in the future. Especially in some application scenarios that broaden the microscopic field of view, a large number of microscopic pictures need to be taken. During this process, it is inevitable to be out of focus due to changes in the sample height. To avoid this situation, it is necessary to quickly judge the out-of-focus state of the current picture and execute the autofocus process. For example, for a 20*20 array, 400 pictures need to be scanned. If the autofocus process is performed for all 400 pictures, then by increasing the autofocus rate by 1 s, the entire scanning process can save 400 s of scanning time.

[0004] The main reason for the bottleneck in the focusing speed of current similar microscope equipment products lies in the use of the PC side for image algorithm evaluation. As mentioned before, the working process of the traditional autofocus system on the market currently is as follows: Step 1, scan through an industrial camera to obtain a microscopic image on the current stage; Step 2, transfer the microscopic image data to the memory of the PC side through a USB or Ethernet interface; Step 3, the PC side reads the microscopic image data in the memory through the CPU; Step 4, the PC side executes the corresponding image sharpness evaluation algorithm on the microscopic image data to obtain the image sharpness evaluation value of the microscopic image; Step 5, according to the comparison result between the image sharpness evaluation value and the image sharpness evaluation value of the previous microscopic image, perform motor motion control to make the stage move in the direction where the image sharpness evaluation value is better; Step 6, repeat the above steps until the vicinity of the focal plane is traced. The existing working process of the autofocus system has the following problems: The architecture is completely serial in the processes of image transfer, image storage, image reading, and execution of the image algorithm, without making full use of time. Especially in the entire focusing process of the microscope, we often need more than a hundred evaluation processes to finally reach the vicinity of the most accurate focal plane. The superposition of hundreds of evaluation processes will result in more than half of the time being spent on the repeated image transfer process, causing the problem of slow focusing speed. Since the architecture executed by the CPU itself is serial, there is no other way to significantly improve the speed except by increasing the working frequency of the CPU itself to improve the operation speed. However, since this is a systematic product and the main frequency performance of each computer is inconsistent, the impact on the speed is very large. Therefore, a new architecture needs to be adopted and the original working process abandoned to achieve high-speed and high-precision autofocus. Otherwise, it is impossible to be compatible with high speed and low cost from the perspective of cost.

[0005] In addition, the market needs a microscope equipment product with a fully automated process. Currently, many autofocus microscope equipment products need to click on the upper computer to complete the focusing process. Different objective lens magnifications generally correspond to different focusing strategies, so repeated selection and clicking will be rather troublesome. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an adaptive focusing microscope equipment.

[0007] To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] An adaptive focusing microscope equipment includes a camera and a motor module; the camera is used to scan the microscope stage to obtain a microscopic image, and the motor module drives the microscope stage to move to the focal plane according to the focusing control instruction to obtain a sufficiently clear observation effect on the items on the microscope stage.

[0009] It is characterized in that:

[0010] It further includes: a data processor, which adopts a ZYNQ heterogeneous multi-core architecture composed of a PS unit and a PL unit, and preferably adopts a Zynq-7020 development board; among them, the PS unit is a common ARM unit (suitable for processing floating-point algorithms and belonging to a serial architecture), and the PL unit is an FPGA system-on-chip (suitable for processing high-speed parallel data streams);

[0011] The adaptive focusing microscope device performs automatic focusing control according to the following process:

[0012] Step S1: The camera scans to obtain a microscopic image in a line-by-line scanning manner, and synchronously outputs the line scanning data of each line of the microscopic image obtained by scanning to the PL unit in the form of a Bayer array format video stream data; among them, the camera is preferably an OV5640, and the clock frequency of the Bayer array format video stream data is preferably 72M, and the resolution is preferably 720P.

[0013] Step S2: Each time the PL unit receives a line of microscopic image line scanning data, it performs pre-stage image processing on it, including: the Bayer array format video stream data first caches the data through a FIFO memory to perform cross-clock domain synchronization with the internal clock domain of the PL unit; the synchronized data is converted into RGB888 video stream data through a color interpolation reduction algorithm to restore the true color; the RGB888 video stream data is converted into YCbcr video stream data through a color space conversion algorithm, and the Y component is extracted as grayscale data to reduce the computational complexity of the subsequent evaluation function algorithm; the grayscale data is filtered by a Gaussian noise filter to obtain a pure grayscale data map;

[0014] Step S3: Starting from the scanning of the third line of microscopic image line scanning data by the camera, each time a line of microscopic image line scanning data starts to be scanned, an image convolution operation in the image sharpness evaluation function is performed on the grayscale data map obtained by processing the first two lines of microscopic image line scanning data in Step S2 until the scanning of the last line of microscopic image line scanning data of the microscopic image is completed and the processing of Step S2 is completed, and the sharpness evaluation value of the microscopic image is calculated based on the result of the image convolution operation by the image sharpness evaluation function;

[0015] Step S4: The PL unit transmits the sharpness evaluation value to the DDR of the PS unit and sends an interrupt notification to the PS unit to prompt that the PL unit has obtained the sharpness evaluation value and the PS unit needs to read it from the DDR; among them, the PL unit preferably sends the sharpness evaluation value to the AXI_lite module suitable for transmitting a small amount of data and transmits it to the DDR of the PS unit through the AXI bus;

[0016] Step S5: When the automatic focusing function is enabled, upon receiving the interruption notification, the PS unit reads the sharpness evaluation value from the DDR and enters the automatic focusing process to perform automatic focusing based on the sharpness evaluation value, generating the focusing control instruction, enabling the motor module to drive the microscope stage to move to the focal plane according to the focusing control instruction.

[0017] Therefore, the present invention adopts a data processor with a ZYNQ heterogeneous multi-core architecture and realizes automatic focusing through steps S1 to S5. Based on the parallel processing architecture of the PL unit, i.e., FPGA, multiple-channel image convolution operations can be executed in parallel in step S3, without affecting the synchronous execution of the line-by-line scanning process of the microscopic image in step S1. This enables the completion of the vast majority of the calculations of the image sharpness evaluation function (i.e., the image convolution operation) in step S3 while the microscopic image scanning in step S1 is completed. When the microscopic image scanning is completed, the sharpness evaluation value of the microscopic image can be calculated within an extremely short time (generally 250 ns). Thus, the present invention replaces the serial execution architecture of the industrial camera + host computer in the prior art with a camera + ZYNQ heterogeneous multi-core architecture, and only requires the scanning time of the microscopic image + an extremely short time (250 ns) to obtain the sharpness evaluation value required for automatic focusing. Compared with the time required by the prior art (the scanning time of the microscopic image + the time for image transmission to the host computer, about 14 - 38 ms + the time for memory reading, about 1 - 5 ms + the execution time of the image sharpness evaluation function, about 10 - 15 ms), the present invention greatly shortens the time from the start of scanning to obtaining the sharpness evaluation value, effectively improving the focusing speed of the microscope's automatic focusing.

[0018] Compared with the execution by the host computer in the prior art, if the working frequency of the device is 60 frames per second, the host computer will waste 30 frames of time in the intermediate invalid data transfer process, and the actual effective number of operations is only 30 times. In contrast, the present invention can complete the focusing process for each frame, achieving 60 focusing processes per second, realizing the most efficient use of time. According to experiments, the present invention can complete automatic focusing in 1 - 2 s at short distances, 2 - 6 s at medium and long distances, and can complete automatic focusing in as fast as 0.812 s.

[0019] In addition, the present invention can achieve the image focusing process at full frame rate. There will be a blanking period during the scanning process of the camera and the camera. By making full use of the blanking device to control the rotation of the motor, it will not interfere with the image evaluation and focusing process of the next frame.

[0020] Preferably, in step S3, the image sharpness evaluation function includes three evaluation functions, namely: a linearity evaluation function with a single peak and good monotonicity, a first sensitivity evaluation function with high sensitivity at the peak point, and a second sensitivity evaluation function; the gray-scale data map output by the Gaussian noise filter is evaluated in parallel by these three evaluation functions to obtain three sharpness evaluation values of the microscopic image;

[0021] See Figure 2 and Figure 3 , in step S5, the automatic focusing process adopts a multi-operator collaborative evaluation system, including:

[0022] Step S5-1: Obtain three sharpness evaluation values of the microscopic image obtained in the current scan and the microscopic image obtained in the previous scan, which are respectively denoted as: the evaluation value of the current linearity evaluation function, the evaluation value of the previous linearity evaluation function, the evaluation value of the current first sensitivity evaluation function, the evaluation value of the previous first sensitivity evaluation function, the evaluation value of the current second sensitivity evaluation function, and the evaluation value of the previous second sensitivity evaluation function;

[0023] Step S5-2: Determine whether the following two conditions are simultaneously satisfied: the change rate of the evaluation value of the current first sensitivity evaluation function relative to the evaluation value of the previous first sensitivity evaluation function is above the preset change rate threshold; and, the change rate of the evaluation value of the current second sensitivity evaluation function relative to the evaluation value of the previous second sensitivity evaluation function is above the preset change rate threshold;

[0024] Among them, if the judgment result is no, then: input the evaluation value of the current linearity evaluation function into the hill climbing search algorithm;

[0025] Among them, if the judgment result is yes, then: input the evaluation value of the current first sensitivity evaluation function and the evaluation value of the current second sensitivity evaluation function into the hill climbing search algorithm respectively;

[0026] Step S5-3: Generate the focusing control instruction according to the judgment result of step S5-2, including: first obtain the movement direction and step distance of the motor module driving the microscope stage to move through the hill climbing search algorithm, and then perform a peak search operation to use the peak point determined by the peak search operation as the moment when the motor module stops driving the microscope stage to move;

[0027] Among them, for the case where the judgment result of step S5-2 is yes, when peak points appear respectively when the evaluation value of the current first sensitivity evaluation function and the evaluation value of the current second sensitivity evaluation function are input into the hill climbing search algorithm, it is determined as the peak point determined by the peak search operation.

[0028] Among them, the existing hill-climbing search algorithm changes the direction when there is a continuous decline for once and reduces the step size for searching. The hill-climbing search algorithm of the present invention is preferably changed to change the direction only after six consecutive declines. Although this will extend the time of partial autofocus, it can effectively reduce the possibility of false detection.

[0029] Therefore, through the judgment in step S5-2 of the present invention, it is made that: when the microscope stage is in the far-focus area far from the focal plane, a judgment result of no is obtained, and the evaluation value of the current linearity evaluation function is input into the hill-climbing search algorithm and the peak search operation is performed accordingly, which can avoid the interference points in the far-focus area during the peak search operation, withstand the influence of local extreme points during the peak search process, and avoid the situation that the clarity evaluation value is less affected by the movement of the microscope stage in the far-focus area and is prone to irregular changes, resulting in focusing errors, ensuring that the movement direction of the microscope stage obtained by the hill-climbing search algorithm is the correct direction moving towards the focal plane, and avoiding the situation of stopping the evaluation due to an incorrect evaluation direction; when the microscope stage moves to the near-focus area close to the focal plane, a judgment result of yes is obtained, and the evaluation value of the current first sensitivity evaluation function and the evaluation value of the current second sensitivity evaluation function are respectively input into the hill-climbing search algorithm, which can ensure the focusing accuracy by using the sensitivity evaluation function with high sensitivity; and since two sensitivity evaluation functions are simultaneously used as the input source of the hill-climbing search algorithm, the reliability of judging the boundary line between the far-focus area and the near-focus area is further improved and the focusing accuracy is further improved; therefore, the present invention has the advantages of high stability, high reliability, high success rate, and high precision in autofocus.

[0030] After testing, the short-distance focusing success rate of the present invention is 99.7%, and the long-distance focusing success rate is 98.0%.

[0031] Preferably: the linearity evaluation function is the Variance evaluation function, the first sensitivity evaluation function is the Brenner evaluation function, and the second sensitivity evaluation function is the Tenegard evaluation function.

[0032] Preferably: in step S2, the grayscale data map output by the Gaussian noise filter also passes through a Gaussian blur filter to become a blurred grayscale data map;

[0033] In step S3, the blurred grayscale data map and the grayscale data Figure 1 Similarly, both pass through an image clarity evaluation function including three evaluation functions in parallel for evaluation, and three clarity evaluation values of the microscopic image obtained by the current scan are obtained, which are respectively recorded as the blurred evaluation value of the current linearity evaluation function, the blurred evaluation value of the current first sensitivity evaluation function, and the blurred evaluation value of the current second sensitivity evaluation function;

[0034] While performing the step S5-2, when the dynamic defocus detection function is enabled, the PS unit determines whether the following conditions are satisfied simultaneously. If so, it is determined that the microscopic image obtained in this scan is defocus blurred, and dynamic defocus correction should be immediately performed to ensure that the microscopic image is clearly visible;

[0035] Conditions: The change rate of the fuzzy evaluation value of the current linearity evaluation function relative to the evaluation value of the current linearity evaluation function is above the preset fuzzy change rate threshold; and, the change rate of the fuzzy evaluation value of the current first sensitivity evaluation function relative to the evaluation value of the current first sensitivity evaluation function is above the preset fuzzy change rate threshold; and, the change rate of the fuzzy evaluation value of the current second sensitivity evaluation function relative to the evaluation value of the current second sensitivity evaluation function is above the preset fuzzy change rate threshold.

[0036] Thus, by passing the grayscale data map through a Gaussian blur filter in step S2 to become a blurred grayscale data map, it is equivalent to putting a mosaic on the picture, making it more blurred. And for the more blurred image, in step S3, the image sharpness evaluation functions of three evaluation functions are used to evaluate in parallel. If the original image corresponding to the grayscale data map is itself a blurred image, then the evaluation values of the blurred image corresponding to the blurred grayscale data map and the original image are not much different. However, if the original image is a clear image, then the evaluation value of the blurred image will change significantly. Therefore, through the dynamic defocus detection in step S5-2, it can be determined whether the microscopic image obtained in this scan is defocus blurred, and when the determination is yes, dynamic defocus correction is performed to ensure that the microscopic image is clearly visible.

[0037] Preferably: The adaptive focusing microscope device further includes an objective turret identification module and a high-power LED light source;

[0038] The objective turret identification module determines the magnification of the current microscope objective by installing magnets with different arrangement methods on each objective magnification channel of the microscope objective turret and identifying the arrangement method of the magnets through a Hall element;

[0039] The PS unit obtains the magnification of the current microscope objective in real time by detecting the level state of the Hall element, and controls the high-power LED light source to adjust the power according to this magnification, so that the brightness of the high-power LED light source adapts to the magnification of the current microscope objective, realizing a brighter observation field of view, and taking into account both the focusing accuracy and the focusing duration.

[0040] Preferably: The PS unit also sets the initial step size of the hill climbing search algorithm in step S5-3 according to the magnification of the current microscope objective to obtain the highest focusing accuracy.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] First, the present invention uses a data processor with a ZYNQ heterogeneous multi-core architecture and realizes autofocus through steps S1 to S5. Based on the parallel processing architecture of the PL unit, i.e., FPGA, in step S3, the image convolution operations of multiple images can be executed in parallel, without affecting the line-by-line scanning process of the microscopic image in step S1. This enables most of the calculations of the image sharpness evaluation function (i.e., the image convolution operation) to be completed in step S3 while the microscopic image is being scanned in step S1. When the scanning of the microscopic image is completed, the sharpness evaluation value of the microscopic image can be calculated within an extremely short time (generally 250 ns). Therefore, the present invention replaces the serial execution architecture of an industrial camera + host computer in the prior art with a camera + ZYNQ heterogeneous multi-core architecture, and only needs the scanning time of the microscopic image + an extremely short time (250 ns) to obtain the sharpness evaluation value required for autofocus. Compared with the time required by the prior art (the scanning time of the microscopic image + the time for image transmission to the host computer, about 14 - 38 ms + the time for memory reading, about 1 - 5 ms + the execution time of the image sharpness evaluation function, about 10 - 15 ms), the present invention greatly shortens the time from the start of scanning to obtaining the sharpness evaluation value, and can effectively improve the focusing speed of the microscope autofocus.

[0043] Second, through the judgment in step S5-2, the present invention enables: when the microscope stage is in a far-focus region far from the focal plane, a judgment result of no is obtained, and the evaluation value of the current linearity evaluation function is input into the hill-climbing search algorithm for peak searching. This can avoid the interference points of the peak searching operation in the far-focus region, withstand the influence of local extreme points during the peak searching process, and avoid the situation where the sharpness evaluation value is less affected by the movement of the microscope stage in the far-focus region and is prone to irregular changes, resulting in focusing errors. It ensures that the movement direction of the microscope stage obtained by the hill-climbing search algorithm is the correct direction towards the focal plane, and avoids the situation where the evaluation direction is incorrect and the evaluation stops. When the microscope stage moves to a near-focus region close to the focal plane, a judgment result of yes is obtained, and the evaluation value of the current first sensitivity evaluation function and the evaluation value of the current second sensitivity evaluation function are respectively input into the hill-climbing search algorithm. This can ensure the focusing accuracy by using the sensitivity evaluation function with high sensitivity. And since two sensitivity evaluation functions are simultaneously used as the input sources of the hill-climbing search algorithm, the reliability of determining the boundary between the far-focus region and the near-focus region is further improved and the focusing accuracy is further enhanced. Therefore, the present invention has the advantages of high stability, high reliability, high success rate, and high precision in autofocus.

[0044] Thirdly, in the present invention, in step S2, the grayscale data map also passes through a Gaussian blur filter to become a blurred grayscale data map, which is equivalent to adding a mosaic to the picture and will be more blurred. Then, for the more blurred image, in step S3, the image sharpness evaluation functions of three evaluation functions are used to evaluate in parallel. If the original image corresponding to the grayscale data map is a blurred image itself, then the evaluation values of the blurred image corresponding to the blurred grayscale data map and the original image are not much different. However, if the original image is a clear image, then the evaluation value of the blurred image will change significantly. Thus, through the dynamic defocus detection in step S5-2, it can be determined whether the microscopic image obtained in this scan is out of focus and blurred, and when the determination is yes, dynamic defocus correction is performed to ensure that the microscopic image is clearly visible. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described in detail below with reference to the drawings and specific embodiments:

[0046] Figure 1 is a circuit principle block diagram of the adaptive focusing microscope device of the present invention;

[0047] Figure 2 is a data flow diagram of the data processor and the host computer in the present invention;

[0048] Figure 3 is a flow chart of the multi-operator collaborative evaluation system in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] The present invention will be described in detail below with reference to the embodiments and their accompanying drawings to help those skilled in the art better understand the inventive concept of the present invention. However, the protection scope of the claims of the present invention is not limited to the following embodiments. For those skilled in the art, all other embodiments obtained without creative labor on the premise of not departing from the inventive concept of the present invention belong to the protection scope of the present invention.

[0050] Embodiment 1

[0051] As Figure 1 and Figure 2 shown, the present invention discloses an adaptive focusing microscope device, including a camera and a motor module; the camera is used to scan the microscope stage to obtain a microscopic image, and the motor module drives the microscope stage to move to the focal plane according to the focusing control instruction to obtain a sufficiently clear observation effect on the items on the microscope stage;

[0052] It further includes: a data processor, which adopts a ZYNQ heterogeneous multi-core architecture composed of a PS unit and a PL unit, preferably using a Zynq-7020 development board; wherein, the PS unit is a common ARM unit (suitable for processing floating-point algorithms and belonging to a serial architecture), and the PL unit is an FPGA system-on-chip (suitable for processing high-speed parallel data streams);

[0053] The adaptive focusing microscope device performs automatic focusing control according to the following process:

[0054] Step S1: The camera scans to obtain a microscopic image by line-by-line scanning, and synchronously outputs the line-scanning data of each line of the microscopic image obtained by scanning to the PL unit in the form of a Bayer array format video stream data; wherein, the camera is preferably an OV5640, and the clock frequency of the Bayer array format video stream data is preferably 72M, and the resolution is preferably 720P.

[0055] Step S2: Every time the PL unit receives a line of microscopic image line-scanning data, it performs pre-stage image processing on it, including: the Bayer array format video stream data first caches the data through a FIFO memory to perform cross-clock domain synchronization with the internal clock domain of the PL unit; the synchronized data is converted into RGB888 video stream data through a color interpolation reduction algorithm to restore the true color; the RGB888 video stream data is converted into YCbcr video stream data through a color space conversion algorithm, and the Y component is extracted as grayscale data to reduce the computational amount of the subsequent evaluation function algorithm; the grayscale data is filtered by a Gaussian noise filter to obtain a pure grayscale data map;

[0056] Step S3: Starting from the scanning of the third line of microscopic image line-scanning data by the camera, every time a line of microscopic image line-scanning data starts to be scanned, an image convolution operation in the image sharpness evaluation function is performed on the grayscale data map obtained by processing the first two lines of microscopic image line-scanning data in Step S2, until the scanning of the last line of microscopic image line-scanning data of the microscopic image is completed and the processing in Step S2 is completed, and the sharpness evaluation value of the microscopic image is calculated based on the result of the image convolution operation by the image sharpness evaluation function;

[0057] Step S4: The PL unit transmits the sharpness evaluation value to the DDR of the PS unit and sends an interrupt notification to the PS unit to prompt that the PL unit has obtained the sharpness evaluation value and the PS unit needs to read it from the DDR; wherein, the PL unit preferably sends the sharpness evaluation value to the AXI_lite module suitable for transmitting a small amount of data and transmits it to the DDR of the PS unit through the AXI bus;

[0058] Step S5: When the automatic focusing function is enabled, upon receiving the interruption notification, the PS unit reads the clarity evaluation value from the DDR and enters the automatic focusing process to perform automatic focusing based on the clarity evaluation value, generating the focusing control instruction, such that the motor module can drive the microscope stage to move to the focal plane according to the focusing control instruction.

[0059] Therefore, the present invention adopts a data processor with a ZYNQ heterogeneous multi-core architecture and realizes automatic focusing through steps S1 to S5. Based on the parallel processing architecture of the PL unit, i.e., FPGA, multiple-channel image convolution operations can be executed in parallel in step S3, without affecting the synchronous execution of the line-by-line scanning process of the microscopic image in step S1. Such that while the microscopic image scanning in step S1 is completed, the vast majority of the calculations of the image clarity evaluation function (i.e., the image convolution operation) in step S3 have been completed. When the microscopic image scanning is completed, the clarity evaluation value of the microscopic image can be calculated within an extremely short time (generally 250 ns); thus, the present invention replaces the serial execution architecture of an industrial camera + host computer in the prior art with a camera + ZYNQ heterogeneous multi-core architecture, and only requires the scanning time of the microscopic image + an extremely short time (250 ns) to obtain the clarity evaluation value required for automatic focusing. Compared with the time required in the prior art (the scanning time of the microscopic image + the time for image transmission to the host computer, about 14 - 38 ms + the time for memory reading, about 1 - 5 ms + the execution time of the image clarity evaluation function, about 10 - 15 ms), the present invention greatly shortens the time from the start of scanning to obtaining the clarity evaluation value, and can effectively improve the focusing speed of the microscope's automatic focusing.

[0060] Compared with the prior art where the host computer performs the operations, if the working frequency of the device is 60 frames per second, the host computer will waste 30 frames of time in the intermediate ineffective data transfer process, and the truly effective number of operations is only 30 times. While the present invention can complete the focusing process for each frame, achieving 60 times of focusing processes per second, realizing the most efficient use of time. According to experiments, the present invention can complete automatic focusing in 1 - 2 s for short distances, 2 - 6 s for medium and long distances, and can complete automatic focusing in as fast as 0.812 s.

[0061] In addition, the present invention can achieve the image focusing process at full frame rate. There will be a blanking period during the scanning process of the camera. By making full use of the blanking device to control the rotation of the motor, it will not interfere with the image evaluation and focusing process of the next frame.

[0062] The above is the basic implementation manner of the present invention, and further optimization, improvement, and limitation can be made on the basis of this basic implementation manner:

[0063] Preferably, in step S3, the image sharpness evaluation function includes three evaluation functions, namely: a linearity evaluation function with a single peak and good monotonicity, a first sensitivity evaluation function with high sensitivity at the peak point, and a second sensitivity evaluation function; the gray-scale data map output by the Gaussian noise filter is evaluated in parallel by these three evaluation functions to obtain three sharpness evaluation values of the microscopic image;

[0064] See Figure 2 and Figure 3 , in step S5, the autofocus process adopts a multi-operator collaborative evaluation system, including:

[0065] Step S5-1: Obtain three sharpness evaluation values of the microscopic image obtained in the current scan and the microscopic image obtained in the previous scan, and record them as: the evaluation value of the current linearity evaluation function, the evaluation value of the previous linearity evaluation function, the evaluation value of the current first sensitivity evaluation function, the evaluation value of the previous first sensitivity evaluation function, the evaluation value of the current second sensitivity evaluation function, and the evaluation value of the previous second sensitivity evaluation function;

[0066] Step S5-2: Determine whether the following two conditions are simultaneously satisfied: the change rate of the evaluation value of the current first sensitivity evaluation function relative to the evaluation value of the previous first sensitivity evaluation function is above a preset change rate threshold; and, the change rate of the evaluation value of the current second sensitivity evaluation function relative to the evaluation value of the previous second sensitivity evaluation function is above a preset change rate threshold;

[0067] Among them, if the judgment result is no, then: input the evaluation value of the current linearity evaluation function into the hill-climbing search algorithm;

[0068] Among them, if the judgment result is yes, then: input the evaluation value of the current first sensitivity evaluation function and the evaluation value of the current second sensitivity evaluation function into the hill-climbing search algorithm respectively;

[0069] Step S5-3: Generate the focus control instruction according to the judgment result of step S5-2, including: first obtain the movement direction and step distance of the motor module to drive the microscope stage to move through the hill-climbing search algorithm, and then perform a peak search operation to use the peak point determined by the peak search operation as the moment when the motor module stops driving the microscope stage to move;

[0070] Among them, for the case where the judgment result of step S5-2 is yes, when peak points appear respectively when the evaluation value of the current first sensitivity evaluation function and the evaluation value of the current second sensitivity evaluation function are input into the hill-climbing search algorithm, it is determined as the peak point determined by the peak search operation.

[0071] Among them, the existing hill climbing search algorithm turns back the direction once a descent occurs continuously and reduces the step size for searching. The hill climbing search algorithm of the present invention is preferably changed to turn back the direction only after six consecutive descents. Although this will extend the time of partial autofocus, it can effectively reduce the possibility of false detection.

[0072] Therefore, through the judgment in step S5-2 of the present invention, it is made that: when the microscope stage is in the far focus region far from the focal plane, a judgment result of no is obtained, and the evaluation value of the current linearity evaluation function is input into the hill climbing search algorithm and the peak searching operation is performed accordingly, which can avoid the interference points in the far focus region during the peak searching operation, withstand the influence of local extreme points during the peak searching process, and avoid the situation that the clarity evaluation value is less affected by the movement of the microscope stage in the far focus region and is prone to irregular changes, resulting in focusing errors, ensuring that the movement direction of the microscope stage obtained by the hill climbing search algorithm is the correct direction towards the focal plane and avoiding the situation of stopping the evaluation due to an incorrect evaluation direction; when the microscope stage moves to the near focus region close to the focal plane, a judgment result of yes is obtained, and the evaluation value of the current first sensitivity evaluation function and the evaluation value of the current second sensitivity evaluation function are respectively input into the hill climbing search algorithm, which can ensure the focusing accuracy by using the sensitivity evaluation function with high sensitivity; and since two sensitivity evaluation functions are simultaneously used as the input sources of the hill climbing search algorithm, the reliability of judging the boundary line between the far focus region and the near focus region is further improved and the focusing accuracy is further improved; therefore, the present invention has the advantages of high stability, high reliability, high success rate, and high precision in autofocus.

[0073] Through experiments, the short-distance focusing success rate of the present invention is 99.7%, and the long-distance focusing success rate is 98.0%.

[0074] Preferably: the linearity evaluation function is the Variance evaluation function, the first sensitivity evaluation function is the Brenner evaluation function, and the second sensitivity evaluation function is the Tenegard evaluation function.

[0075] Preferably: in step S2, the grayscale data map output by the Gaussian noise filter also passes through a Gaussian blur filter to become a blurred grayscale data map;

[0076] In step S3, the blurred grayscale data map and the grayscale data Figure 1 Similarly, both pass through an image clarity evaluation function including three evaluation functions in parallel for evaluation, and three clarity evaluation values of the microscopic image obtained in the current scan are obtained, which are respectively recorded as the fuzzy evaluation value of the current linearity evaluation function, the fuzzy evaluation value of the current first sensitivity evaluation function, and the fuzzy evaluation value of the current second sensitivity evaluation function;

[0077] While performing the step S5-2, when the dynamic defocus detection function is enabled, the PS unit determines whether the following conditions are simultaneously met. If so, it is determined that the microscopic image obtained in this scan is defocused and blurred, and dynamic defocus correction should be immediately performed to ensure that the microscopic image is clearly visible;

[0078] Conditions: The change rate of the fuzzy evaluation value of the current linearity evaluation function relative to the evaluation value of the current linearity evaluation function is above the preset fuzzy change rate threshold; and, the change rate of the fuzzy evaluation value of the current first sensitivity evaluation function relative to the evaluation value of the current first sensitivity evaluation function is above the preset fuzzy change rate threshold; and, the change rate of the fuzzy evaluation value of the current second sensitivity evaluation function relative to the evaluation value of the current second sensitivity evaluation function is above the preset fuzzy change rate threshold.

[0079] Thus, by passing the grayscale data map through a Gaussian blur filter in step S2 to become a blurred grayscale data map, it is equivalent to putting a mosaic on the picture, making it more blurred. And the more blurred image is evaluated in parallel by the image sharpness evaluation functions of three evaluation functions in step S3. If the original image corresponding to the grayscale data map is itself a blurred image, then the evaluation values of the blurred image corresponding to the blurred grayscale data map and the original image are not much different. But if the original image is a clear image, then the evaluation value of the blurred image will change greatly. Therefore, through the dynamic defocus detection in step S5-2, it can be determined whether the microscopic image obtained in this scan is defocused and blurred, and when the determination is yes, dynamic defocus correction is performed to ensure that the microscopic image is clearly visible.

[0080] Preferably: The adaptive focusing microscope device further includes an objective turret identification module and a high-power LED light source;

[0081] The objective turret identification module determines the magnification of the current microscope objective by installing magnets with different arrangement methods on each objective magnification channel of the microscope objective turret and identifying the arrangement method of the magnets through a Hall element;

[0082] The PS unit obtains the magnification of the current microscope objective in real time by detecting the level state of the Hall element, and controls the high-power LED light source to adjust the power according to this magnification, so that the brightness of the high-power LED light source adapts to the magnification of the current microscope objective, achieving a brighter observation field of view, and taking into account both the focusing accuracy and the focusing duration.

[0083] Preferably: The PS unit also sets the initial step size of the hill climbing search algorithm in step S5-3 according to the magnification of the current microscope objective to obtain the highest focusing accuracy.

[0084] In addition, the present invention can also adopt the following designs and functions:

[0085] The image output by the Gaussian noise filter is transmitted to the DDR of the PS unit through the AXI_DMA module to realize the storage of the video, facilitating the host computer to read the video stream data in the DDR of the PS unit. Alternatively, the PL unit sends an instruction to request the PS unit to transmit the video stream data to the HDMI display screen. After receiving the image data, the HDMI display screen configures the image resolution through the VGA module and then sends the video stream to the external IO interface (here it is the HDMI interface). In addition, the PL unit also encapsulates three motor drive interfaces, receives the drive instructions from the PS unit, and controls the movement of the motor.

[0086] The main tasks of the PS unit (i.e., the ARM side) include camera register configuration, motor drive initialization, construction of the LWIP protocol stack, initialization of the gigabit Ethernet, and reception and drive of serial port instructions of the light source management system. By receiving the interrupt signal from the PL, the PS unit executes specific functions, such as monitoring the motion state of the three-axis motor, obtaining the image sharpness, and judging the focusing state, etc.

[0087] The PS unit also constructs multiple interrupts, including the interrupt from the PL, which is used to obtain the evaluation value, frame buffer number, and motor completion instruction. The timer interrupt is used to execute specific instructions and count the running time. In addition, the PS unit constructs the LWIP protocol stack to transmit the video stream data through the gigabit Ethernet and receive the control instructions from the QT host computer. At the same time, the PS unit also constructs the FAT32 file system to store the microscopic images taken on the SD card for subsequent testing and observation.

[0088] The host computer mainly realizes the display of videos and the issuance of control commands through the QT host computer. It uses Winsocket and the OpenCV library to capture video stream data from the Ethernet and convert it into QT images for display in a window. The entire process covers the reception of video stream data and the issuance of control commands, both of which are efficiently transmitted through the Ethernet. According to different display requirements, the QT host computer sends control commands to the PS unit, and the PS unit changes different DDR addresses of the Ethernet-transmitted data according to the commands, thereby realizing the display of different video data. This flexible control mechanism enables users to switch different video sources as needed and observe microscopic images in real time. In addition to the video display function, the QT host computer also provides a rich set of control options. Users can send commands to achieve one-key automatic focusing, select the focusing direction and magnification, and can also activate the dynamic defocus scanning function, the objective lens conversion function, and the real-time detection of defocus conditions. In addition, the QT host computer can control the xyz three-axis motors to achieve precise position control. In addition, the host computer also has the functions of picture storage and video recording, and can control the SD card to store picture data, etc. In addition to the display of video stream data, the QT host computer can also display information such as evaluation operator values, evaluation directions, evaluation states, motor positions, and real-time frame rates. The display of this information enables users to understand the status and performance of the system in real time, providing convenience for the operation and debugging of the microscope.

[0089] The PS unit encapsulates the protocol functions of multiple motors and can drive multiple motors simultaneously to achieve the three-axis linkage of the stage of the electric microscope. Build the LWIP protocol stack, read the video stream data in the specified memory of the DDR (the size of the memory stream data is 1280*1024), send it out through the UDP network port, and capture it on the PC with WinSocket. Call the qt and opencv libraries to write the UDP to receive the host computer, and the image display and operation can be realized. Control the fixed-plane automatic focusing of the focusing system, take pictures and store them in the TF card, switch the video processing source, and turn on and off the dynamic defocus through the control commands issued by the host computer.

[0090] The host computer mainly realizes the display of videos and the sending of control instructions through the QT host computer. It uses Winsocket and the OpenCV library to capture video stream data from the Ethernet and convert it into QT images for display in a window. The entire process covers the reception of video stream data and the sending of control instructions, both of which are efficiently transmitted through the Ethernet. According to different display requirements, the QT host computer sends control instructions to the PS side, and the PS side changes different DDR addresses of the Ethernet-transmitted data according to the instructions, thereby realizing the display of different video data. This flexible control mechanism enables users to switch different video sources as needed and observe microscopic images in real time. In addition to the video display function, the QT host computer also provides rich control options. Users can achieve one-key automatic focusing, select the focusing direction and magnification by sending instructions, and can also enable functions such as dynamic defocus detection, scanning stitching, and intelligent objective lens conversion to achieve intelligent control. In addition, the QT host computer can also control the xyz three-axis motors to achieve precise position control. In addition, the host computer also has the functions of picture storage and video recording, and can control the SD card to store picture data, etc. In addition to the display of video stream data, the QT host computer can also display information such as evaluation operator values, evaluation directions, evaluation states, motor positions, and real-time frame rates. The display of this information allows users to understand the status and performance of the system in real time, providing convenience for the operation and debugging of the microscope.

[0091] The present invention is not limited to the above specific embodiments. Based on the above content, according to the common general knowledge and customary means in the art, without departing from the above basic technical idea of the present invention, the present invention can also make various other forms of equivalent modifications, substitutions, or changes, all of which fall within the protection scope of the present invention.

Claims

1. An adaptive focusing microscope device, comprising a camera and a motor module; the camera is used to align the microscope platform to scan and obtain a microscopic image, and the motor module drives the microscope platform to move to a focal plane according to a focusing control instruction; Features: Also included are: a data processor that uses a ZYNQ heterogeneous multi-core architecture consisting of a PS unit and a PL unit; The adaptive focusing microscope device performs automatic focusing control according to the following process: Step S1, the camera scans to obtain a microscopic image by line-by-line scanning, and synchronously outputs each line of microscopic image line scanning data obtained by scanning to the PL unit in the form of Bayer array format video stream data; Step S2, each time the PL unit receives a line of microscopic image line scan data, it performs pre-stage image processing on it, including: the Bayer array format video stream data is first cached through the FIFO memory to synchronize with the internal clock domain of the PL unit across clock domains; the synchronized data is converted into RGB888 video stream data through a color interpolation restoration algorithm; the RGB888 video stream data is converted into YCbcr video stream data through a color space conversion algorithm, and the Y component therein is extracted as grayscale data; the grayscale data is filtered out through a Gaussian noise filter to obtain a grayscale data graph; Step S3, the PL unit starts scanning the third line of microscopic image line scanning data from the camera. When each line of microscopic image line scanning data starts to be scanned, the image convolution operation in the image clarity evaluation function is performed on the grayscale data graph obtained by processing the first two lines of microscopic image line scanning data in step S2, until the last line of microscopic image line scanning data of the microscopic image is scanned and processed in step S2, and the image clarity evaluation function is used to calculate the clarity evaluation value of the microscopic image based on the result of the image convolution operation; Step S4, the PL unit transmits the clarity evaluation value to the DDR of the PS unit and sends an interrupt notification to the PS unit; Step S5: When the automatic focusing function is turned on, the PS unit reads the clarity evaluation value from the DDR upon receiving the interrupt notification, and enters the automatic focusing process to perform automatic focusing processing based on the clarity evaluation value and generate the focus control instruction.

2. The adaptive focusing microscope device according to claim 1, characterized in that: In step S3, the image clarity evaluation function includes three evaluation functions, namely, a linearity evaluation function, a first sensitivity evaluation function and a second sensitivity evaluation function; the grayscale data graph output by the Gaussian noise filter is evaluated in parallel by the three evaluation functions to obtain three clarity evaluation values ​​of the microscopic image; In step S5, the automatic focusing process includes: Step S5-1, obtaining three clarity evaluation values ​​of the microscopic image obtained by this scan and the microscopic image obtained by the previous scan, which are recorded as: the current linearity evaluation function evaluation value, the previous linearity evaluation function evaluation value, the current first sensitivity evaluation function evaluation value, the previous first sensitivity evaluation function evaluation value, the current second sensitivity evaluation function evaluation value, the previous second sensitivity evaluation function evaluation value; Step S5-2, judging whether the following two conditions are met at the same time: the change rate of the evaluation value of the first sensitivity evaluation function this time relative to the evaluation value of the first sensitivity evaluation function last time is above the preset change rate threshold; and the change rate of the evaluation value of the second sensitivity evaluation function this time relative to the evaluation value of the second sensitivity evaluation function last time is above the preset change rate threshold; If the judgment result is no, then: the evaluation value of the linearity evaluation function is input into the hill climbing search algorithm; If the judgment result is yes, then: the evaluation value of the first sensitivity evaluation function and the evaluation value of the second sensitivity evaluation function are respectively input into the hill climbing search algorithm; Step S5-3, generating the focus control instruction according to the judgment result of step S5-2, comprising: first obtaining the movement direction and step distance of the motor module driving the microscope loading platform to move by a hill climbing search algorithm, and then performing a peak search operation, so as to use the peak point determined by the peak search operation as the moment when the motor module stops driving the microscope loading platform to move; Among them, when the judgment result of step S5-2 is yes, when the first sensitivity evaluation function evaluation value and the second sensitivity evaluation function evaluation value are respectively input into the hill climbing search algorithm and peak points appear, it is determined to be the peak point determined by the peak finding operation.

3. The adaptive focusing microscope device according to claim 2, characterized in that: The linearity evaluation function is a Variance evaluation function, the first sensitivity evaluation function is a Brenner evaluation function, and the second sensitivity evaluation function is a Tenegard evaluation function.

4. The adaptive focusing microscope device according to claim 2 or 3, characterized in that: In step S2, the grayscale data image output by the Gaussian noise filter is further passed through a Gaussian blur filter to become a blurred grayscale data image; In step S3, the fuzzy grayscale data graph is the same as the grayscale data graph, and is evaluated in parallel by the image clarity evaluation function including three evaluation functions, so as to obtain three clarity evaluation values ​​of the microscopic image obtained by this scanning, which are respectively recorded as the fuzzy evaluation value of this linearity evaluation function, the fuzzy evaluation value of this first sensitivity evaluation function, and the fuzzy evaluation value of this second sensitivity evaluation function; While executing step S5-2, when the dynamic defocus detection function is turned on, the PS unit determines whether the following conditions are met at the same time. If so, it is determined that the microscopic image obtained by this scan is out of focus and blur, and dynamic defocus correction should be performed immediately; Condition: The change rate of the fuzzy evaluation value of the linearity evaluation function relative to the evaluation value of the linearity evaluation function is above the preset fuzzy change rate threshold; and the change rate of the fuzzy evaluation value of the first sensitivity evaluation function relative to the evaluation value of the first sensitivity evaluation function is above the preset fuzzy change rate threshold; And, a change rate of the current second sensitivity evaluation function fuzzy evaluation value relative to the current second sensitivity evaluation function evaluation value is above a preset fuzzy change rate threshold.

5. The adaptive focusing microscope device according to claim 2 or 3, characterized in that: The adaptive focusing microscope device also includes an objective lens turret identification module and a high-power LED light source; The objective lens turret recognition module determines the magnification of the current microscope objective lens by installing magnets with different arrangements on each objective lens magnification channel of the microscope objective lens turret and identifying the arrangement of the magnets through the Hall element; The PS unit acquires the magnification of the current microscope objective lens in real time by detecting the level state of the Hall element, and controls the high-power LED light source to adjust the power according to the magnification so that the brightness of the high-power LED light source adapts to the magnification of the current microscope objective lens.

6. The adaptive focusing microscope device according to claim 5, characterized in that: The PS unit also sets the initial step size of the hill climbing search algorithm in step S5-3 according to the magnification of the current microscope objective lens.

Images