An adaptive focus microscope apparatus

Abstract

The application discloses a kind of self-adapting focusing microscope equipment, adopt the data processor of ZYNQ heterogeneous multi-core architecture, and realize automatic focusing by step S1 to step S5, based on the parallel processing architecture of PL unit, namely FPGA, image convolution operation of multiple images can be executed in parallel in step S3, will not affect the line-by-line scanning process of microimage in step S1 is executed synchronously, so that while completing microimage scanning in step S1, step S3 has completed the most part of calculation of image definition evaluation function, when microimage scanning is completed, image definition evaluation function can calculate the definition evaluation value of microimage in very short time;Therefore, the application uses camera+ZYNQ heterogeneous multi-core architecture to replace the serial execution architecture of industrial camera+host computer in the prior art, only needs the scanning time of microimage+very short time to obtain definition evaluation value, greatly shorten the time of obtaining definition evaluation value, improve focusing speed.

Description

TECHNICAL FIELD

[0001] The present application relates to a microscope device, in particular to an adaptive focusing microscope device. BACKGROUND

[0002] The microscope device structure commonly used in the market at present is composed of a microscope body, a USB camera, a PC end, a monitor, a chip on a slice, a motor driver, and a transmission mechanical structure. The microscope body includes a light source system, but generally has no intelligent control function, and the light source brightness is manually adjusted by a person. An industrial camera is generally arranged at the top end of the microscope, and after the industrial camera scans the image, the image is transmitted to the PC end through a USB or Ethernet interface. The PC end generally uses QT host computer or Labview to realize a visual interface and control instruction issuing. At the same time, an embedded hardware operating system is externally built, and the control instruction is issued to the embedded chip through a serial port and the like, the chip controls the motor driver to drive the motor, and the up and down movement of the object platform is realized. There is a transmission structure between the motor and the object platform, and the transmission structure drives the object platform to move up and down stably. The microscope body is also generally provided with a manual adjustment button, and the up and down movement or XY axis plane movement of the object platform can be controlled by a manual knob.

[0003] The focusing speed of the similar microscope device product at present is not fast enough, especially in the environment of multiple sample detection, the focusing process of multiple samples accumulated will occupy a lot of time, so the single sample focusing which can ensure stability and speed is the main development direction in the future. Especially in some application scenarios of widening the microscopic field of view, a large number of microscopic pictures need to be taken, and in this process, it is inevitable to cause defocusing due to the change of sample height. In order to avoid this situation, it is necessary to quickly judge the defocusing of the current picture and execute the automatic focusing process. For example, a 20*20 array needs to scan 400 pictures, if the automatic focusing process is performed on all the 400 pictures, then the scanning process can save 400s of scanning time if the automatic focusing speed is improved by 1s.

[0004] The main reason for the bottleneck of the focusing rate of the same kind of microscope equipment product at present is that the PC end is used for image algorithm evaluation. The traditional automatic focusing system workflow on the market is mentioned in the foregoing: step one, a microscope image on the current objective platform is obtained by scanning through an industrial camera; step two, the microscope image data is transmitted to the memory of the PC end through a USB or Ethernet interface; step three, the CPU of the PC end reads the microscope image data in the memory; step four, the PC end performs a corresponding image sharpness evaluation algorithm on the microscope image data to obtain an image sharpness evaluation value of the microscope image; step five, motor motion control is performed according to the comparison result of the image sharpness evaluation value and the image sharpness evaluation value of the previous microscope image, so that the objective platform moves to a direction in which the image sharpness evaluation value is better; and step six, the foregoing steps are repeated until the focal plane is tracked. The existing automatic focusing system workflow has the following problems: the image carrying, image storage, image reading and image algorithm execution are completely executed in a serial execution architecture, and time is not fully utilized. Especially in the entire focusing process of the microscope, we often need more than one hundred evaluation processes to finally reach the vicinity of the most accurate focal plane, and the evaluation processes are stacked for hundreds of times, which will cause more than half of the time to be spent in the repeated image carrying process, resulting in a slow focusing rate. Since the execution architecture of the CPU itself is serial execution, there is no other way to significantly improve the speed except to increase the working frequency of the CPU itself to improve the operation speed. However, since it is a systematic product, the performance of the main frequency of each computer is not consistent, and the impact on the speed is very large, so a new architecture needs to be adopted to abandon the original workflow to realize high-speed and high-precision automatic focusing, otherwise it is impossible to simultaneously realize high speed and low cost from the cost point of view.

[0005] In addition, the market needs a microscope equipment product with a fully automatic process. Many automatic focusing microscope equipment products need to click focus on the host computer to complete the focusing process, and different objective magnifications generally correspond to different focusing strategies, so repeated selection and clicking are more troublesome. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a self-adaptive focusing microscope equipment.

[0007] To solve the above technical problems, the technical solution adopted by the present application is as follows:

[0008] A self-adaptive focusing microscope equipment comprises a camera and a motor module; the camera is used to scan the objective platform of the microscope to obtain a microscope image, and the motor module drives the objective platform of the microscope to move to the focal plane according to a focusing control instruction, so as to obtain a clear observation effect of the object on the objective platform of the microscope.

[0009] characterized in that:

[0010] Further comprising: a data processor which adopts a ZYNQ heterogeneous multi-core architecture composed of a PS unit and a PL unit, preferably adopts a Zynq-7020 development board; wherein the PS unit is a common ARM unit (suitable for processing floating-point number algorithm, belonging to serial architecture), and the PL unit is an FPGA system-on-chip (suitable for processing high-speed parallel data stream);

[0011] The adaptive focusing microscope device performs automatic focusing control according to the following process:

[0012] Step S1, the camera obtains a microscopic image by scanning in a line-by-line manner, and synchronously outputs each line of the scanned microscopic image line scanning data to the PL unit in the form of Bayer array format video stream data; wherein the camera is preferably OV5640, and the clock frequency of the Bayer array format video stream data is preferably 72M, and the resolution is preferably 720P.

[0013] Step S2, the PL unit performs pre-stage image processing on each line of the microscopic image line scanning data as soon as it is received, including: the Bayer array format video stream data is first buffered in the FIFO memory to synchronize with the internal clock domain of the PL unit; the synchronized data is converted into RGB888 video stream data by a color interpolation restoration algorithm to restore the true color; the RGB888 video stream data is converted into YCbcr video stream data by a color space conversion algorithm, and the Y component is extracted as grayscale data to reduce the computational complexity of the subsequent evaluation function algorithm; the grayscale data is filtered by a Gaussian noise filter to obtain pure grayscale data graph;

[0014] Step S3, the PL unit starts scanning the third line of the microscopic image line scanning data from the camera, and when each line of the microscopic image line scanning data starts scanning, the image convolution operation in the image sharpness evaluation function is performed on the grayscale data graphs obtained by processing the previous two lines of the microscopic image line scanning data by step S2, until the last line of the microscopic image line scanning data is scanned and processed by step S2, and then the sharpness evaluation value of the microscopic image is calculated based on the results of the image convolution operation in the image sharpness evaluation function;

[0015] Step S4, the PL unit transmits the sharpness evaluation value to the DDR of the PS unit and sends an interrupt notification to the PS unit to prompt that the PL unit has obtained the sharpness evaluation value, and the PS unit needs to read from the DDR; wherein the PL unit preferably sends the sharpness evaluation value to the AXI_lite module suitable for transmitting a small amount of data, and transmits it to the DDR of the PS unit through the AXI bus;

[0016] Step S5, in the case of starting the auto-focusing function, the PS unit reads the definition evaluation value from the DDR when receiving the interrupt notification, and enters the auto-focusing flow to perform auto-focusing processing based on the definition evaluation value, to generate the focus control instruction, so that the motor module can drive the microscope stage to move to the focal plane according to the focus control instruction.

[0017] Therefore, the application adopts the data processor of ZYNQ heterogeneous multi-core architecture, and realizes auto-focusing through steps S1 to S5, based on the parallel processing architecture of the PL unit, i.e., FPGA, the image convolution operation of multiple images can be performed in parallel in step S3, which does not affect the synchronous execution of the line-by-line scanning flow of the microscopic image in step S1, so that while the microscopic image scanning in step S1 is completed, most of the calculation of the image definition evaluation function (i.e., image convolution operation) in step S3 is completed, and when the microscopic image scanning is completed, the definition evaluation value of the microscopic image can be calculated in a very short time (generally 250 ns) by the image definition evaluation function; therefore, the application replaces the serial execution architecture of the industrial camera + host computer in the prior art with the camera + ZYNQ heterogeneous multi-core architecture, and only the scanning time of the microscopic image + a very short time (250 ns) is needed to obtain the definition evaluation value required for auto-focusing, compared with the time required by the prior art (the scanning time of the microscopic image + the time for transmitting the image to the host computer, about 14-38 ms + the time for reading the memory, about 1-5 ms + the execution time of the image definition evaluation function, about 10-15 ms), the application greatly shortens the time from starting scanning to obtaining the definition evaluation value, and can effectively improve the focusing speed of the microscope auto-focusing.

[0018] Compared with the execution by the host computer in the prior art, if the working frequency of the device is 60 frames, the host computer will waste 30 frames of time in the intermediate invalid data transfer process, and the real effective working time is only 30 times, while the application can perform the focusing process every frame, complete 60 times of focusing process per second, and realize the highest efficiency of time utilization. According to experiments, the application can complete auto-focusing in 1-2 seconds at a short distance, and can complete auto-focusing in 2-6 seconds at a medium or long distance, and can complete auto-focusing in only 0.812 seconds at the fastest.

[0019] In addition, the application can realize image focusing process at full frame rate, and there will be a period of eclipse during the scanning process of the camera, and the motor is controlled to rotate by fully utilizing the eclipse device, without interfering with the image evaluation focusing process of the next frame.

[0020] Preferably, in the step S3, the image sharpness evaluation function comprises three evaluation functions, i.e., a linearity evaluation function with a single peak and good monotonicity, a first sensitivity evaluation function with high sensitivity at the peak point, and a second sensitivity evaluation function; the gray data image output by the Gaussian noise filter is evaluated in parallel by the three evaluation functions to obtain three sharpness evaluation values of the microscopic image.

[0021] Referring to Figure 2 and Figure 3 , in the step S5, the automatic focusing process adopts a multi-operator collaborative evaluation system, which comprises:

[0022] In step S5-1, three sharpness evaluation values of the microscopic image obtained in this scan and the microscopic image obtained in the last scan are obtained, which are denoted as the linear evaluation function evaluation value in this scan, the linear evaluation function evaluation value in the last scan, the first sensitivity evaluation function evaluation value in this scan, the first sensitivity evaluation function evaluation value in the last scan, the second sensitivity evaluation function evaluation value in this scan, and the second sensitivity evaluation function evaluation value in the last scan, respectively.

[0023] In step S5-2, it is determined whether the following two conditions are met simultaneously: the change rate of the first sensitivity evaluation function evaluation value in this scan relative to the first sensitivity evaluation function evaluation value in the last scan is above a preset change rate threshold; and the change rate of the second sensitivity evaluation function evaluation value in this scan relative to the second sensitivity evaluation function evaluation value in the last scan is above a preset change rate threshold.

[0024] If the determination result is no, the linear evaluation function evaluation value in this scan is input into the hill climbing search algorithm.

[0025] If the determination result is yes, the first sensitivity evaluation function evaluation value in this scan and the second sensitivity evaluation function evaluation value in this scan are input into the hill climbing search algorithm, respectively.

[0026] In step S5-3, the focusing control instruction is generated according to the determination result of step S5-2, which comprises: first obtaining the movement direction and step distance of the motor module driving the movement of the microscope stage by the hill climbing search algorithm, and then performing peak searching operation, so that the peak value point determined by the peak searching operation is used as the time when the motor module stops driving the movement of the microscope stage.

[0027] For the case where the determination result of step S5-2 is yes, when the first sensitivity evaluation function evaluation value in this scan and the second sensitivity evaluation function evaluation value in this scan are input into the hill climbing search algorithm and peak value points are obtained, respectively, the peak value point determined by the peak searching operation is determined.

[0028] Among them, the existing mountain climbing search algorithm is to turn back the direction after one continuous decline, and to search by reducing the step length, and the mountain climbing search algorithm of the application is preferably changed to turn back the direction after six continuous declines, which although will prolong the time of automatic focusing, but can effectively improve the possibility of false detection.

[0029] Therefore, by the judgment of step S5-2, when the microscope object platform is in the far focus area far from the focal plane, the judgment result is no, the linear evaluation function evaluation value of this time is input into the mountain climbing search algorithm and used for peak searching operation, the influence of local extreme points in the peak searching process can be avoided, the clearness evaluation value in the far focus area is less affected by the movement of the microscope object platform and is easy to change irregularly, the focusing failure can be avoided, the movement direction of the microscope object platform obtained by the mountain climbing search algorithm is ensured to be the correct direction of moving to the focal plane, and the situation of stopping evaluation due to the evaluation direction error is avoided; when the microscope object platform moves to the near focus area close to the focal plane, the judgment result is yes, the first sensitivity evaluation function evaluation value and the second sensitivity evaluation function evaluation value of this time are input into the mountain climbing search algorithm, the focusing accuracy can be ensured by using the sensitivity evaluation function with high sensitivity; and since two sensitivity evaluation functions are used as the input sources of the mountain climbing search algorithm, the reliability of the demarcation line between the far focus area and the near focus area is further improved, and the focusing accuracy is further improved; therefore, the application has the advantages of high stability, high reliability, high success rate and high accuracy in automatic focusing.

[0030] Through experiments, the short distance focusing success rate of the application is 99.7%, and the long distance focusing success rate is 98.0%.

[0031] Preferably, the linearity evaluation function is a Variance evaluation function, the first sensitivity evaluation function is a Brenner evaluation function, and the second sensitivity evaluation function is a Tenegard evaluation function.

[0032] Preferably, in step S2, the gray scale data graph output by the Gaussian noise filter is further subjected to a Gaussian blur filter to become a blurred gray scale data graph.

[0033] In step S3, the blurred gray scale data graph and the gray scale data Figure 1 The blurred gray scale data graph and the gray scale data

[0034] While executing the step S5-2, in the case that the dynamic defocus detection function is turned on, the PS unit judges whether the following conditions are met simultaneously, if yes, it is determined that the microscopic image obtained by this time scanning is out of focus, and dynamic defocus correction should be performed immediately to ensure that the microscopic image is clear and visible;

[0035] The conditions are: the change rate of the blurring evaluation value of the linear evaluation function this time relative to the evaluation value of the linear evaluation function this time is above the preset blurring change rate threshold; and the change rate of the blurring evaluation value of the first sensitivity evaluation function this time relative to the evaluation value of the first sensitivity evaluation function this time is above the preset blurring change rate threshold; and the change rate of the blurring evaluation value of the second sensitivity evaluation function this time relative to the evaluation value of the second sensitivity evaluation function this time is above the preset blurring change rate threshold.

[0036] Therefore, by passing the gray data graph through a Gaussian blur filter to become a blurred gray data graph in step S2, the image is equivalent to being covered with a mosaic and will be more blurred. The more blurred image is evaluated by the image clarity evaluation function of the three evaluation functions in parallel in step S3. If the original image corresponding to the gray data graph is a blurred image itself, the evaluation value of the blurred image corresponding to the blurred gray data graph will not differ much from that of the original image. However, if the original image is a clear image itself, the evaluation value of the blurred image will change greatly. Thus, by the dynamic defocus detection of step S5-2, it can be determined whether the microscopic image obtained by this time scanning is out of focus, and dynamic defocus correction is performed when it is determined that it is, to ensure that the microscopic image is clear and visible.

[0037] Preferably, the adaptive focusing microscope device further comprises an objective turret identification module and a high-power LED light source.

[0038] The objective turret identification module judges the magnification of the current microscope objective by installing magnets in different arrangements on each objective magnification channel of the microscope objective turret and identifying the arrangement of the magnets by a Hall element.

[0039] The PS unit acquires the magnification of the current microscope objective in real time by detecting the level state of the Hall element, and controls the high-power LED light source to adjust the power according to the magnification, so that the brightness of the high-power LED light source is adapted to the magnification of the current microscope objective, a brighter observation field of view is realized, and both focusing accuracy and focusing time are taken into account.

[0040] Preferably, the PS unit further sets the initial step size of the hill climbing search algorithm in step S5-3 according to the magnification of the current microscope objective, to obtain the highest focusing accuracy.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] Firstly, the application adopts the data processor of ZYNQ heterogeneous multi-core architecture, and realizes the automatic focusing through steps S1 to S5. Based on the parallel processing architecture of the PL unit, i.e., FPGA, the image convolution operation of multiple images can be performed in step S3, which does not affect the line-by-line scanning process of the microscopic image in step S1, so that while the microscopic image scanning is completed in step S1, most of the calculation of the image definition evaluation function (i.e., image convolution operation) is completed in step S3. When the microscopic image scanning is completed, the definition evaluation value of the microscopic image can be calculated in a very short time (generally 250 ns) by the image definition evaluation function. Therefore, the camera + ZYNQ heterogeneous multi-core architecture is used to replace the serial execution architecture of the industrial camera + host computer in the prior art, and the definition evaluation value required for automatic focusing can be obtained only by the scanning time of the microscopic image + a very short time (250 ns). Compared with the time required by the prior art (the scanning time of the microscopic image + the time for transmitting the image to the host computer, about 14-38 ms + the time for reading the memory, about 1-5 ms + the execution time of the image definition evaluation function, about 10-15 ms), the application greatly shortens the time from starting scanning to obtaining the definition evaluation value, and effectively improves the focusing speed of the microscope automatic focusing.

[0043] Secondly, through the judgment of step S5-2, when the microscope object platform is in the far focus area far from the focal plane, the judgment result is no, the linear evaluation function value of this time is input into the hill climbing search algorithm for peak searching operation, which can avoid the interference points in the far focus area in the peak searching operation, resist the influence of local extreme points in the peak searching process, avoid the irregular change of the definition evaluation value in the far focus area due to the small influence of the movement of the microscope object platform, and prevent the focusing failure caused by the irregular change. The movement direction of the microscope object platform obtained by the hill climbing search algorithm is the correct direction of moving to the focal plane, which avoids the situation that the evaluation direction is wrong and the evaluation is stopped. When the microscope object platform moves to the near focus area close to the focal plane, the judgment result is yes, and the first sensitivity evaluation function value and the second sensitivity evaluation function value of this time are input into the hill climbing search algorithm, which can ensure the focusing accuracy by using the sensitivity evaluation function with high sensitivity. Since two sensitivity evaluation functions are used as the input source of the hill climbing search algorithm, the reliability of the demarcation line between the far focus area and the near focus area is further improved, and the focusing accuracy is further improved. Therefore, the application has the advantages of high stability, high reliability, high success rate and high accuracy in automatic focusing.

[0044] Thirdly, the present application is equivalent to that the picture is covered with mosaic and is more blurred by passing the gray data graph through a Gaussian blur filter to become a blurred gray data graph in step S2, and the more blurred image is evaluated by the image definition evaluation function of the three evaluation functions in parallel in step S3, if the original image corresponding to the gray data graph is a blurred image, the evaluation value of the blurred image corresponding to the blurred gray data graph and the original image has little difference, but if the original image is a clear image, the evaluation value of the blurred image will have a greater change, thus, through the dynamic defocus detection of step S5-2, it can be judged whether the microscopic image obtained by this time scanning appears defocus blur phenomenon, and dynamic defocus correction is performed when it is judged to be, to ensure that the microscopic image is clear and visible. BRIEF DESCRIPTION OF DRAWINGS

[0045] The present application will be further described below in combination with the drawings and specific embodiments:

[0046] Figure 1 The circuit principle block diagram of the adaptive focusing microscope device of the present application is shown in the figure;

[0047] Figure 2 The data flow chart of the data processor and the host computer in the present application is shown in the figure;

[0048] Figure 3 The flow chart of the multi-operator collaborative evaluation system in the present application is shown in the figure. DETAILED DESCRIPTION

[0049] The present application will be further described below in combination with the drawings and specific embodiments:

[0050] Example 1

[0051] As shown in Figure 1 and Figure 2 , the present application discloses an adaptive focusing microscope device, which comprises a camera and a motor module; the camera is used for scanning the microscope object platform to obtain a microscopic image, and the motor module drives the microscope object platform to move to the focal plane according to the focusing control instruction, so as to obtain a clear observation effect of the object on the microscope object platform;

[0052] Further comprising: a data processor which adopts a ZYNQ heterogeneous multi-core architecture composed of a PS unit and a PL unit, preferably adopts a Zynq-7020 development board; wherein the PS unit is a common ARM unit (suitable for processing floating-point number algorithm, belonging to serial architecture), and the PL unit is an FPGA system-on-chip (suitable for processing high-speed parallel data stream);

[0053] The adaptive focusing microscope device performs automatic focusing control according to the following process:

[0054] Step S1, the camera obtains a microscopic image by line-by-line scanning, and synchronously outputs each line of the scanned microscopic image line scanning data to the PL unit in the form of Bayer array format video stream data; wherein the camera is preferably OV5640, the clock frequency of the Bayer array format video stream data is preferably 72M, and the resolution is preferably 720P.

[0055] Step S2, the PL unit performs pre-stage image processing on each line of the received microscopic image line scanning data, including: the Bayer array format video stream data is first cached by a FIFO memory to synchronize with the internal clock domain of the PL unit; the synchronized data is converted into RGB888 video stream data by a color interpolation restoration algorithm to restore the true color; the RGB888 video stream data is converted into YCbcr video stream data by a color space conversion algorithm, and the Y component is extracted as grayscale data to reduce the computational complexity of the subsequent evaluation function algorithm; the grayscale data is filtered by a Gaussian noise filter to obtain pure grayscale data;

[0056] Step S3, the PL unit starts scanning the third line of the microscopic image line scanning data from the camera, and when each line of the microscopic image line scanning data starts scanning, the image convolution operation in the image sharpness evaluation function is performed on the grayscale data graphs obtained by processing the previous two lines of the microscopic image line scanning data by step S2, until the last line of the microscopic image line scanning data is scanned and processed by step S2, and then the sharpness evaluation value of the microscopic image is calculated based on the results of the image convolution operation by the image sharpness evaluation function;

[0057] Step S4, the PL unit transmits the sharpness evaluation value to the DDR of the PS unit and sends an interrupt notification to the PS unit to prompt that the PL unit has obtained the sharpness evaluation value, and the PS unit needs to read from the DDR; wherein the PL unit preferably sends the sharpness evaluation value to the AXI_lite module suitable for transmitting a small amount of data, and transmits it to the DDR of the PS unit through the AXI bus;

[0058] Step S5, in the case of starting the auto-focusing function, the PS unit reads the definition evaluation value from the DDR when receiving the interrupt notification, and enters the auto-focusing flow to perform auto-focusing processing based on the definition evaluation value, to generate the focus control instruction, so that the motor module can drive the microscope stage to move to the focal plane according to the focus control instruction.

[0059] Therefore, the application adopts the data processor of ZYNQ heterogeneous multi-core architecture, and realizes auto-focusing through steps S1 to S5, based on the parallel processing architecture of the PL unit, i.e., FPGA, the image convolution operation of multiple images can be performed in parallel in step S3, without affecting the synchronous execution of the line-by-line scanning flow of the microscopic image in step S1, so that while the scanning of the microscopic image is completed in step S1, most of the calculation of the image definition evaluation function (i.e., image convolution operation) is completed in step S3, and when the scanning of the microscopic image is completed, the definition evaluation value of the microscopic image can be calculated by the image definition evaluation function in a very short time (generally 250 ns); therefore, the application replaces the serial execution architecture of the industrial camera + host computer in the prior art with a camera + ZYNQ heterogeneous multi-core architecture, and the definition evaluation value required for auto-focusing can be obtained only in the scanning time of the microscopic image + a very short time (250 ns), compared with the time required in the prior art (the scanning time of the microscopic image + the time for transmitting the image to the host computer, about 14-38 ms + the time for reading the memory, about 1-5 ms + the execution time of the image definition evaluation function, about 10-15 ms), the application greatly shortens the time from starting scanning to obtaining the definition evaluation value, and can effectively improve the focusing speed of the microscope auto-focusing.

[0060] Compared with the execution by the host computer in the prior art, if the working frequency of the device is 60 frames, the host computer will waste 30 frames of time in the intermediate invalid data transfer flow, and the real effective working time is only 30 times, while the application can perform the focusing flow every frame, complete the focusing flow 60 times per second, and realize the highest efficiency of time utilization. According to experiments, the application can complete auto-focusing in 1-2 seconds at a short distance, and can complete auto-focusing in 2-6 seconds at a medium or long distance, and can complete auto-focusing in only 0.812 seconds at the fastest.

[0061] In addition, the application can realize the image focusing flow at full frame rate, and there will be a period of eclipse during the scanning process of the camera, and the motor can be controlled to rotate by fully utilizing the eclipse device, without interfering with the image evaluation focusing flow of the next frame.

[0062] The above is the basic embodiment of the application, and further optimization, improvement and limitation can be made on the basis of the basic embodiment.

[0063] Preferably, in the step S3, the image sharpness evaluation function comprises three evaluation functions, i.e., a linearity evaluation function with a single peak and good monotonicity, a first sensitivity evaluation function with high sensitivity at the peak point, and a second sensitivity evaluation function; the gray data image output by the Gaussian noise filter is evaluated in parallel by the three evaluation functions to obtain three sharpness evaluation values of the microscopic image.

[0064] Referring to Figure 2 and Figure 3 , in the step S5, the automatic focusing process adopts a multi-operator collaborative evaluation system, which comprises:

[0065] In step S5-1, three sharpness evaluation values of the microscopic image obtained in this scan and the microscopic image obtained in the last scan are obtained, which are denoted as the linear evaluation function evaluation value in this scan, the linear evaluation function evaluation value in the last scan, the first sensitivity evaluation function evaluation value in this scan, the first sensitivity evaluation function evaluation value in the last scan, the second sensitivity evaluation function evaluation value in this scan, and the second sensitivity evaluation function evaluation value in the last scan, respectively.

[0066] In step S5-2, it is determined whether the following two conditions are met simultaneously: the change rate of the first sensitivity evaluation function evaluation value in this scan relative to the first sensitivity evaluation function evaluation value in the last scan is above a preset change rate threshold; and the change rate of the second sensitivity evaluation function evaluation value in this scan relative to the second sensitivity evaluation function evaluation value in the last scan is above a preset change rate threshold.

[0067] If the determination result is no, the linear evaluation function evaluation value in this scan is input into the hill climbing search algorithm.

[0068] If the determination result is yes, the first sensitivity evaluation function evaluation value in this scan and the second sensitivity evaluation function evaluation value in this scan are input into the hill climbing search algorithm, respectively.

[0069] In step S5-3, the focusing control instruction is generated according to the determination result of step S5-2, which comprises: first obtaining the movement direction and step distance of the motor module driving the movement of the microscope stage by the hill climbing search algorithm, and then performing peak searching operation, so that the peak value point determined by the peak searching operation is used as the time when the motor module stops driving the movement of the microscope stage.

[0070] For the case where the determination result of step S5-2 is yes, when the first sensitivity evaluation function evaluation value in this scan and the second sensitivity evaluation function evaluation value in this scan are input into the hill climbing search algorithm and peak value points are obtained, respectively, the peak value point determined by the peak searching operation is determined.

[0071] Among them, the existing hill climbing search algorithm is to turn back the direction after one continuous decline, and to search by reducing the step length, and the hill climbing search algorithm of the application is preferably changed to turn back the direction after six continuous declines, which although will prolong the time of automatic focusing, but can effectively improve the possibility of false detection.

[0072] Therefore, by the judgment of step S5-2, when the microscope stage is in the far focus area far from the focal plane, the judgment result is no, the linear evaluation function evaluation value of this time is input into the hill climbing search algorithm and used for peak searching operation, the influence of local extreme points in the peak searching process can be avoided, the clearness evaluation value in the far focus area is less affected by the movement of the microscope stage and is easy to change irregularly, the focusing failure can be avoided, the movement direction of the microscope stage obtained by the hill climbing search algorithm is ensured to be the correct direction of moving to the focal plane, and the situation of stopping evaluation due to the evaluation direction error is avoided; when the microscope stage moves to the near focus area close to the focal plane, the judgment result is yes, the first sensitivity evaluation function evaluation value and the second sensitivity evaluation function evaluation value of this time are input into the hill climbing search algorithm, the focusing accuracy can be ensured by using the sensitivity evaluation function with high sensitivity; and since two sensitivity evaluation functions are used as the input sources of the hill climbing search algorithm, the reliability of the demarcation line between the far focus area and the near focus area is further improved, and the focusing accuracy is further improved; therefore, the application has the advantages of high stability, high reliability, high success rate and high accuracy in automatic focusing.

[0073] Through experiments, the short distance focusing success rate of the application is 99.7%, and the long distance focusing success rate is 98.0%.

[0074] Preferably, the linearity evaluation function is a Variance evaluation function, the first sensitivity evaluation function is a Brenner evaluation function, and the second sensitivity evaluation function is a Tenegard evaluation function.

[0075] Preferably, in step S2, the gray scale data graph output by the Gaussian noise filter is further subjected to a Gaussian blur filter to become a blurred gray scale data graph.

[0076] In step S3, the blurred gray scale data graph and the gray scale data Figure 1 The blurred gray scale data graph and the gray scale data

[0077] In the step S5-2, when the dynamic defocus detection function is turned on, the PS unit determines whether the following conditions are met simultaneously, if yes, it is determined that the microscopic image obtained by the current scanning is out of focus, and the dynamic defocus correction should be performed immediately to ensure that the microscopic image is clear and visible.

[0078] Conditions: the change rate of the current linearity evaluation function blur evaluation value relative to the current linearity evaluation function evaluation value is above the preset blur change rate threshold; and the change rate of the current first sensitivity evaluation function blur evaluation value relative to the current first sensitivity evaluation function evaluation value is above the preset blur change rate threshold; and the change rate of the current second sensitivity evaluation function blur evaluation value relative to the current second sensitivity evaluation function evaluation value is above the preset blur change rate threshold.

[0079] Therefore, by passing the grayscale data graph through the Gaussian blur filter to become a blurred grayscale data graph in step S2, the image is equivalent to being covered with a mosaic, which will be more blurred. The more blurred image is evaluated by the image clarity evaluation function of the three evaluation functions in parallel in step S3. If the original image corresponding to the grayscale data graph is a blurred image itself, the evaluation value of the blurred image corresponding to the blurred grayscale data graph will not differ much from that of the original image. However, if the original image is a clear image itself, the evaluation value of the blurred image will change greatly. Thus, through the dynamic defocus detection of step S5-2, it can be determined whether the microscopic image obtained by the current scanning is out of focus, and the dynamic defocus correction is performed when it is determined to be out of focus, to ensure that the microscopic image is clear and visible.

[0080] Preferably, the adaptive focusing microscope device further comprises an objective turret recognition module and a high-power LED light source.

[0081] The objective turret recognition module judges the magnification of the current microscope objective by installing magnets in different arrangements on each objective magnification channel of the microscope objective turret, and recognizing the arrangement of the magnets by a Hall element.

[0082] The PS unit obtains the magnification of the current microscope objective in real time by detecting the level state of the Hall element, and controls the high-power LED light source to adjust the power according to the magnification, so that the brightness of the high-power LED light source is adapted to the magnification of the current microscope objective, to realize a brighter observation field, and both focusing accuracy and focusing time are taken into account.

[0083] Preferably, the PS unit further sets the initial step size of the hill climbing search algorithm in step S5-3 according to the magnification of the current microscope objective, to obtain the highest focusing accuracy.

[0084] In addition, the present application can also adopt the following designs and functions:

[0085] The image output by the Gaussian noise filter is transmitted to the DDR of the PS unit through the AXI_DMA module, realizing the storage of the video, and facilitating the host computer to read the video stream data in the DDR of the PS unit. Alternatively, the PL unit sends a command to request the PS unit to transmit the video stream data in the DDR to the HDMI display screen. After receiving the image data, the HDMI display screen configures the image resolution through the VGA module and then sends the video stream to the external IO interface (HDMI interface in this case). In addition, the PL unit also encapsulates three motor drive interfaces to receive the drive instructions of the PS unit and control the movement of the motor.

[0086] The main tasks of the PS unit (i.e., the ARM end) include camera register configuration, motor drive initialization, LWIP protocol stack building, gigabit Ethernet initialization, and light source management system serial instruction reception and driving. By receiving the interrupt signals from the PL, the PS unit executes specific functions, such as monitoring the movement state of the three-axis motor, obtaining the image definition, and judging the focusing state.

[0087] The PS unit also builds multiple interrupts, including interrupts from the PL, for obtaining evaluation values, frame buffer numbers, and motor completion instructions. The timer interrupt is used to execute specific instructions and count the running time. In addition, the PS unit builds an LWIP protocol stack to transmit video stream data through the gigabit Ethernet and receive control instructions from the QT host computer. At the same time, the PS unit also builds a FAT32 file system to store the microscopic images taken by the camera into the SD card, facilitating subsequent testing and observation.

[0088] The host computer mainly realizes the display of video and the issuing of control instructions through the QT host computer. It uses Winsocket and OpenCV library to capture video stream data from Ethernet and converts it into QT image to display in the window. The whole process covers the reception of video stream data and the issuing of control instructions, both of which are efficient data transmission through Ethernet. According to different display needs, the QT host computer will send control instructions to the PS unit, and the PS unit will change the different DDR addresses of the Ethernet transmission data according to the instructions, so as to realize the display of different video data. This flexible control mechanism enables users to switch different video sources as needed and observe the microscopic image in real time. In addition to the display function of video, the QT host computer also provides rich control options. Users can achieve one-key automatic focusing, select focusing direction and focusing multiple, and also can turn on the dynamic defocusing scanning function, objective lens conversion function and real-time defocusing detection. In addition, the QT host computer can also control the xyz three-axis motor to realize accurate position control. In addition, the host computer also has the functions of picture storage and video recording, and can control the SD card to store picture data, etc. In addition to the display of video stream data, the QT host computer can also display the evaluation operator value, evaluation direction, evaluation state, motor position and real-time frame rate, etc. The display of these information enables users to understand the state and performance of the system in real time, which provides convenience for the operation and debugging of the microscope.

[0089] The PS unit encapsulates the protocol function of multiple motors, which can drive multiple motors at the same time to realize the three-axis linkage of the motorized microscope stage. The LWIP protocol stack is built to read the video stream data of the specified memory (the memory stream data size is 1280*1024) through UDP port, and the WinSocket is used to capture it on the PC, and the UDP receiving host computer is programmed by calling the qt and opencv library, which can realize the display and operation of the image. Through the host computer, the control instructions are issued to control the focusing system, including the automatic focusing of the fixed plane, the storage of the picture into the TF card, the switching of the video processing source, and the opening and closing of the dynamic defocusing.

[0090] The host computer mainly realizes the display of video and the issuing of control instructions through the QT host computer. It uses Winsocket and OpenCV library to capture video stream data from the Ethernet and converts it into a QT image to display in the window. The whole process covers the receiving of video stream data and the issuing of control instructions, and efficient data transmission is realized through the Ethernet. According to different display requirements, the QT host computer will send control instructions to the PS end, and the PS end will change the different DDR addresses of the Ethernet transmission data according to the instructions, so as to realize the display of different video data. This flexible control mechanism enables users to switch different video sources as needed and observe the microscopic image in real time. In addition to the video display function, the QT host computer also provides rich control options. Users can achieve one-key automatic focusing, select the focusing direction and focusing multiple by sending instructions, and can also turn on dynamic defocusing detection, scanning splicing, intelligent objective conversion and other functions to realize intelligent control. In addition, the QT host computer can also control the xyz three-axis motor to realize accurate position control. In addition, the host computer also has the functions of picture storage and video recording, and can control the SD card to store picture data. In addition to the display of video stream data, the QT host computer can also display evaluation operator value, evaluation direction, evaluation state, motor position and real-time frame rate and other information. The display of these information enables users to understand the state and performance of the system in real time, providing convenience for the operation and debugging of the microscope.

[0091] The present application is not limited to the above specific embodiments, according to the above content, according to the ordinary technical knowledge and conventional means in the art, without departing from the above basic technical idea of the present application, the present application can also make other various forms of equivalent modification, replacement or change, all fall within the protection scope of the present application.

Claims

1. An adaptive focusing microscope device, comprising a camera and a motor module; the camera is used to scan the microscope platform to obtain a microscopic image, and the motor module drives the microscope platform to move to the focal plane according to a focusing control command; Its features are: It also includes: a data processor, which adopts a ZYNQ heterogeneous multi-core architecture consisting of PS units and PL units; The adaptive focusing microscope device performs automatic focusing control according to the following procedure: Step S1: The camera scans and obtains microscopic images by line-by-line scanning, and simultaneously outputs the scanned line scan data of each line of the microscopic image to the PL unit in Bayer array format video stream data; Step S2: Each time the PL unit receives a line of microscopic image scan data, it performs pre-processing, including: Bayer array format video stream data is first buffered in a FIFO memory to synchronize with the internal clock domain of the PL unit; the synchronized data is converted into RGB888 video stream data through a color interpolation and restoration algorithm; the RGB888 video stream data is converted into YCbcr video stream data through a color space conversion algorithm, and the Y component is extracted as grayscale data; the grayscale data is filtered by a Gaussian noise filter to obtain a grayscale data image. Step S3: The PL unit starts scanning the third row of microscopic image row scan data from the camera. When each row of microscopic image row scan data begins to be scanned, the image convolution operation in the image sharpness evaluation function is performed on the grayscale data obtained by processing the first two rows of microscopic image row scan data in step S2. This continues until the last row of microscopic image row scan data is scanned and processed in step S2. Then, the image sharpness evaluation value of the microscopic image is calculated based on the result of the image convolution operation using the image sharpness evaluation function. Step S4: The PL unit transmits the sharpness evaluation value to the DDR of the PS unit and sends an interrupt notification to the PS unit; Step S5: When the autofocus function is enabled, the PS unit reads the sharpness evaluation value from the DDR when it receives the interrupt notification, and enters the autofocus process to perform autofocus processing based on the sharpness evaluation value and generate the focus control command.

2. The adaptive focusing microscope device according to claim 1, characterized in that: In step S3, the image sharpness evaluation function includes three evaluation functions: linearity evaluation function, first sensitivity evaluation function, and second sensitivity evaluation function; the grayscale data image output by the Gaussian noise filter is evaluated in parallel by these three evaluation functions to obtain three sharpness evaluation values ​​for the microscopic image; In step S5, the autofocus process includes: Step S5-1: Obtain three sharpness evaluation values ​​for the microscopic image obtained in this scan and the microscopic image obtained in the previous scan, which are respectively denoted as: current linearity evaluation function evaluation value, previous linearity evaluation function evaluation value, current first sensitivity evaluation function evaluation value, previous first sensitivity evaluation function evaluation value, current second sensitivity evaluation function evaluation value, and previous second sensitivity evaluation function evaluation value; Step S5-2: Determine whether the following two conditions are met simultaneously: the rate of change of the current evaluation value of the first sensitivity evaluation function relative to the previous evaluation value of the first sensitivity evaluation function is above the preset rate of change threshold; and the rate of change of the current evaluation value of the second sensitivity evaluation function relative to the previous evaluation value of the second sensitivity evaluation function is above the preset rate of change threshold. If the judgment result is negative, then: the evaluation value of the linearity evaluation function is input into the hill climbing search algorithm; If the judgment result is yes, then: input the evaluation value of the first sensitivity evaluation function and the evaluation value of the second sensitivity evaluation function into the hill climbing search algorithm respectively; Step S5-3: Based on the judgment result of step S5-2, generate the focusing control command, which includes: first obtaining the movement direction and step distance of the motor module driving the microscope platform to move through the hill climbing search algorithm, and then performing peak finding operation, so as to take the peak point determined by the peak finding operation as the moment when the motor module stops driving the microscope platform to move. Specifically, if the judgment result of step S5-2 is yes, and both the evaluation value of the first sensitivity evaluation function and the evaluation value of the second sensitivity evaluation function appear as peak points when they are input into the hill climbing search algorithm, then the peak point is determined by the peak finding operation.

3. The adaptive focusing microscope device according to claim 2, characterized in that: The linearity evaluation function is the Variance evaluation function, the first sensitivity evaluation function is the Brenner evaluation function, and the second sensitivity evaluation function is the Tenegard evaluation function.

4. The adaptive focusing microscope apparatus according to claim 2 or 3, characterized in that: In step S2, the grayscale data image output by the Gaussian noise filter is further processed by a Gaussian blur filter to become a blurred grayscale data image. In step S3, the blurred grayscale data image, like the grayscale data image, is evaluated in parallel by an image sharpness evaluation function containing three evaluation functions to obtain three sharpness evaluation values ​​of the microscopic image obtained in this scan, which are respectively denoted as the blurred evaluation value of the linearity evaluation function, the blurred evaluation value of the first sensitivity evaluation function, and the blurred evaluation value of the second sensitivity evaluation function. While performing step S5-2, with the dynamic defocus detection function enabled, the PS unit determines whether the following conditions are met simultaneously. If so, it is determined that the microscopic image obtained from this scan has a defocus blur phenomenon, and dynamic defocus correction should be performed immediately. Conditions: The rate of change of the fuzzy evaluation value of the linearity evaluation function relative to the evaluation value of the linearity evaluation function is above the preset fuzzy change rate threshold; and the rate of change of the fuzzy evaluation value of the first sensitivity evaluation function relative to the evaluation value of the first sensitivity evaluation function is above the preset fuzzy change rate threshold. Furthermore, the rate of change of the fuzzy evaluation value of the second sensitivity evaluation function relative to the evaluation value of the second sensitivity evaluation function is above the preset fuzzy change rate threshold.

5. The adaptive focusing microscope apparatus according to claim 2 or 3, characterized in that: The adaptive focusing microscope device also includes an objective lens turret recognition module and a high-power LED light source; The objective turret recognition module determines the magnification of the current microscope objective by installing magnets in different arrangements on each objective magnification channel of the microscope objective turret and identifying the arrangement of the magnets using a Hall element. The PS unit obtains the magnification of the current microscope objective in real time by detecting the level state of the Hall element, and controls the high-power LED light source to adjust the power according to the magnification so that the brightness of the high-power LED light source is adapted to the magnification of the current microscope objective.

6. The adaptive focusing microscope apparatus according to claim 5, characterized in that: The PS unit also sets the initial step size of the hill-climbing search algorithm in step S5-3 according to the magnification of the current microscope objective.

Images