A method and system for automatically searching and imaging freely moving samples

Abstract

The present invention discloses an automated search imaging method and system for freely moving samples. The method includes a search thread and a tracking thread; when the sample is not detected in the microscope field image, the search thread is entered, and the coordinates of the sample platform at the center of the microscope field of view at the next moment are changed according to the calculation results of the search algorithm until the sample is detected in the microscope field of view image, and the tracking thread is entered; when the sample is detected in the microscope field of view, the tracking thread is entered, and the coordinates of the sample platform at the center of the microscope field of view at the next moment are changed according to the position of the sample's centroid, until the sample is not detected in the microscope field of view, and the search thread is entered. The present invention solves the problem of collection interruption when the sample escapes the field of view in the traditional real-time tracking strategy of freely moving samples by following a pre-set search thread at the beginning of the search and when the tracking is interrupted. The system using the method simplifies the optical path into a single-path high-power behavior-fluorescence integrated detection system.

Description

Technical Field

[0001] The present invention belongs to the field of in vivo microscopic imaging, and more specifically, relates to an automated search imaging method and system for freely moving samples. Background Art

[0002] The exploration of the dynamic changes in the life processes of various living organisms and their real-time behaviors is an important topic of concern in the current field of neuroscience research, such as the microscopic observation of neuronal signals of living nematodes, fruit flies, and zebrafish and the collection and analysis of the control patterns of their behavioral performance in a free-motion state. The structures of these samples are usually distributed in three-dimensional space and have high movement speeds and large activity ranges. Therefore, the observation of these living samples must dynamically capture the change information in three-dimensional space with high resolution and high speed and keep the samples in the field of view at all times, which poses a great challenge to the spatiotemporal performance of current optical microscopes.

[0003] In order to ensure that the living samples are within the imaging field of view during the long acquisition time, the current related research methods are to fix the samples within the imaging field of view through anesthesia, adhesion, chamber restriction, etc., and then perform a three-dimensional scan on them; the other method is to make the samples themselves mistakenly believe that they are in dynamic motion through the display of a virtual environment, thereby generating near-real neural activity and corresponding real-time behavior. However, since the sample individuals themselves do not actually move freely, the above two methods have unpredictable effects on the dynamic changes of neural circuits in these states and the real free motion state, which can mislead the biological conclusions drawn in this state. The most widely accepted method at present is to track the samples in real time through an electric translation stage and control software, so as to obtain their real free motion behavior and the associated neural activity dynamics.

[0004] The current mainstream hardware design of free-motion sample tracking combined with real-time fluorescence microscopy is usually designed as a dual-channel detection channel, namely a behavior detection channel and a fluorescence detection channel. In the behavior detection channel, the light source uses near-infrared LED illumination, and the detection end uses a low-magnification objective lens (such as a 4x objective lens) combined with a detection camera to capture the behavior of the moving sample in real time. In the fluorescence detection channel, a high-power wide-spectrum light source and a color filter generate excitation light to excite specific fluorescent marker proteins. The detection end uses a high-magnification microscope objective lens (such as a 20x objective lens) combined with a high-quantum-efficiency sCMOS detector to collect high-resolution fluorescence images of the sample for subsequent neural activity analysis. The reason why the behavior detection channel in the above optical path design uses a low-magnification detection objective lens is that when imaging the free-motion sample, if the behavior detection channel and the fluorescence detection channel use the same high-magnification objective lens for detection, the field of view of the high-magnification objective lens itself will be greatly reduced compared to the low-magnification detection objective lens, so the moving sample is very easy to escape the detection field of view. The traditional tracking algorithm assumes that the image collected by the behavior channel detection camera contains the contour information of the moving sample, and further extracts the center position coordinates of the sample through the center line extraction algorithm for subsequent motion control. Therefore, when the moving sample escapes the field of view, the traditional tracking strategy cannot make a correct response, resulting in wrong positioning and loss of the sample. Considering the above reasons, the traditional tracking algorithm requires the detection camera of the behavior channel to always detect the contour information of the sample, so the sample needs to be kept in the imaging field of view at all times, and then the hardware design needs to use a low-magnification detection objective lens to achieve a larger imaging field of view. Therefore, in this kind of traditional free-motion sample tracking design based on the behavior channel to capture the sample contour information, the detection end needs independent fluorescence detection channels composed of high-magnification detection objectives and behavior detection channels composed of low-magnification objectives, which is obviously incompatible with the single-objective detection mode of commercial microscopes, so it introduces additional complexity in the actual construction of the imaging system; more importantly, in this detection mode, the moving sample needs to be manually placed in the imaging field of view during the initial tracking, and if the sample moves too fast and escapes the field of view at a certain moment during the tracking process, the algorithm cannot respond correctly, and timely human intervention is required. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides an automated search imaging system for free-motion samples, which aims to achieve automated sample search and simultaneous free-motion sample behavior posture capture and fluorescence data acquisition in a conventional commercial microscope single high-power detection objective imaging mode by utilizing a free-motion sample tracking algorithm combined with a motion search strategy, thereby reducing the overall complexity of the hardware system, improving the stability of the real-time tracking algorithm, further reducing the cost of human intervention, and improving the automation characteristics of the application. This solves the technical problems of the prior art of manual intervention initialization, tracking breakpoints, and the need for a low-power objective lens that leads to complex optical path design.

[0006] To achieve the above object, according to one aspect of the present invention, there is provided an automated search imaging method for a freely moving sample, comprising a search thread and a tracking thread;

[0007] When the sample is not detected in the microscope field image, the search thread is entered, and the coordinates of the sample platform at the center of the microscope field at the next moment are changed according to the calculation results of the search algorithm until the sample is detected in the microscope field image, and the tracking thread is entered;

[0008] When a sample is detected in the microscope field of view, the tracking thread is entered, and the coordinates of the sample platform at the center of the microscope field of view at the next moment are changed according to the position of the sample's center of mass, until the sample is no longer detected in the microscope field of view, and the search thread is entered.

[0009] Preferably, the automated search imaging method for free-moving samples, wherein the search thread comprises the following steps:

[0010] (1) Determine the search field of view motion route: Based on the coordinates of the sample platform at the center of the microscope field of view at the current moment, a motion search algorithm is used to search and obtain the target coordinates of the sample platform at the center of the microscope field of view at the next moment;

[0011] (2) Acquiring an image within the search field of view: moving the target coordinates of the sample platform obtained in step (1) to the center of the microscope field of view, and acquiring an illumination light imaging image within the current field of view for sample detection;

[0012] (3) Sample detection: Perform sample contour recognition on the illumination light imaging image in the current field of view obtained in step (2) to determine whether there is a sample in the field of view. If it is determined that there is a sample, calculate the current center of mass coordinates of the sample to initialize the tracking thread, exit the search thread, and enter the tracking thread; otherwise, repeat step (1).

[0013] Preferably, in the automated search imaging method for freely moving samples, the motion search algorithm is selected from a global search algorithm, a three-step search algorithm, a diamond search algorithm or a hexagonal search algorithm.

[0014] Preferably, in the method for automated search imaging of freely moving samples, the step (3) of determining whether a sample exists in the field of view specifically comprises the following sub-steps:

[0015] (3-1) Counting judgment: Get the number of identified sample contours. When the number of identified contours is 0, it is judged that there is no sample in the field of view; when the number of identified contours is greater than or equal to 1, go to step (3-2);

[0016] (3-2) Differential discrimination: wait for a preset time, keep the coordinates of the sample platform at the center of the microscope field of view unchanged, obtain an illumination light imaging differential image with a preset time difference from the illumination light imaging image in the current field of view, and judge the degree of difference between the illumination light imaging image in the current field of view and the illumination light imaging differential image. When the degree of difference exceeds a preset threshold, it is judged that there is a sample in the field of view; otherwise, it is judged that there is no sample in the field of view.

[0017] Preferably, in the method for automated search imaging of freely moving samples, the step (3) of calculating the coordinates of the center of mass of the current sample is specifically:

[0018] (3-3) Calculating the sample centroid: The contour of the difference between the illumination light imaging difference image and the image in the current field of view identified by the sample contour recognition method is used as the sample contour, and its centroid is calculated as the coordinate of the current sample centroid;

[0019] Preferably, the tracking thread of the automated search imaging method for freely moving samples comprises the following steps:

[0020] S1, taking the coordinates of the current sample centroid as the center of the microscope field of view at the next moment, so that the coordinates of the current sample centroid move to the center of the microscope field of view, and obtaining an illumination light imaging image in the current field of view for sample detection;

[0021] S2. Sample detection: perform sample contour recognition on the illumination light imaging image in the current field of view obtained in step S1 to determine whether there is a sample in the field of view. If it is determined that there is a sample, update the current centroid coordinates of the sample and repeat step S1; otherwise, use the current sample centroid coordinates as the sample platform coordinates of the microscope field of view center at the next moment, and initialize to enter the search thread.

[0022] Preferably, in the method for automated search and imaging of freely moving samples, the step S2 of determining whether there is a sample in the field of view specifically includes the following sub-steps:

[0023] S2.1 Counting judgment: Get the number of identified sample contours. When the number of identified contours is 0, it is judged that there is no sample in the field of view; when the number of identified contours is equal to 1, it is judged that there is a sample and the centroid coordinates of the current sample are updated with the identified contours. Otherwise, if the number of identified contours is greater than or equal to 2, enter step S2.2;

[0024] S2.2 Differential judgment: wait for a preset time, keep the coordinates of the sample platform at the center of the microscope field of view unchanged, obtain the illumination light imaging differential image with a preset time difference from the image in the current field of view, and judge the difference between the illumination light imaging image in the current field of view and the illumination light imaging differential image. When the difference exceeds the preset threshold, there is a sample in the field of view, and go to step S2.3; otherwise, judge that there is no sample in the field of view

[0025] Preferably, in the method for automatically searching and imaging the freely moving sample, the updating of the current centroid coordinates of the sample in step S2 is specifically:

[0026] S2.3 The contour of the difference between the illumination light imaging difference image and the illumination light imaging image in the current field of view is identified by the sample contour recognition method as the sample contour, and its center of mass is calculated as the coordinate of the current sample center of mass.

[0027] According to another aspect of the present invention, an automated search imaging system is provided, which applies the automated search imaging method for freely moving samples provided by the present invention.

[0028] Preferably, the automated search imaging system comprises: an illumination light source, a fluorescence excitation light source, a behavior detection channel, a fluorescence detection channel, a lower computer, and a motion mechanism for driving the sample platform to move;

[0029] The illumination light source and the fluorescence excitation light source are projected into the working range of the microscope;

[0030] The behavior detection channel and the fluorescence detection channel respectively collect illumination light imaging images and fluorescence images, and output the images to the lower computer; the behavior detection channel and the fluorescence detection channel can share the same high-magnification detection objective lens to reduce the complexity of the overall optical path, or use different magnification detection objective lenses respectively; the fluorescence detection channel can be single-wavelength, dual-wavelength or multi-wavelength detection.

[0031] The fluorescence detector of the fluorescence detection channel is preferably a CCD or CMOS with high sensitivity and high quantum efficiency; the camera of the behavior detection channel can be an industrial camera with lower resolution, or other scientific research cameras with sufficiently high sensitivity.

[0032] The illumination light emitted by the illumination light source is imaged through the behavior detection channel to obtain an illumination light imaging image, which is used to collect a behavior image of the freely moving sample;

[0033] The excitation light emitted by the fluorescence excitation light source, when a sample is present therein, excites the sample to generate fluorescence, and is imaged through the fluorescence detection channel to obtain a sample fluorescence image, which is used to collect the sample fluorescence image;

[0034] The lower computer is used to run the automated search imaging method for freely moving samples provided by the present invention, obtain the target coordinates of the sample platform at the center of the microscope field of view at the next moment, and drive the motion mechanism to move the target coordinates of the sample platform to the center position of the microscope field of view.

[0035] The motion mechanism can be two-dimensional, one-dimensional or three-dimensional, depending on the movement mode of the sample. It is preferably a high-speed two-dimensional translation stage. The high-speed two-dimensional translation stage can adopt a dual-axis brushless DC motor, or other electric translation stages or piezoelectric translation stages that meet high-speed tracking requirements.

[0036] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0037] The automated search and imaging method for freely moving samples provided by the present invention is based on a motion search strategy and a real-time tracking strategy for freely moving samples. By following a pre-set search thread at the beginning of the search and at the interruption of tracking, the problem of acquisition interruption when the sample escapes the field of view in the traditional real-time tracking strategy for freely moving samples is solved, thereby reducing the cost of human intervention and achieving highly automated acquisition and imaging.

[0038] The preferred solution adds a differential discrimination step, which improves the success rate of the search thread and reduces the misjudgment rate of the tracking thread. At the same time, according to the different characteristics of artifacts in the search thread and the tracking thread and the characteristics of the probability of sample occurrence, different conditions for differential discrimination are made to further reduce the judgment time, improve the probability of successful tracking and extend the time of continuous tracking.

[0039] The automated search imaging system for freely moving samples provided by the present invention simplifies the complex dual-path low-magnification behavior detection-high-magnification fluorescence detection system used in traditional tracking into a single-path high-magnification behavior-fluorescence integrated detection system, thereby achieving compatibility with the single objective lens detection mode of conventional commercial microscopes, greatly simplifying the optical path mode, and facilitating the overall construction of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic flow chart of an automated search imaging method for a freely moving sample provided by an embodiment of the present invention;

[0041] Figure 2 It is a schematic diagram of the structure of an automated search imaging system for freely moving samples provided by an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of a single tracking according to an embodiment of the present invention, where P1 to P9 represent different positions, wherein P1 to P6 are the search process, and P7 to P9 are the tracking process;

[0043] Figure 4 This is a flow chart of the image acquisition analysis feedback algorithm of the behavior channel of an embodiment of the present invention;

[0044] In all the drawings, the same figure numbers are used to represent the same elements or structures, among which: 11 is a wide-field illumination light source, 12 is an infrared LED illumination light source, 21 is a two-dimensional tracking electric translation stage, 31 is a high-magnification microscope objective, 32 is a one-dimensional piezoelectric ceramic translation stage, 33 is a dichroic mirror, 34 is a tube lens, 41 is a dichroic mirror, 42 is a microlens array (MLA), 43 and 44 are a set of relay lens systems, 45 is a dichroic mirror, 46 and 48 are color filters, 47 and 49 are sCMOS detection cameras, and 51 is an industrial camera. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] An automated search imaging method for a freely moving sample, comprising a search thread and a tracking thread;

[0047] When the sample is not detected in the microscope field image, the search thread is entered, and the coordinates of the sample platform at the center of the microscope field at the next moment are changed according to the calculation results of the search algorithm until the sample is detected in the microscope field image, and the tracking thread is entered;

[0048] When a sample is detected in the microscope field of view, the tracking thread is entered, and the coordinates of the sample platform at the center of the microscope field of view at the next moment are changed according to the position of the sample's center of mass, until the sample is no longer detected in the microscope field of view, and the search thread is entered.

[0049] The search thread includes the following steps:

[0050] (1) Determine the search field of view motion route: Based on the coordinates of the sample platform at the center of the microscope field of view at the current moment, a motion search algorithm is used to search and obtain the target coordinates of the sample platform at the center of the microscope field of view at the next moment;

[0051] The motion search algorithm may include but is not limited to a global search algorithm, a three-step search algorithm, a diamond search algorithm or a hexagonal search algorithm; wherein the global search algorithm is to traverse all coordinates within a certain search range, calculate the matching degree with the search area one by one, select the coordinate with the highest matching degree as the best search point, and use the position vector of the point and the center point of the search range as the motion vector. The three-step search method and the diamond search method are fast search type algorithms, which improve the search speed by searching for matching blocks according to certain mathematical rules, but the cost is that a sub-optimal matching block may be obtained.

[0052] (2) Acquiring images within the search field of view: Move the target coordinates of the sample platform obtained in step (1) to the center of the microscope field of view, and obtain the illumination light imaging image within the current field of view for sample detection; preferably, a high-speed motion platform is used as the sample platform, with a moving speed of 60-120 mm / s, and the speed of the sample platform needs to match the sample movement speed.

[0053] (3) Sample detection: Perform sample contour recognition on the illumination light imaging image in the current field of view obtained in step (2) to determine whether there is a sample in the field of view. If it is determined that there is a sample, calculate the current centroid coordinates of the sample to initialize the tracking thread, exit the search thread, and enter the tracking thread; otherwise, repeat step (1);

[0054] The step of determining whether there is a sample in the field of view specifically includes the following sub-steps:

[0055] (3-1) Counting judgment: Get the number of identified sample contours. When the number of identified contours is 0, it is judged that there is no sample in the field of view; when the number of identified contours is greater than or equal to 1, go to step (3-2);

[0056] (3-2) Differential discrimination: waiting for a preset time, keeping the coordinates of the sample platform at the center of the microscope field of view unchanged, obtaining an illumination light imaging differential image with a preset time difference from the illumination light imaging image in the current field of view, judging the degree of difference between the illumination light imaging image in the current field of view and the illumination light imaging differential image, and judging that a sample exists in the field of view when the degree of difference exceeds a preset threshold; otherwise, judging that a sample does not exist in the field of view;

[0057] The calculation of the centroid coordinates of the current sample is specifically as follows:

[0058] (3-3) Calculating the sample centroid: taking the contour of the difference between the illumination light imaging difference image and the image in the current field of view identified by the sample contour recognition method of the illumination light imaging difference image as the sample contour, and calculating its centroid as the coordinate of the current sample centroid; when the contour of the difference between the illumination light imaging difference image and the image in the current field of view cannot be identified by the sample contour recognition method of the illumination light imaging difference image, repeating step (1);

[0059] The following steps are involved in tracking threads:

[0060] S1, taking the coordinates of the current sample centroid as the center of the microscope field of view at the next moment, so that the coordinates of the current sample centroid move to the center of the microscope field of view, and obtaining an illumination light imaging image in the current field of view for sample detection;

[0061] S2, sample detection: perform sample contour recognition on the illumination light imaging image in the current field of view obtained in step S1, and determine whether there is a sample in the field of view. If it is determined that there is a sample, update the current centroid coordinates of the sample and repeat step S1; otherwise, use the current sample centroid coordinates as the coordinates of the microscope field of view center on the sample platform at that moment, and initialize and enter the search thread;

[0062] The step of determining whether there is a sample in the field of view specifically includes the following sub-steps:

[0063] S2.1 Counting judgment: Get the number of identified sample contours. When the number of identified contours is 0, it is judged that there is no sample in the field of view; when the number of identified contours is equal to 1, it is judged that there is a sample and the centroid coordinates of the current sample are updated with the identified contours. Otherwise, if the number of identified contours is greater than or equal to 2, enter step S2.2;

[0064] S2.2 Differential judgment: wait for a preset time, keep the coordinates of the sample platform at the center of the microscope field of view unchanged, obtain an illumination light imaging differential image with a preset time difference from the image in the current field of view, and judge the difference between the illumination light imaging image in the current field of view and the illumination light imaging differential image. When the difference exceeds a preset threshold, there is a sample in the field of view, and go to step S2.3; otherwise, it is judged that there is no sample in the field of view;

[0065] The current centroid coordinates of the updated sample are specifically:

[0066] S2.3 The contour of the difference between the illumination light imaging difference image and the illumination light imaging image in the current field of view is identified by the sample contour recognition method as the sample contour, and its center of mass is calculated as the coordinate of the current sample center of mass.

[0067] Contour recognition can use a series of traditional image algorithms such as the contour extraction algorithm based on OpenCV, the center extraction algorithm or the skeleton recognition algorithm for contour positioning, or a contour recognition algorithm based on deep learning for fast position feedback; the contour extraction algorithm based on OpenCV includes the steps of Gaussian blur, binarization, corrosion, and contour extraction. Contour recognition can set a variety of conditions to match samples of different stages or types, such as the area, shape characteristics of the contour, and the intensity characteristics of the pixels within the contour. The contour recognition algorithm can adaptively set the binarization threshold and contour satisfaction conditions based on the image, or manually set the threshold, exposure time, contour conditions and other parameters to meet the imaging requirements of different lighting conditions and different samples.

[0068] Optionally, the illumination light can directly adopt the fluorescence excitation light, or even use the fluorescence image as the illumination light imaging, so as to perform automated search and real-time tracking based on the fluorescence intensity.

[0069] An automated search imaging system using the automated search imaging method for freely moving samples provided by the present invention comprises: an illumination light source, a fluorescence excitation light source, a behavior detection channel, a fluorescence detection channel, a lower computer, and a motion mechanism that drives the sample platform to move;

[0070] The illumination light source and the fluorescence excitation light source are projected into the working range of the microscope;

[0071] The behavior detection channel and the fluorescence detection channel collect illumination light imaging images and fluorescence images respectively, and output the images to the lower computer; the behavior detection channel and the fluorescence detection channel can share the same high-magnification detection objective lens to reduce the complexity of the overall optical path; or use different magnification detection objective lenses respectively, which can reduce the response time and speed requirements for the tracking translation stage, and ensure that the samples overflow the field of view as little as possible during the tracking process, but the search algorithm is still required to ensure automated search and tracking. The fluorescence detection channel can be single-wavelength, dual-wavelength or multi-wavelength detection.

[0072] The fluorescence detector of the fluorescence detection channel is preferably a CCD or CMOS with high sensitivity and high quantum efficiency; the camera of the behavior detection channel can be an industrial camera with lower resolution, or other scientific research cameras with sufficiently high sensitivity.

[0073] The illumination light emitted by the illumination light source is imaged through the behavior detection channel to obtain an illumination light imaging image, which is used to collect a behavior image of the freely moving sample;

[0074] The excitation light emitted by the fluorescence excitation light source excites the sample to generate fluorescence when there is a sample therein, and is imaged through the fluorescence detection channel to obtain a sample fluorescence image, which is used to collect a sample fluorescence image, such as a sample nerve cell fluorescence image, etc.

[0075] The lower computer is used to run the automated search imaging method for freely moving samples provided by the present invention, obtain the target coordinates of the sample platform at the center of the microscope field of view at the next moment, and drive the motion mechanism to move the target coordinates of the sample platform to the center position of the microscope field of view.

[0076] The motion mechanism can be two-dimensional, one-dimensional or three-dimensional, depending on the movement mode of the sample. It is preferably a high-speed two-dimensional translation stage. The high-speed two-dimensional translation stage can adopt a dual-axis brushless DC motor, or other electric translation stages or piezoelectric translation stages that meet high-speed tracking requirements.

[0077] The collected behavioral images of the freely moving samples and the fluorescence images of the samples can be analyzed synchronously to obtain the relationship between the changes in the fluorescence generation site of the sample and the sample behavior.

[0078] The following are examples:

[0079] The automated search imaging system provided in this embodiment is as follows: Figure 2 As shown, the overall hardware system is built based on a conventional commercial microscope:

[0080] Illumination light source and fluorescence excitation light source: A high-power broad-spectrum light source combined with an excitation filter is used at the illumination end to excite specific fluorescent-labeled proteins, and a near-infrared LED or a bright-field light source after filtering out interfering light with a bright-field filter is used to illuminate the behavioral channel. Both parts are standard accessories for conventional commercial microscopes. The illumination light source can be an infrared LED light source or a bright-field light source after filtering out interfering fluorescence channel wavelengths in a conventional commercial microscope.

[0081] At the detection end, the behavior detection channel and the fluorescence detection channel use a single high-power detection objective to collect the specific fluorescence information emitted by the sample and the contour information generated by the behavior channel light source illumination, and correct the existing aberrations through the built-in sleeve lens of the microscope, and further transmit them through the microscope camera port to the fluorescence detection channel camera and the eyepiece port to the behavior channel detection camera installed on the microscope eyepiece module via an adapter sleeve.

[0082] The detection mode of the fluorescence detection channel can adopt fluorescence microscopy imaging detection methods such as confocal imaging, wide field detection or light field detection.

[0083] The overall optical path of this embodiment is built based on the Olympus conventional commercial microscope IX73. Figure 2The fluorescence channel illumination at the illumination end shown uses a conventional halogen light source to achieve dual-channel fluorescence excitation through epi-illumination from the bottom of the sample, and the behavior channel illumination uses an infrared LED light source for transmissive illumination from the upper side of the sample to facilitate subsequent sample contour recognition. The fluorescence light field detection optical path and the behavior channel detection optical path at the detection end occupy the camera installation port and eyepiece port of the IX73 microscope respectively, where the light field detection optical path realizes the acquisition of dual-channel two-dimensional light field images based on the traditional light field detection modality, and the detection camera of the behavior channel is installed at the eyepiece end of the microscope through an adapter sleeve to collect contour information of living samples.

[0084] The sample platform includes a preset imaging system such as a culture medium and a motion mechanism. The upper sample is located in the preset imaging system such as a culture medium, and can achieve complete free movement. The motion mechanism of this embodiment adopts a high-speed two-dimensional translation stage. The entire culture medium is located on the high-speed two-dimensional translation stage. The high-speed two-dimensional translation stage is driven to compensate for the real-time position so that the sample can be kept in the center of the field of view. The culture medium is placed on the high-speed two-dimensional translation stage as a whole. The high-speed two-dimensional translation stage used is a two-dimensional electric translation stage MLS-203 adapted to commercial Olympus microscopes produced by Sorebo. Its response time and speed under high-speed movement meet the feedback requirements for tracking freely moving nematodes. Here, MLS-203 can be replaced with any two-dimensional high-speed electric translation stage or piezoelectric displacement stage that meets the requirements.

[0085] In this embodiment, by combining the traditional light field detection optical path mode (placing the microlens array in the original image plane of the detection optical path) and the above-mentioned free sample real-time tracking algorithm combined with the motion search strategy, the automated real-time three-dimensional fluorescence high-resolution and behavioral imaging of the traditional model animal free-moving Caenorhabditis elegans is achieved. Before the experiment, the L4-stage Caenorhabditis elegans were selected in advance into the culture medium to allow it to move freely, and the culture medium was placed on a high-speed two-dimensional translation stage.

[0086] like Figure 1 As shown, the automated search imaging system provided in this embodiment implements an automated search imaging method for freely moving samples, comprising the following steps:

[0087] Hardware initialization:

[0088] In this module, the hardware modules in the system are first initialized, including the two-dimensional tracking translation stage, behavior and fluorescence channel detection cameras, etc. After the initialization is completed, the tracking translation stage is reset and some camera shooting parameters are preset to realize the customized acquisition process, and then the camera acquisition process is started.

[0089] First, the algorithm initializes and resets the hardware to achieve normal subsequent calls and stable operation, and sets parameters related to image acquisition, such as exposure time, and then the camera starts to acquire images. After acquiring a single image, the algorithm enters the search thread.

[0090] Enter the search thread:

[0091] When the sample is not detected in the microscope field image, the search thread is entered, and the coordinates of the sample platform at the center of the microscope field at the next moment are changed according to the calculation results of the search algorithm until the sample is detected in the microscope field image, and the tracking thread is entered;

[0092] During actual operation, the sample will be manually moved to the vicinity of the center of the microscope field of view during initialization, and then the search thread will be run. At this time, the sample is generally still moving around the current field of view. Similarly, when entering the search thread from the tracking thread, the sample should still be moving around the current field of view. Using the global search method can ensure a more accurate search for the sample. Although other methods may be faster, it is easy to lose the sample during the search, resulting in the need for manual alignment.

[0093] After starting the search thread, since the moving sample moves freely in the imaging system and the system as a whole uses a single high-magnification detection objective with a small detection field of view, the initial acquired image will not contain complete moving sample contour information when the sample moves quickly. Even if the moving sample is manually adjusted to the center of the field of view in advance, the program will enter the tracking thread when the sample has escaped the field of view due to the delay in program startup and initialization. Therefore, the tracking thread in the program needs to re-enter the search thread in the event of tracking failure to ensure that the entire observation and recording process is automated.

[0094] Different search views of the search path set based on the global search method can be just adjacent, or have a certain degree of overlap to ensure the integrity of the search and prevent missed searches; the search direction of the search path set based on the global search method can be spiral clockwise or counterclockwise, or other logical search methods.

[0095] This embodiment takes the global search method as an example, and the specific steps are as follows:

[0096] (1) Determine the search field of view motion route: Based on the coordinates of the sample platform at the center of the microscope field of view at the current moment, a motion search algorithm is used to search and obtain the target coordinates of the sample platform at the center of the microscope field of view at the next moment;

[0097] In this embodiment, since there is no excessive requirement on the time cost occupied by the search module, the path search method used is preferentially a global search method to ensure that the freely moving biological samples can be accurately searched.

[0098] (2) Collecting images within the search field of view: Move the target coordinates of the sample platform obtained in step (1) to the center of the microscope field of view, and obtain the illumination light imaging image within the current field of view for sample detection; use a high-speed motion platform as the sample platform, with a moving speed of 60-120 mm / s

[0099] The search window is initialized with the starting point of the current search window as the center point of the search path, and searches are performed outward in a clockwise or counterclockwise direction from the center point. The specific implementation is to drive the high-speed two-dimensional translation stage to perform step scanning according to the search path pre-set according to the global search method;

[0100] (3) Sample detection: Perform sample contour recognition on the illumination light imaging image in the current field of view obtained in step (2) to determine whether there is a sample in the field of view. If it is determined that there is a sample, calculate the current centroid coordinates of the sample to initialize the tracking thread, exit the search thread, and enter the tracking thread; otherwise, repeat step (1);

[0101] First, the collected image is downsampled to reduce the amount of processing data and improve the imaging flux. Then, Gaussian blur, adaptive threshold binarization, image erosion and contour extraction are performed on the image in sequence. Contours that meet the preset contour conditions are screened out and their contour centroids are calculated. The set contour conditions include contour area, intensity information of pixels within the contour, etc.

[0102] (3-1) Counting judgment: Get the number of identified sample contours. When the number of identified contours is 0, it is judged that there is no sample in the field of view; when the number of identified contours is greater than or equal to 1, go to step (3-2);

[0103] Image acquisition and judgment of conditional contour information are performed at each search window to determine whether the current field of view contains motion samples. When suspected motion samples are found, that is, when the number of identified contours is greater than or equal to 1, the illumination imaging difference image after a fixed time interval continues to be acquired at the current position, and the judgment is made through the difference image of the two frames.

[0104] (3-2) Differential discrimination: waiting for a preset time, keeping the coordinates of the sample platform at the center of the microscope field of view unchanged, obtaining an illumination light imaging differential image with a preset time difference from the illumination light imaging image in the current field of view, judging the degree of difference between the illumination light imaging image in the current field of view and the illumination light imaging differential image, and judging that a sample exists in the field of view when the degree of difference exceeds a preset threshold; otherwise, judging that a sample does not exist in the field of view;

[0105] When a contour that meets the conditions is identified, in order to confirm whether it is a pseudo contour caused by a static impurity background, the image after a fixed time interval is continuously collected in the current field of view, and the adjacent acquisition frames are differentially operated to remove the static background. Then, the differential image is subjected to the same preprocessing operation to extract the body displacement part contour information of the motion sample from it. If there is contour information that meets the conditions, it can be confirmed that there is a motion sample in the search field of view, and then the subsequent tracking thread is entered. The conditions here include the position information, intensity information, contour area information, etc. of the pixels in the identified contour. If the conditions are not met, the above judgment is continued after the field of view is translated by moving the high-speed two-dimensional translation stage according to the pre-set search path.

[0106] When there is a real moving sample in the field of view, the movement of the sample will cause obvious differences in large-area intensity values ​​at certain positions in the previous and next frame images, thereby determining that the current search window contains a moving sample.

[0107] When the field of view is a pseudo contour caused by static impurity background interference, since it is relatively static in the acquired image, the differential image does not have an obvious area with a large intensity value, so the search continues along the current search path until the motion sample in the current well plate is found. By applying the above motion sample search strategy including the image difference algorithm, the actual background interference is effectively solved, the tracking progress is improved, and the search-track-search control logic ensures the overall automation performance of the algorithm.

[0108] (3-3) Calculating the sample centroid: taking the contour of the difference between the illumination light imaging difference image and the image in the current field of view identified by the sample contour recognition method of the illumination light imaging difference image as the sample contour, and calculating its centroid as the coordinate of the current sample centroid; when the contour of the difference between the illumination light imaging difference image and the image in the current field of view cannot be identified by the sample contour recognition method of the illumination light imaging difference image, repeating step (1);

[0109] To perform contour recognition, firstly, the contour information in the single-frame illumination light imaging image is identified and judged based on the traditional image processing algorithm. When the number of suspected contours that meet the conditions in the collected image is 0 due to the excessive movement of the sample, the algorithm uses the current field of view as the initial position of the center of the search path, and performs the above-mentioned motion sample search based on the global search strategy to ensure automatic tracking; when the number of suspected contours contained in the collected image is only one, the contour information at this time is assumed to be the contour information introduced by the actual motion sample, and the center position coordinates of the contour are calculated based on the contour information of the difference part between the current illumination light imaging differential image and the image in the current field of view, and input into the tracking thread; when the number of suspected contours contained in the collected image is at least two, the motion sample in the field of view may move to the vicinity of the impurity background, resulting in the identification of redundant false contours. The algorithm performs differential operations on adjacent frames by collecting another frame image after a fixed time interval under the same field of view. Similar to the differential principle in the aforementioned search module, the remaining static impurity background does not show obvious intensity changes after differentiation, while the head and tail parts of the moving sample produce a large area of ​​obvious intensity difference due to the motion displacement, which is manifested as a large local intensity value in the differential image.

[0110] Since the body displacement causes obvious changes in the intensity of the head and tail, the center of mass of the moving object in this case can be fitted by calculating the mean of the white pixel position coordinates in the result of differential image binarization, as shown in the following formula.

[0111]

[0112]

[0113] where x i ,y i Represents the position coordinates of white pixels, N represents the total number of white pixels, and x0, y0 are the approximate coordinates of the center of mass of the moving object.

[0114] When a sample is detected in the microscope field of view, the tracking thread is entered, and the coordinates of the sample platform at the center of the microscope field of view at the next moment are changed according to the position of the sample's center of mass, until the sample is no longer detected in the microscope field of view, and the search thread is entered.

[0115] The following steps are involved in tracking threads:

[0116] S1, taking the coordinates of the current sample centroid as the center of the microscope field of view at the next moment, so that the coordinates of the current sample centroid move to the center of the microscope field of view, and obtaining an illumination light imaging image in the current field of view for sample detection;

[0117] S2, sample detection: perform sample contour recognition on the illumination light imaging image in the current field of view obtained in step S1, and determine whether there is a sample in the field of view. If it is determined that there is a sample, update the current centroid coordinates of the sample and repeat step S1; otherwise, use the current sample centroid coordinates as the coordinates of the microscope field of view center on the sample platform at that moment, and initialize and enter the search thread;

[0118] The step of determining whether there is a sample in the field of view specifically includes the following sub-steps:

[0119] S2.1 Counting judgment: Get the number of identified sample contours. When the number of identified contours is 0, it is judged that there is no sample in the field of view; when the number of identified contours is equal to 1, it is judged that there is a sample and the centroid coordinates of the current sample are updated with the identified contours. Otherwise, if the number of identified contours is greater than or equal to 2, enter step S2.2;

[0120] S2.2 Differential judgment: wait for a preset time, keep the coordinates of the sample platform at the center of the microscope field of view unchanged, obtain an illumination light imaging differential image with a preset time difference from the image in the current field of view, and judge the difference between the illumination light imaging image in the current field of view and the illumination light imaging differential image. When the difference exceeds a preset threshold, there is a sample in the field of view, and go to step S2.3; otherwise, it is judged that there is no sample in the field of view;

[0121] S2.3 The contour of the difference between the illumination light imaging difference image and the illumination light imaging image in the current field of view is identified by the sample contour recognition method as the sample contour, and its center of mass is calculated as the coordinate of the current sample center of mass.

[0122] In the tracking thread, the center of mass coordinates of the moving object fed back in the contour recognition feedback module are first obtained, and the difference between the feedback center of mass coordinates and the center of view position coordinates is calculated based on the feedback center of mass coordinates. The high-speed two-dimensional translation stage is input through the translation stage drive function interface and driven to perform feedback movement to achieve the purpose of keeping the moving sample in the center of the view in real time, thereby realizing long-term tracking imaging.

[0123] The automatic search imaging method for freely moving samples provided in this embodiment can be expressed as the following pseudo code:

[0124] Main Program

[0125] Statement 1: The program executes the initialization module, including hardware interface initialization, tracking stage reset, search path setting, contour condition setting and acquisition parameter setting;

[0126] Statement 2: The search thread flag position is true, the tracking thread flag position is false, and the search thread is entered;

[0127] Search Threads:

[0128] Statement 1: The camera captures a single image in the current field of view;

[0129] Statement 2: Execute sample judgment function based on input image processing;

[0130] Statement 3: If it is determined that the sample is found: the search thread flag position is false, the tracking thread flag position is true, and the tracking thread is entered;

[0131] Otherwise, after receiving the next moment position, drive the translation stage to move to that position and execute statement 1

[0132] Tracking Threads:

[0133] Statement 1: The camera captures a single image in the current field of view;

[0134] Statement 2: Execute sample judgment function based on input image processing;

[0135] Statement 3: If it is determined that the sample is found, the stage is driven to move to the position after receiving the next moment position, and statement 1 is executed; otherwise:

[0136] The search thread flag position is true, the tracking thread flag position is false, and the search thread is entered;

[0137] Sample judgment function:

[0138] Statement 1: If the search thread flag is true:

[0139] When the number of contours that meet the search condition is 0, no sample is detected in the field of view, and the next search position in the search path is returned; if the number of contours is not 0, the differential discriminant function is executed.

[0140] Statement 2: If the search thread flag is false:

[0141] If the number of contours that meet the condition is 0, the motion sample may escape from the field of view, and the current position is returned to be the search center point, and the search thread is entered; if the number of contours that meet the condition is 1, the target position of the tracking stage at the next moment is calculated by the centroid coordinates of the contour, and the target position of the tracking stage at the next moment is returned; if the number of contours that meet the condition is greater than 1, the differential discriminant function is executed;

[0142] Difference Discriminant Function

[0143] Statement 1: Save the last frame of image acquisition and continue to acquire the next frame of image in the current field of view;

[0144] Statement 2: Execute the adjacent frame difference algorithm to differenciate the two adjacent images and determine the number of contours in the differenced image that meet the difference conditions. At this time:

[0145] If the search flag is true and the number of contours is 0, the contour originally detected in the field of view is a static impurity background, so the next search position in the search path is returned; if the number of contours is not 0, a moving sample is searched, the search thread flag position is false, the tracking thread flag position is true, and the tracking thread is entered.

[0146] If the search flag is false and the number of contours is 0, it is confirmed that the motion sample has escaped the field of view, the search thread flag position is true, the tracking thread flag position is false, and the search thread is entered; if the number of contours is not 0, the target position of the tracking stage at the next moment is calculated based on the differential image through the centroid coordinates of the contour envelope rectangular box and returned.

[0147] In this method, the illumination light imaging images collected continuously are as follows: Figure 3 As shown in , the process of contour recognition and differential discrimination is performed on one of the images, as Figure 4 shown.

[0148] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An automated search imaging method for freely moving samples, characterized in that: Includes search threads and tracking threads; When the sample is not detected in the microscope field image, the search thread is entered, and the coordinates of the sample platform at the center of the microscope field at the next moment are changed according to the calculation results of the search algorithm until the sample is detected in the microscope field image, and the tracking thread is entered; The search thread includes the following steps: (1) Determine the search field of view motion route: Based on the coordinates of the sample platform at the center of the microscope field of view at the current moment, a motion search algorithm is used to search and obtain the target coordinates of the sample platform at the center of the microscope field of view at the next moment; (2) Acquiring an image within the search field of view: moving the target coordinates of the sample platform obtained in step (1) to the center of the microscope field of view, and acquiring an illumination light imaging image within the current field of view for sample detection; (3) Sample detection: Perform sample contour recognition on the illumination light imaging image in the current field of view obtained in step (2) to determine whether there is a sample in the field of view. If it is determined that there is a sample, calculate the current centroid coordinates of the sample to initialize the tracking thread, exit the search thread, and enter the tracking thread; otherwise, repeat step (1); When a sample is detected in the microscope field of view, the tracking thread is entered, and the coordinates of the sample platform at the center of the microscope field of view at the next moment are changed according to the position of the sample's centroid, until the sample is no longer detected in the microscope field of view, and the search thread is entered; The tracking thread includes the following steps: S1, taking the coordinates of the current sample centroid as the center of the microscope field of view at the next moment, so that the coordinates of the current sample centroid move to the center of the microscope field of view, and obtaining an illumination light imaging image in the current field of view for sample detection; S2, sample detection: perform sample contour recognition on the illumination light imaging image in the current field of view obtained in step S1, and determine whether there is a sample in the field of view. If it is determined that there is a sample, update the current center of mass coordinates of the sample and repeat step S1; otherwise, use the current sample center of mass coordinates as the sample platform coordinates of the microscope field of view center at the next moment, and initialize to enter the search thread.

2. The method for automated search imaging of a freely moving sample as claimed in claim 1, characterized in that: The motion search algorithm is selected from a global search algorithm, a three-step search algorithm, a diamond search algorithm or a hexagonal search algorithm.

3. The automated search imaging method for a freely moving sample as claimed in claim 1, characterized in that: Step (3) of determining whether there is a sample in the field of view specifically includes the following sub-steps: (3-1) Counting judgment: Get the number of identified sample contours. When the number of identified contours is 0, it is judged that there is no sample in the field of view; when the number of identified contours is greater than or equal to 1, go to step (3-2); (3-2) Differential judgment: waiting for a preset time, keeping the coordinates of the sample platform at the center of the microscope field of view unchanged, obtaining an illumination light imaging differential image with a preset time difference from the illumination light imaging image in the current field of view, judging the degree of difference between the illumination light imaging image in the current field of view and the illumination light imaging differential image, and judging that a sample exists in the field of view when the degree of difference exceeds a preset threshold; Otherwise, it is judged that there is no sample in the field of view.

4. The method for automated search imaging of a freely moving sample as claimed in claim 1, characterized in that: The calculation of the centroid coordinates of the current sample in step (3) is specifically as follows: (3-3) Calculate the sample centroid: Use the sample contour recognition method to identify the contour of the difference between the illumination light imaging difference image and the image in the current field of view as the sample contour, and calculate its centroid as the coordinate of the current sample centroid.

5. The method for automated search imaging of a freely moving sample as claimed in claim 1, characterized in that: Step S2 of determining whether there is a sample in the field of view specifically includes the following sub-steps: S2.1 Counting judgment: Get the number of identified sample contours. When the number of identified contours is 0, it is judged that there is no sample in the field of view; when the number of identified contours is equal to 1, it is judged that there is a sample and the centroid coordinates of the current sample are updated with the identified contours. Otherwise, if the number of identified contours is greater than or equal to 2, enter step S2.2; S2.2 Differential judgment: wait for a preset time, keep the coordinates of the sample platform at the center of the microscope field of view unchanged, obtain an illumination light imaging differential image with a preset time difference from the image in the current field of view, and judge the degree of difference between the illumination light imaging image in the current field of view and the illumination light imaging differential image. When the degree of difference exceeds a preset threshold, there is a sample in the field of view, and proceed to step S2.3; otherwise, it is judged that there is no sample in the field of view.

6. The method for automated search imaging of a freely moving sample as claimed in claim 1, characterized in that: The updating of the current centroid coordinates of the sample in step S2 is specifically: S2.3 The contour of the difference between the illumination light imaging difference image and the illumination light imaging image in the current field of view is identified by the sample contour recognition method as the sample contour, and its center of mass is calculated as the coordinate of the current sample center of mass.

7. An automated search imaging system, characterized in that: Applying the automated search imaging method of a freely moving sample as described in any one of claims 1 to 6; It includes: an illumination light source, a fluorescence excitation light source, a behavior detection channel, a fluorescence detection channel, a lower computer, and a motion mechanism that drives the sample platform to move; The illumination light source and the fluorescence excitation light source are projected into the working range of the microscope; The behavior detection channel and the fluorescence detection channel collect illumination light imaging images and fluorescence images respectively, and output the images to the lower computer; The illumination light emitted by the illumination light source is imaged through the behavior detection channel to obtain an illumination light imaging image, which is used to collect a behavior image of the freely moving sample; The excitation light emitted by the fluorescence excitation light source, when a sample is present therein, excites the sample to generate fluorescence, and is imaged through the fluorescence detection channel to obtain a sample fluorescence image, which is used to collect the sample fluorescence image; The lower computer is used to run the automated search imaging method for freely moving samples as described in any one of claims 1 to 6, obtain the target coordinates of the sample platform that is located at the center of the microscope field of view at the next moment, and drive the motion mechanism to move the target coordinates of the sample platform to the center position of the microscope field of view.

8. The automated search imaging system of claim 7, wherein: The behavior detection channel and the fluorescence detection channel share the same high-magnification detection objective lens or respectively use detection objective lenses with different magnifications; The fluorescence detection channel can be single wavelength, dual wavelength or multi-wavelength detection.

9. The automated search imaging system according to claim 7, wherein: The fluorescence detector of the fluorescence detection channel is selected from CCD or CMOS with high sensitivity and high quantum efficiency; the camera of the behavior detection channel is an industrial camera or a scientific research camera.

10. The automated search imaging system according to claim 7, wherein: The motion mechanism is two-dimensional, one-dimensional or three-dimensional depending on the motion mode of the sample.

11. The automated search imaging system of claim 10, wherein: The motion mechanism is a high-speed two-dimensional translation stage, which can be a dual-axis brushless DC motor, an electric translation stage or a piezoelectric translation stage.

Images