Colony picking system based on optical manipulation technology, and picking method

Abstract

Provided is a colony picking system based on optical manipulation technology. The system comprises: a chip platform, which is provided with a chip placement unit for placing a chip thereon; a microplate platform, which is provided with a microplate placement unit for placing a microplate thereon; a photothermal module, which is used for emitting a photothermal light path, wherein there is one photothermal light path, which irradiates a photo-induced heating unit on one chamber of the chip, or there are a plurality of photothermal light paths, which irradiate, in parallel, photo-induced heating units on a plurality of chambers of the chip; an imaging module, which is used for identifying the state of colonies in the chambers of the chip within the current field of view; and a control module, which is configured to identify and calibrate a target colony in one chamber of the chip by means of the identification performed by the imaging module, control the photothermal module to perform photothermal heating on the photo-induced heating unit on the chamber where the identified and calibrated target colony is located, and adjust the position of the chip platform and / or the microplate platform, such that the target colony is transported from the chip to collection wells of the microplate.

Description

Bacterial colony picking system and picking method based on light manipulation technology

[0001] The present application claims priority to the Chinese patent application No. 2024111259136, filed on August 16, 2024, and entitled "A bacterial colony picking system and picking method based on light manipulation technology", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application belongs to the technical field of bacterial colony picking, and particularly relates to a bacterial colony picking system and picking method based on light manipulation technology. BACKGROUND

[0003] With the development of biotechnology, traditional manual picking of microbial colonies has gradually been replaced. Microbial colony picking instruments replace manual picking, which not only improves picking efficiency, but also greatly improves picking accuracy. Currently, microbial colony picking instruments on the market mainly use 96-needle picking heads, which move in X, Y and Z three orthogonal coordinate axes. First, based on machine vision technology, a camera is used to identify and locate microbial colonies in a culture dish, and then 96-needle picking heads pick the colonies one by one and inoculate them into an inoculation well plate. Finally, the 96 picking needles are cleaned and disinfected. For details, please refer to Chinese invention applications CN113604348A, CN108641903A and CN113667588A.

[0004] The main disadvantages of the above-mentioned scheme are as follows: 1. contamination problem, the metal picking needle is repeatedly used in the working process, even if there is a disinfection process, there is a risk of incomplete disinfection, which reduces the reliability and scientificity of the experimental results; 2. complex structure and high cost, the 96-needle picking head structure, the 96-needle picking head is a key component, which needs to realize high-precision movement in X and Y axes, and each of the 96 needles needs to move in the Z (up-down direction) direction. In addition, the 96 needles need to be designed with high temperature resistance and high rigidity, which makes the structure design complex and the cost high; 3. increased energy consumption, including high-temperature disinfection of 96 needles and other disinfection links, which consumes a lot of energy; 96-needle picking method, the source plate is a commonly used culture dish, however, to realize high-throughput picking, multiple culture dishes, even dozens of culture dishes, are needed to complete a picking experiment, which results in a large number of consumables and increased energy consumption. SUMMARY

[0005] In order to overcome some problems in the prior art, the present application provides a bacterial colony picking system and picking method based on light manipulation technology.

[0006] The first aspect of the present application provides a bacterial colony picking system based on light manipulation technology, comprising:

[0007] The chip platform is provided with a chip placement part for placing the arrayed chamber chip;

[0008] The well plate platform is provided with a well plate placement part for placing the well plate;

[0009] The photothermal module is used for emitting a photothermal light path;

[0010] The number of the photothermal light path is one and the light-induced heating part is irradiated on one chamber of the chip; or the number of the photothermal light path is multiple and the light-induced heating parts are irradiated on multiple chambers of the chip in parallel;

[0011] The imaging module is used for identifying the state of the colony in the chamber of the chip in the current field of view; and

[0012] The automatic collection control module (which can be referred to as the control module) is configured to identify and calibrate the target colony in the chamber of the chip through the identification of the imaging module, control the photothermal heating of the light-induced heating part on the chamber where the identified and calibrated target colony is located by the photothermal module, and adjust the position of the chip platform and / or the well plate platform so that the target colony is transported from the chip to the collection hole of the well plate.

[0013] In an embodiment, the horizontal height of the chip placement part is higher than the horizontal height of the well plate placement part.

[0014] In an embodiment, the picking system further comprises a fluorescence detection module for emitting a fluorescence light path; the photothermal module, the imaging module and the fluorescence detection module share the same microscope objective which is fixedly installed above the chip placement part and can be aligned with the chip.

[0015] In an embodiment, the picking system comprises a detection and identification module for detecting the colony in the chamber; the detection and identification module is at least one of a fluorescence detection module, a Raman detection module, a laser confocal detection module and a mass spectrometry detection module. Each detection module can emit detection light which passes through the microscope objective to the chip; the light excited by the target colony in the chamber of the chip passes through the microscope objective and the detection and identification module in reverse, passes through the relay lens to the image collector to obtain the chamber image.

[0016] In an embodiment, the photothermal module comprises a colony picking light source and a light beam shaping module; the colony picking light source is a laser light source or an LED light source; the light beam shaping module is used for shaping the light beam from the colony picking light source into single-beam or multi-beam and shrinking the light beam; and a photothermal light path capable of irradiating the light-induced heating part on the chip is obtained.

[0017] In an embodiment, the imaging module comprises a relay lens and an image collector; the photothermal module (single beam) comprises a laser, an optical beam expander system and a collimating lens; wherein,

[0018] The imaging light path of the chip collected by the microscope objective passes through the transmission of the dichroic mirror to the relay lens and is transmitted to the image collector;

[0019] The fluorescence detection module (or detection and identification module) can be arranged between the dichroic mirror and the relay lens;

[0020] The laser is used to emit a laser beam, which is expanded by the optical beam expander system, and then passes through the collimating lens to obtain parallel light, and finally passes through the reflection of the dichroic mirror to reach the microscope objective.

[0021] The control module comprises at least one processor; the image collector is a digital camera; the image collector and the laser are in communication with the control module; the image collector transmits the collected chamber image of the chip to the control module, the control module identifies and calibrates the target colony in the chamber image, the control module controls the opening of the laser and provides a laser beam, and the photothermal light path is formed through the photothermal module, which performs photothermal heating on the light-sensitive heating part on the chamber where the target colony is located through the microscope objective.

[0022] When the detection and identification module is not arranged, the chamber image collected by the image collector is bright field imaging; when the detection and identification module is arranged, the chamber image collected by the image collector is at least one of fluorescence imaging, Raman light imaging, laser imaging or mass spectrometry imaging. The control module can communicate with the detection and identification module to control the emission of detection light.

[0023] In an embodiment, the imaging module comprises a relay lens, an image collector; the photothermal module (multiple beams) comprises a laser, a half glass, an optical beam expander system, a collimating lens, a spatial light modulator, a quarter glass and a polarization beam splitter prism;

[0024] The imaging light path of the chip collected by the microscope objective passes through the transmission of the dichroic mirror to the relay lens and is transmitted to the image collector, to obtain a chamber image of the chip;

[0025] The fluorescence detection module (or detection and identification module) can be arranged between the dichroic mirror and the relay lens;

[0026] The laser is used to emit a laser beam, and the polarization direction of the light beam is adjusted by the rotation angle of the half-wave plate, and then the light beam is expanded by the optical beam expander, and then the parallel light is obtained by the collimating lens, and then the parallel light is transmitted to the polarizing beam splitter prism; the reflection of the polarizing beam splitter prism makes it enter the quarter-wave plate and the spatial light modulator in turn; the spatial light modulator modulates a multi-point phase light field by loading the phase image processed based on the chamber image collected by the image collector, and then forms an array of multiple optical-thermal light paths through the transmission of the quarter-wave plate and the polarizing beam splitter prism, and then the multiple optical-thermal light paths are reflected by the dichroic mirror and then enter the microscope objective.

[0027] The image collector, the laser, and the spatial light modulator are in communication with the control module (processor); the image collector transmits the collected chamber image about the chip to the control module, the control module identifies and calibrates the target colony in the chamber image, and obtains the phase image about the scatter point where the target colony is located; the control module controls the opening of the laser and provides the laser beam, and loads the phase image on the spatial light modulator to form multiple optical-thermal light paths through the optical-thermal module, and the multiple optical-thermal light paths respectively perform optical-thermal heating on the light-sensitive heating parts on the multiple chambers where the target colony is located through the microscope objective.

[0028] In an embodiment, the selection system further comprises a detection module for identifying whether the target colony is contained in the droplet discharged from the chip, and judging whether the end of the droplet discharged from the chip is aligned with the collection well of the well plate.

[0029] In an embodiment, the detection module comprises:

[0030] A visual camera is arranged below the well plate platform;

[0031] A second beam splitter prism is arranged above the visual camera;

[0032] An illumination light source and a sixth mirror are respectively arranged on the two sides of the second beam splitter prism; the illumination light source emits a light beam and divides the light beam into reflected light and transmitted light through the second beam splitter prism; the reflected light is used for illuminating the collection well of the well plate for judging the alignment state of the end of the droplet discharged from the chip with the collection well of the well plate; the transmitted light is used for chip illumination through the sixth mirror for imaging of the imaging module.

[0033] In an embodiment, the chip platform and the well plate platform move along the XY axis in the horizontal direction and move relatively in the vertical direction; the visual camera is configured to identify whether the target colony is contained in the droplet discharged from the end of the chip, and the relative movement of the chip platform and the well plate platform in the vertical direction is controlled to realize the collection of the target colony.

[0034] In a specific embodiment, the visual camera, the chip platform and the well plate platform are all in communication with the control module, the visual camera transmits the real-time captured images to the control module; when a droplet reaches the end of the chip, the control module determines whether the droplet is wrapped with the target colony by analyzing the images from the visual camera, and issues a droplet collection instruction when it is determined that the droplet is wrapped with the target colony; the control module further determines whether the end of the chip is aligned with the collection well of the well plate by analyzing the images from the visual camera according to the droplet collection instruction, and controls the chip platform and / or the well plate platform to move so as to be aligned when they are not aligned, so as to collect the droplet wrapped with the target colony in the collection well.

[0035] In an embodiment, the outlet of the chip is connected with an outlet tube; the outlet tube is vertically arranged at the outlet of the chip or fixedly arranged between the chip platform and the well plate platform and connected with the outlet of the chip through a flexible connecting tube. At this time, the end of the outlet tube is the end of the chip.

[0036] The second aspect of the present application provides a colony picking method based on light manipulation technology, which adopts the picking system of any one of the preceding embodiments, and the picking method comprises the following steps:

[0037] Placing the chip cultured with the target colony on the chip placement part of the chip platform; and aligning the micro objective lens with the chip.

[0038] Placing the well plate on the well plate placement part of the well plate platform.

[0039] The imaging module identifies the state of the colonies in each chamber of the chip within the current field of view of the micro objective lens.

[0040] The photothermal module emits one or more photothermal light paths capable of irradiating the light-sensitive heating part of one chamber of the chip or the light-sensitive heating parts of multiple chambers.

[0041] The control module identifies the target colony in the chamber of the chip through the identification of the imaging module, controls the photothermal module to perform photothermal heating on the light-sensitive heating part on the chamber where the identified target colony is located, and adjusts the positions of the chip platform and / or the well plate platform so as to transport the target colony from the chip to the collection well of the well plate.

[0042] The third aspect of the present application provides a colony picking method based on light manipulation technology, which adopts the picking system of any one of the preceding embodiments, and the picking method comprises: a synchronous mode and an asynchronous mode.

[0043] The synchronous mode is to obtain information of target colonies in the chip within the current field of view, plan a picking path of the target colonies according to a greedy algorithm, and collect the target colonies according to the planned path. For other chips within the field of view, the step is repeated until the picking of the target colonies in the whole chip is completed.

[0044] The asynchronous mode is to complete image acquisition of the whole chip, identify and calibrate all target colonies according to the image acquisition, plan a path of the target colonies according to a greedy algorithm, and complete the picking of the target colonies in sequence.

[0045] In an embodiment, the picking method further comprises a chip inclination correction method: image information of the chip after placement is collected and each chamber is identified, a data point column is formed by the center points of each chamber located on the same horizontal reference line to obtain corresponding two-dimensional array coordinates; the average slope Slope is solved by fitting the connection between adjacent arrays on each horizontal reference line, the Slope corresponds to positive and negative, K is solved according to the Slope, the K value is the total inclination distance of each chamber, and the chip platform is moved according to the K value, the K value is positive, and the chip platform is moved upward; the K value is negative, and the chip platform is moved downward.

[0046] In an embodiment, the method for the photothermal module to irradiate the light-sensitive heating part on the chamber of the chip is:

[0047] The single photothermal light path formed by the laser is irradiated on the chamber where the target colony is located by adjusting the position of the chip platform, and the target colony is guided out in a photothermal heating manner; or

[0048] The light beam formed by the laser is expanded and parallelized, and is made to enter a spatial light modulator. The spatial light modulator forms a plurality of photothermal light paths in the current field of view according to the chamber image (i.e. the chamber image collected by the image collector) where the target colony is located, and irradiates the plurality of photothermal light paths on each chamber where the target colony is located. Since the distances from each chamber in the current field of view to the outlet of the chip are different, all the target colonies are guided out in sequence in a photothermal heating manner.

[0049] Compared with the prior art, the application has the following beneficial effects:

[0050] The colony picking system or method provided by at least one embodiment of the application changes the traditional needle picking method for picking colonies, utilizes the structure of the array chamber chip and is based on the photothermal effect, realizes the contactless guiding of the target colonies in a photothermal manner, and can realize online culture of the colonies by utilizing the chamber characteristics in the structure of the array chamber chip itself, so as to ensure the success rate and efficiency of the colony picking.

[0051] The colony picking system or method provided by at least one embodiment of the present application further increases the corresponding detection module to detect whether the target colony is contained in the derived droplet, so as to ensure the accuracy of the picking process.

[0052] The colony picking method provided by at least one embodiment of the present application forms two modes of synchronous extraction and asynchronous extraction to adapt to different working conditions, thereby improving the efficiency of colony picking.

[0053] In the colony picking system or method provided by at least one embodiment of the present application, the picked light-heat module can realize the mode of single light-heat light path or multiple light-heat light paths, so as to realize single sequential picking or rapid multiple picking.

[0054] The colony picking system or method provided by at least one embodiment of the present application can greatly improve the sorting efficiency by avoiding the drawbacks of the traditional colony picking instrument. BRIEF DESCRIPTION OF DRAWINGS

[0055] FIG. 1 is a structural schematic diagram of a picking system according to one embodiment of the present application;

[0056] FIG. 2 is a structural schematic diagram of a multi-beam light path module according to one embodiment of the present application;

[0057] FIG. 3 is a structural schematic diagram of a detection module according to one embodiment of the present application;

[0058] FIG. 4 is a partial schematic diagram of an array type chamber chip according to one embodiment of the present application;

[0059] FIG. 5 is a schematic diagram of machine vision camera alignment imaging according to one embodiment of the present application;

[0060] FIG. 6 is a schematic diagram of machine vision camera droplet detection according to one embodiment of the present application;

[0061] FIG. 7 is a connection schematic diagram of a control module and other components according to one embodiment of the present application;

[0062] Wherein, 1 chip platform, 101 chip placement part; 2 chip, 201 chamber, 202 light-induced heating part, 203 flow channel, 204 outlet, 205 droplet, 206 outlet pipe, 207 vertical section, 208 end, 209 flexible connecting pipe; 3 hole plate platform, 301 hole plate placement part; 4 hole plate, 401 collection hole; 5 optical path module; 51 light-heat module, 511 colony picking light source, 512 beam shaping module; 52 imaging module, 5201 image collector, 5202 relay lens, 5203 fifth mirror; 53 fluorescence detection module, 531 light cube, 532 transmission device; 54 microscope objective, 55 dichroic mirror; 6 control module; 7 pumping module; 8 detection module, 81 visual camera, 82 second light splitting prism, 83 illumination light source, 84 sixth mirror, 86 collimation and light uniformity module, 87 telecentric lens; B1 light-heat optical path, B2 reflected light; B3 transmitted light. DETAILED DESCRIPTION

[0063] The technical solutions of the present application will be described in detail below in combination with specific embodiments. However, it should be understood that the elements, structures and features in one embodiment can also be beneficially combined into other embodiments without further description.

[0064] In the description of the present application, it should be understood that the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features.

[0065] In the description of the present application, it should be understood that the terms "up", "down", "bottom", "inner" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in Figure 1, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0066] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "connected", "connected" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be directly connected, or indirectly connected through an intermediate medium, or it can be connected inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0067] As shown in Figures 1-7, the first embodiment of the present application provides a colony picking system based on light manipulation technology (which can be simply referred to as picking system or system), which comprises:

[0068] A chip platform 1 is provided with a chip placement part 101 for placing an array chamber chip 2 (which can be referred to as chip 2);

[0069] A well plate platform 3 is provided with a well plate placement part 301 for placing a well plate 4;

[0070] A photothermal module 51 is used for emitting a photothermal light path B1; the number of photothermal light paths B1 is one and the photothermal light path B1 is irradiated on a light-induced heating part 202 in one chamber 201 in the chip 2, or the number of photothermal light paths B1 is multiple and the photothermal light paths B1 are irradiated on light-induced heating parts 202 in multiple chambers 201 in the chip 2 in parallel;

[0071] An imaging module 52 is used for identifying the state of a colony in a chamber 201 of the chip 2 in a current field of view;

[0072] An automatic collection control module 6 (which can be referred to as control module 6) is configured to identify and calibrate a target colony in a chamber 201 in the chip 2 through the identification of the imaging module 52, to control the photothermal module 51 to perform photothermal heating on a light-induced heating part 202 on the chamber 201 where the identified and calibrated target colony is located, and to adjust the position of the chip platform 1 and / or the well plate platform 3 so that the target colony is transported from the chip 2 to a collection well 401 in the well plate 4.

[0073] As described in FIG. 1 and FIG. 7, the selection system can also integrate a pumping module 7 for sample injection / liquid supply of the chip 2. The pumping module 7 can communicate with the control module 6; the control module 6 can control the start or stop of the fluid pump in the pumping module 7 to inject or stop injection (the injection object is a culture solution containing a colony), or to supply or stop supply (the supply object is generally an oil phase capable of wrapping a colony).

[0074] The structure of the arrayed chamber chip 2 is shown in FIG. 4, which is a conventional chip structure. Specifically, the chip 2 has one or more flow channels 203; a plurality of chambers 201 are arranged on both sides of the flow channel 203, and the light-responding heating part 202 is arranged at the chamber 201 (i.e., the laser point position at the top of the chamber in CN112871227B); of course, the front end of the flow channel 203 is connected to a sample inlet (not shown), and the rear end is connected to a sample outlet (also referred to as a guide outlet 204), which is well known to those skilled in the art; the heating part 202 is irradiated by the light-thermal light path B1, a bubble is formed in the chamber 201, the target colony is squeezed out of the chamber 201 into the flow channel 203, is wrapped by the oil phase in the flow channel, and moves together with the oil phase in the flow, and exits the chip 2 from the guide outlet 204; in addition, the oil phase in the flow channel 203 also exits the chip 2 from the guide outlet 204; for other structures or more specific understanding of the chip 2, please refer to CN112871227B, which is incorporated herein by reference, and therefore will not be described in more detail. The selection system is a matching system developed and designed based on the chip structure. The innovation of the present application breaks the needle-picking mode of colony picking, injects the colony sample into the chamber 201 of the chip 2, realizes non-contact picking by light-thermal, and completes the picking of the target colony.

[0075] In an embodiment, the horizontal height of the chip placement part 101 is higher than the horizontal height of the well plate placement part 301, so that the chip 2 is higher than the well plate 4, so that the oil phase droplet 205 wrapped with the target colony from the chip 2 can drop under the action of gravity and be collected in the collection hole 401 of the well plate 4. Chip clamps and well plate clamps are respectively arranged on the chip placement part 101 and the well plate placement part 301 to respectively clamp and fix the chip 2 and the well plate 4. By designing the height difference between the upper and lower positions, the outflow of the droplet 205 is facilitated, and the compactness of the equipment structure is also ensured.

[0076] In an embodiment, as shown in FIG. 2, the selection system can also include a fluorescence detection module 53 for emitting and forming a fluorescence light path. The light-thermal module 51, the imaging module 52, and the fluorescence detection module 53 can share the same microscope objective 54 and be fixedly installed above the chip placement part 101 to be able to align the chip 2. The light-thermal module 51, the imaging module 52, the fluorescence detection module 53, and the microscope objective 54 together constitute a light path module 5, wherein the microscope objective 54 can belong to a certain module or can be independent of the module according to the working condition.

[0077] A plurality of fluorescence light cubes 531 can be arranged in the fluorescence detection module 53 for different switching.

[0078] Specifically, as shown in FIG. 2, the fluorescence detection module 53 includes a plurality of arranged optical cubes 531 and a transmission device 532; the transmission device 532 is configured to be capable of switching the optical cubes 531 and switching the bright field imaging and the fluorescence imaging. For example, three optical cubes 531 arranged in up and down are shown in FIG. 2, and the light emitted by different optical cubes is different, including different parameters such as color and wavelength. Through the action of the transmission device 532, different optical cubes can be switched to emit light, which is used for fluorescence imaging; in addition, it can also be switched to no optical cube (at this time the fluorescence detection module 53 does not work), which provides a channel for bright field imaging. The transmission device 532 can use the transmission mode commonly used in the mechanical field such as belt transmission and gear transmission to realize this function.

[0079] In an embodiment, the fluorescence detection module 53 is located between the microscope objective 54 and the image collector 5201. The fluorescence emitted by the optical cube of the fluorescence detection module 53 passes through the reflection of the fifth mirror 5203 from the first side (corresponding to the left side in FIG. 2) of the fluorescence detection module 53 to the microscope objective 54, and irradiates in the chamber 201 of the chip 2; the fluorescence excited by the target colony passes through the microscope objective 54, the optical cube 531 in reverse, and then passes through the relay lens 5202 from the second side opposite to the first side of the fluorescence detection module 53 to the image collector 5201, to obtain the fluorescence imaging (chamber image). Through the fluorescence image (chamber image), it can be judged whether there is a target colony in each chamber 201; specifically, the image collector 5201 communicates with the control module 6, and can transmit the collected fluorescence image to the control module 6, and the control module 6 can identify and calibrate the target colony in the chamber 201 of the chip 2, and further control the photo-thermal module 51 to perform photo-thermal heating on the light-sensitive heating part 202 in the chamber 201 where the identified and calibrated target colony is located.

[0080] The fluorescence detection module 53 is a detection and recognition module for detecting the colonies in the chamber 201. The detection and recognition module can be at least one of a Raman detection module, a laser confocal detection module, and a mass spectrum detection module in addition to the fluorescence detection module 53. The working mode of the detection and recognition module is similar to that of the fluorescence detection module 53. The detection and recognition module emits detection light (such as Raman light, laser light, mass spectrum light, etc.) from a first side to the objective lens 54, and the light is irradiated on the chamber 201 of the chip 2. The light excited by the target colony is reversely passed through the objective lens 54 and the detection and recognition module, and is passed through the relay lens 5202 from a second side opposite to the first side of the detection and recognition module to the image collector 5201 to obtain a chamber image (such as a Raman image, a laser image, a mass spectrum image (spectrum), etc.). The target colony in the chamber image is further recognized and calibrated by the control module 6, and the corresponding chamber 201 is subjected to photothermal heating. Different detection and recognition modules are used for different colonies, so that the selection system can be suitable for the selection of various colonies. For the convenience of understanding, the fluorescence imaging of the fluorescence detection module 53 is mainly taken as an example for description in the embodiment.

[0081] In an embodiment, the detection and recognition module can not be provided. At this time, a first light source for illumination can be provided below the objective lens 54, and the chip 2 is located between the objective lens 54 and the first light source. At this time, the light from the first light source is passed through the chip 2, the objective lens 54, the fifth mirror 5203, and the relay lens 5202 to the image collector 5201 to realize bright field imaging and obtain a chamber image. For different colonies, different imaging can be used, which can be bright field imaging, fluorescence imaging after adding a detection and recognition module, Raman imaging, laser imaging, mass spectrum imaging, etc. Chamber images about the colonies in the chamber can be obtained.

[0082] In order to match the multiple photothermal light paths B1 of the photothermal module 51, the control module 6 can also make a scatter plot of the light-sensitive heating part 202 of each chamber 201 in which the target colony is located based on the chamber image collected by the image collector 5201 to obtain a phase image about each scatter point.

[0083] As shown in FIG. 2, the photothermal module 51 includes a colony picking light source 511 and a beam shaping module 512. The colony picking light source 511 is a laser light source or an LED light source. The beam shaping module 512 is used for shaping the light from the light source 511 into a single beam or multiple beams and shrinking the light beam, so as to obtain a photothermal light path B1 that can irradiate on the light-sensitive heating part 202 of the chip 2.

[0084] There are two structural forms of the photothermal module 51:

[0085] The first is a relatively conventional optical path setting mode. Laser 5101 generates laser light, and then a single-beam photothermal light path B1 is formed.

[0086] The second photothermal module 51 is shown in FIG. 2. Laser 5101 emits laser light with a near-infrared wavelength (approximately 785-1100 nm), and the laser light passes through or is blocked by optical shutter 5102. The laser light continues to propagate to half-wave plate 5103, and the polarization direction of the laser light can be changed by rotating half-wave plate 5103. The laser light with the changed polarization direction reaches optical expander 5105 (which can be an expander) after passing through first mirror 5104, and the laser light is expanded. The expanded laser light passes through collimating lens 5106 to obtain parallel light. At this point, the expanded laser light with the changed polarization direction is obtained, and the expansion ratio is determined by the cooperation of optical expander 5105 and collimating lens 5106. The expanded parallel light is reflected by second mirror 5107 into polarization beam splitter prism 5110. The polarization beam splitter prism 5110 cooperates with half-wave plate 5103 to achieve the transmission and reflection ratio of the laser light, and the maximum reflection of the laser light in the polarization beam splitter prism 5110 can be achieved by rotating and adjusting the angle of half-wave plate 5103. Specifically, by adjusting half-wave plate 5103, it can be adjusted whether the light from second mirror 5107 is directly transmitted at the polarization beam splitter prism 5110 (represented as moving to the left in FIG. 2) or reflected to spatial light modulator 5108 (represented as moving downward in FIG. 2). In addition, quarter-wave plate 5109 can be arranged between the polarization beam splitter prism 5110 and the modulator 5108. The light reflected from the polarization beam splitter prism 5110 reaches the modulator 5108 through the quarter-wave plate 5109. The quarter-wave plate 5109 can adjust the contrast of the light emitted by the modulator 5108. The spatial light modulator 5108 loads a phase image of a multi-spot pattern (i.e., a phase image about the target colony based on the chamber image of the image collector 5201, corresponding to the chamber where the target colony is located). The laser light emitted by the modulator 5108 is a multi-spot phase light field, which corresponds to a plurality of photothermal light paths B1 that need to be corresponded. The multi-spot phase light field passes through the polarization beam splitter prism 5110 again (and passes through the quarter-wave plate 5109 again), and the transmitted light beam is the required light beam. The transmitted light beam enters the 4f system through third mirror 5111. The 4f system is composed of first lens 5112 and second lens 5114, and fourth mirror 5113 can be arranged between the two lenses to change the direction of the light. The 4f system can achieve beam shrinking to obtain the plurality of photothermal light paths B1. The shrinking ratio is determined by the entrance pupil of the microscope objective 54. The shrunk light beam (i.e., the photothermal light path B1) can be introduced into the imaging module 52 through the dichroic mirror 55.

[0087] Specifically, the multiple photothermal light paths B1 (achieved by modulator 5108) from the photothermal module 51 can reach the microscope objective 54 and further irradiate on the chip 2 below the microscope objective 54, respectively irradiate on the light-responding heating parts 202 of the multiple target colony-located chambers of the chip 2, in a one-to-one irradiation relationship, so as to cause the photothermal pyrogenation of each target colony-located chamber 201, and make it be squeezed out of the chamber 201, and the target colony is wrapped by the oil phase in the flow channel 203, enters the flow channel 203, and is picked out.

[0088] The first and second photothermal modules 51 both heat the light-responding heating parts 202 of the target colony-located chambers based on the chamber images obtained by the imaging module 52; but the difference between them is that the first one mainly includes the laser 5101, the shutter 5102, the first mirror 5104, the beam expander system 5105, the collimating lens 5106 and the second mirror 5107, and other optical elements can be omitted. In the light path of the laser 5101 to the second mirror 5107, except that there is no half glass 5103, the light path is the same as the second photothermal module. In addition, in the first or second photothermal module 51, each mirror is mainly used to change the direction of light and does not change the properties of light (such as intensity, wavelength, etc.), so the number of mirrors can be increased or decreased according to actual needs.

[0089] The imaging module 52 is mainly used for microscopic imaging, specifically including bright field imaging and fluorescence imaging, both of which are chamber image imaging, as described above. The fluorescence imaging needs to be implemented in cooperation with the fluorescence detection module 53, and also mainly includes the fifth mirror 5203, the microscope objective 54, the relay lens 5202 and the image collector 5201. The image collector 5201 can be selected as a digital camera; in addition, different cameras can be used according to actual detection and identification modules, for example, when the detection and identification module is the fluorescence detection module 53, a fluorescence camera can be used.

[0090] A dichroic mirror 55 can be arranged between the photothermal module 51, the microscope objective 54 and the fluorescence detection module 53 (or detection and identification module). On the one hand, the fluorescence from the fluorescence detection module 53 can pass through the dichroic mirror 55, pass through the fifth mirror 5203 to reach the microscope objective 54, and then irradiate on the chip 2; the light excited by the target colony passes through the microscope objective 54, the fifth mirror 5203, passes through the dichroic mirror 55, passes through the fluorescence detection module 53, and reaches the image collector 5201 through the relay lens 5202 to obtain a fluorescence image (chamber image) for identifying the target colony; on the other hand, the photothermal light path B1 from the photothermal module 51 can be reflected by the dichroic mirror 55 and pass through the fifth mirror 5203 to reach the microscope objective 54, and then irradiate on the chip 2 to realize the selection of the target colony. That is, the target colony is first identified and calibrated by the detection and identification module and the imaging module 52, and then selected by the photothermal module 51 based on the identification and calibration to make it flow out of the chip 2.

[0091] The fluorescence detection module 53 is arranged between the dichroic mirror 55 and the relay lens 5202. The dichroic mirror 55 is high-reflective and low-pass, and can reflect the long-wavelength light beam (photothermal light path B1, wavelength about 785-1100 nm) from the photothermal module 51 and transmit the short-wavelength light (wavelength about 300-785 nm) from the fluorescence detection module 53; the photothermal light path B1 is reflected into the fifth mirror 5203, and then into the microscope objective 54, and the microscope objective 54 focuses the photothermal light path on the chip 2. The dichroic mirror 55 only reflects the light beam of the photothermal module 51, and the light beams of the fluorescence imaging and the bright field imaging can pass through.

[0092] As described above, the imaging module 52 includes the first light source, the microscope objective 54, the fifth mirror 5203, the dichroic mirror 55, the relay lens 5202 and the image collector 5201 whether for bright field imaging or fluorescence imaging; when fluorescence imaging, the fluorescence detection module 53 is added between the dichroic mirror 55 and the relay lens 5202.

[0093] In some embodiments, the selection system further includes a detection module 8 configured to identify whether the target colony is contained in the droplet 205 discharged from the chip 2, and to determine whether the end 208 of the droplet 205 discharged from the chip 2 is aligned with the collection well 401 of the well plate 4.

[0094] Specifically, as shown in FIG. 3, the detection module 8 includes:

[0095] A visual camera 81 arranged below the well plate platform 3;

[0096] A second light splitting prism 82 arranged above the visual camera 81;

[0097] The illumination light source 83 and the sixth mirror 84 are respectively arranged on the opposite sides of the second light splitting prism 82; the illumination light source 83 emits a light beam and splits the light beam into reflected light B2 (shown as upward in FIG. 3) and transmitted light B3 (shown as leftward in FIG. 3) through the second light splitting prism 82. The reflected light B2 is used for illuminating the collection hole 401 of the well plate 4 for judging the alignment state of the end 208 of the chip 2 discharging the droplet 205 with the collection hole 401; whether the droplet 205 wraps the target colony can also be judged. The transmitted light B3 is used for illuminating the chip 2 for imaging through the sixth mirror 84. The transmitted light B3 enters from the lower side of the chip 2, which is used as the first light source for illuminating the chip when bright field imaging; the transmitted light B3 is also used as the light source for illuminating when fluorescence imaging and the photothermal light path B1 of the photothermal module 51 reaches the chip 2; the transmitted light B3 can also be used when imaging the chamber image.

[0098] The chip platform 1 and the well plate platform 3 are configured to move along the XY axis in the horizontal direction and relatively move in the vertical direction Z; X, Y and Z are mutually perpendicular coordinate systems. The chip platform 1 and the well plate platform 3 can select the three-axis displacement platform, the cross slide, various electric translation stages capable of realizing the related functions and the like in the prior art, which is well known to those skilled in the art. The vision camera 81 is configured to identify whether the droplet 205 discharged from the end 208 contains the target colony; the chip platform 1 and the well plate platform 3 are configured to relatively move in the vertical direction Z to realize the collection of the target colony.

[0099] The vision camera 81, the chip platform 1 and the well plate platform 3 are in communication with the control module 6; wherein the vision camera 81 transmits the images collected in real time to the control module 6; before the well plate 4 collects the droplet 205, the control module 6 judges whether the end 208 of the chip 2 is aligned with the collection hole 401 of the well plate 4 by analyzing the images from the vision camera 81; when there is no alignment, the chip platform 1 and / or the well plate platform 3 are controlled to move so as to be aligned; when the droplet 205 reaches the end 208 of the chip 2, the control module 6 judges whether the droplet 205 wraps the target colony by analyzing the images from the vision camera 81; when it is judged that the droplet wraps the target colony, the well plate platform 3 is moved to collect the droplet 205 in the collection hole 401.

[0100] In addition, the collection hole 401 collecting the droplet wrapping the target colony can also be marked, which facilitates quickly finding the corresponding collection hole 401 in subsequent work.

[0101] Chip export pipe 206 is used to connect with the export port 204 of the chip 2. The export pipe 206 has a vertical section 207 arranged along the Z axis, and the end of the vertical section 207 is the end 208 of the chip 2 for discharging the droplet 205. The vertical section 207 can be directly vertically arranged at the export port 204 or can be fixed between the chip platform 1 and the hole plate platform 3 and connected with the export port 204 of the chip through a flexible connecting pipe 209. The flexible connecting pipe 209 refers to that the connecting pipe 209 can be deformed, is made of relatively soft material, is easy to bend, and is convenient to move.

[0102] In the direct vertical fixing mode, the export pipe 206 follows the chip 2, so it is necessary to continuously adjust the collection hole 401 of the corresponding hole plate 4 according to the position of the export pipe 206. In the fixed arrangement mode between the two, the position of the export pipe 206 is fixed, and the displacement of the flexible connecting pipe 209 can compensate for the displacement of the chip platform 1 during movement, so that the adjustment is more convenient.

[0103] The main functions of the detection module 8 are: 1. detecting the coaxiality of the end 208 of the export pipe and any initial collection hole 401 on the hole plate 4; 2. detecting whether the target colony is wrapped in the oil phase droplet; 3. integrated design: this part not only has a detection function, but also provides an illumination function, illuminates the consumable chip and detects the target, that is, provides the transmitted light B3 to the chip 2.

[0104] The illumination light source 83 can be a white light LED lamp emitting white light; the collimating and uniform light module 86 light source becomes a parallel and uniform light beam; the collimating and uniform light module 86 can use a convex mirror; the light beam continues to propagate to the second light splitting prism 82, which divides the light beam into reflected light B2 and transmitted light B3; the second light splitting prism 82 has a transmission-reflection ratio of 8:2 (or other ratios); wherein the transmitted light B3 is reflected into the consumable chip 2 through the sixth reflecting mirror 84 for illumination, coaxial with the microscope objective 54; the microscope objective 54 forms a bright field image (when there is no fluorescence imaging). The reflected light B2 is used to illuminate the collection hole 401 (such as A1 hole) of the hole plate 4, the droplet 205, and the end 208 of the chip export pipe. The hole plate 4 is installed on a three-axis displacement platform (hole plate platform 3), and the movement of the three-axis displacement platform realizes the up-down, left-right, and forward-backward movement of the hole plate 4. Then the telecentric lens 87 images. The telecentric lens 87 has the characteristics of high resolution, ultra-wide depth of field, ultra-low distortion, and parallel light design, and the image is displayed on the machine vision camera 81 for detection requirements.

[0105] Taking the collection hole 401 as an example: the imaging effect of the end of the guide tube (i.e., the end 208 of the chip 2 discharging the droplet 205) and the A1 hole site of the hole plate 4 on the machine vision camera 81 is shown in FIG. 5, the black dot is the center position (i.e., the target position) of the A1 hole site, and the gray dot is the position (i.e., the actual position) of the end of the chip guide tube. It is detected that the target position and the actual position are inconsistent, and the position difference can be calculated. The specific calculation method is: assuming that the black dot position is (x0, y0), and the actual position is (x, y), then Δx = x - x0; Δy = y - y0, so the XY axis of the hole plate platform 3 moves according to the difference, so that the target position coincides with the actual position. The coaxial / center of the A1 hole site and the end of the guide tube is completed, so that the droplet 205 discharged from the end 208 of the chip can accurately and accurately drop into the A1 hole site of the hole plate.

[0106] As shown in FIG. 6, the droplet 205 discharged from the end 208 of the chip includes two types: a droplet without wrapping the target colony and a droplet wrapping the target colony; the droplet without wrapping the target colony is generally an oil phase buffer, and the droplet wrapping the target colony is an oil phase buffer wrapped with a target colony. The imaging of the two types of droplets in the machine vision camera 81 is also different. When the imaging signal is the droplet wrapping the target colony, the control module 6 gives a feedback signal to start the droplet collection instruction, that is, to control the hole plate 4 to perform a moving action, so that the droplet 205 drops into the collection hole 401.

[0107] The control module 6 includes at least one processor, for example, a central processing unit (CPU), or a specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present application. The above at least some modules can be functional modules or program modules, which can be implemented by software or hardware. For the modules implemented by hardware, each module can be located in the same processor, or can be located in different processors in any combination. The selection system can also include a memory and at least one program, wherein the at least one program is stored in the memory and is configured to be executed by the at least one processor to implement the control functions or instructions related to the processor described above, which is well known to those skilled in the art.

[0108] Based on the system in the above embodiment, the second embodiment of the present application provides a colony selection method based on light manipulation technology, including the following steps:

[0109] Placing the chip 2 cultured with the target colony on the chip placement part 101 of the chip platform 1;

[0110] Placing the hole plate 4 on the hole plate placement part 301 of the hole plate platform 3;

[0111] The imaging module 52 identifies the state of the colonies in each chamber 201 of the chip within the current field of view of the microscope objective 54;

[0112] The photo-thermal module 51 emits one or more photo-thermal light paths capable of irradiating the light-sensitive heating part 202 of a chamber of the chip, or the light-sensitive heating parts 202 of multiple chambers;

[0113] The control module 6 identifies the target colonies in the chambers 201 of the chip 2 through the identification of the imaging module 52, controls the photo-thermal module 51 to perform photo-thermal heating on the light-sensitive heating part 202 on the chamber 201 where the identified target colonies are located, and adjusts the position of the chip platform 1 and / or the well plate platform 3 to transport the target colonies from the chip 2 to the collection well 401 of the well plate 4.

[0114] Based on the system in the above embodiment, the third embodiment of the present application provides another colony picking method based on light manipulation technology, which includes two modes, namely, synchronous mode and asynchronous mode.

[0115] The synchronous mode is to obtain the information of the target colonies in the array chamber chip 2 within the current field of view, plan the picking path of the target colonies according to the greedy algorithm, and collect according to the planned path; move to the next field of view of the microscope objective, and collect according to the same method; until all the target colonies in the chip 2 are picked.

[0116] The asynchronous mode is to complete the image acquisition of the entire array chamber chip 2, identify all the target colonies according to the image acquisition, plan the path of all the identified target colonies according to the greedy algorithm, and complete the sequential picking.

[0117] In the picking, it is also possible to calibrate the position of the chip 2. When it is placed on the chip platform 1, it cannot be ensured to be in a horizontal state, and it can be in an inclined state in the microscopic field, and thus a corresponding chip inclination correction method is provided. Specifically, the image collector 5201 collects the image of the placed chip 2 and identifies each chamber 201 thereof, forms a data point column through the center points of each chamber 201 on the same horizontal reference line, and obtains the corresponding two-dimensional array coordinates; according to the connection between adjacent arrays on each horizontal reference line, the average slope Slope is fitted and solved, Slope corresponds to positive and negative, K is solved according to Slope, K value is the total inclination distance of each chamber, and the chip platform 1 is moved according to the K value, K value is positive, and the chip platform 1 is moved upward; K value is negative, and the chip platform 1 is moved downward.

[0118] In combination with the photo-thermal module 51 of the two different structural forms described above, there are two corresponding heating methods.

[0119] First, the fluorescent detection module 53 and the imaging module 52 work together as a detection recognition module, and the image collector 5201 obtains the chamber image (bright field image, fluorescent image, Raman image, etc.) containing the target colony. Then:

[0120] Single photothermal light path: by adjusting the position of the chip platform 1, the single photothermal light path formed by the laser 5101 is irradiated on the chamber containing the target colony, and the target colony is guided out in a photothermal heating manner.

[0121] More specifically: the first photothermal module is used; the light spot presented by the single-beam photothermal light path B1 is located at the center of the field of view of the microscope objective 54; based on the identification and calibration of each target colony in the collected chamber image by the control module 6, the control module 6 controls the movement of the chip platform 1, which drives the movement of the chip 2, so that the light-responding heating part 202 of the chamber where the single target colony is located moves to the light spot; the shutter 5102 of the photothermal module 51 is opened, and the photothermal light path B1 reaches the light-responding heating part 202, so that the single target colony is squeezed out of the chamber 201 into the flow channel 203 to form a droplet 205 and flow out of the chip 2. Then move the light-responding heating part 202 of the chamber where the next target colony is located to the light spot, so that the photothermal light path B1 reaches the light-responding heating part 202 and squeezes the target colony into the flow channel 203 to form a droplet 205 and flow out of the chip 2. Process in turn until all target colonies are selected.

[0122] Multiple photothermal light paths: by expanding and parallelizing the beam formed by the laser 5101 and making it enter the modulator 5108, the modulator 5108 forms multiple photothermal light paths in the current field of view range according to the chamber position image containing the target colony in the current field of view range and irradiates on each chamber containing the target colony. Due to the different distances of each chamber in the current field of view range from the guide outlet 204, all target colonies are sequentially guided out in a photothermal heating manner.

[0123] More specifically: when the second photothermal module is used; based on the chamber image collected by the image collector 5201, the control module 6 makes a scatter plot of the light-responding heating part 202 at the top of each chamber where the target colony is located, and obtains the phase image of each scatter point; load the phase image to the modulator 5108, modulate the phase light field of multiple light spots, and finally obtain multiple photothermal light paths B1, which can respectively reach the light-responding heating part 202 of each chamber where the target colony is located, so that each target colony is squeezed out of the chamber into the flow channel 203 and wrapped by the oil phase, and flows out of the chip 2. Compared with the first photothermal module, the processing efficiency of the second photothermal module is higher.

[0124] The working principle is: light and matter interaction light-induced heating, i.e. the thermal effect of light, is the phenomenon that local heat is generated on the surface of the object after the light beam is irradiated on the surface of the object. When the focused near-infrared light is irradiated in the microcavity 201 of the chip 2, local heat is generated, the liquid in the cavity 201 is expanded, the liquid droplet is extruded out of the flow channel 203 outside the cavity, and the flow channel 203 has flowing liquid (oil phase), so that the liquid droplet wrapped with the bacterial colony is flushed out to the collection position, and the collection device is started to collect the bacterial colony liquid droplet.

[0125] The whole workflow is introduced as follows:

[0126] I. Sample pretreatment; mainly including two links: sample introduction link and culture link.

[0127] 1. Sample introduction link: this link includes sample spotting, inspection, sealing, and mainly realizes that the bacterial colony sample enters the microfluidic chip 2 and is distributed into each cavity 201.

[0128] 2. Culture link: place the chip 2 in the incubator, pay attention to keep the humidity of the chip 2, and prevent the evaporation of the liquid in the chip 2. The temperature of the incubator is confirmed according to the type of the sample.

[0129] II. Preparation before picking up; the main steps are as follows:

[0130] 1.1. Place the chip 2: connect the pipeline, place it in the chip clamp, and place it on the chip platform 1;

[0131] 1.2. Oil feeding: open the chip power source (pumping module 7), and feed the oil phase. The oil phase feeding time is about 10-60 min;

[0132] 1.3. Positioning: manually / automatically move to the upper left corner of the chip imaging, manually / automatically adjust the focal length to the clear image, set the chip three-axis platform as the zero point position at this time; adjust the moving hole plate three-axis position to make the chip capillary and the initial hole of the hole plate coaxial under the observation of the human eye, and set the hole plate three-axis to make it the zero point position. When the coaxial setting is performed, the automatic recognition and adjustment of the foregoing detection module 8 can also be used to perform automatic coaxial adjustment.

[0133] III. Detection / picking up

[0134] The detection can be performed in two modes of field and fluorescence. The specific selection can be made according to the requirements. The selection mode refers to the setting of the foregoing fluorescence detection module 53 and imaging module 52.

[0135] Due to the influence of the microscopic imaging field of view, the chip needs to be imaged multiple times to complete the image acquisition of all the cavities. Each image acquisition is one screen. The chip platform 1 is moved to realize the switching of each screen, so the detection / picking up step is divided into two modes: synchronous mode and asynchronous mode.

[0136] (I) synchronous mode: image acquisition, target recognition, picking in the same image acquisition screen, that is, image recognition and target picking are completed simultaneously; the specific steps are as follows:

[0137] 1. Bright field

[0138] 1.1 Identify the initial position; initial screen identification, based on the preparation before picking -> positioning -> initial screen setting, identify the initial screen based on AI technology, and the specific identification algorithm is as follows (taking the upper left corner of the image as an example):

[0139] S1: image acquisition;

[0140] S2: image input, the collected image is sent to the model (the model is a model established in the early stage);

[0141] S3: image discrimination, based on the model to discriminate whether the image is an image with a chamber;

[0142] S4-1: if yes, continue to move the chip left to the next screen, repeat steps S1-S3;

[0143] S4-2: if not, return to the previous screen;

[0144] S5: boundary correction, set the Padding value, so that the leftmost chamber edge and the image left boundary are Padding, and the chambers are symmetrically distributed in the image frame.

[0145] Similarly, the upper and lower position recognition is completed, and the initial screen recognition is finally realized. The platform moves to the initial screen position.

[0146] 1.2 automatic focusing / correction; image automatic focusing and chip inclination correction are performed, and the relevant data of the current screen are recorded, including position information, focusing information and correction parameters, and the specific focusing / correction method is as follows:

[0147] Automatic focusing (climbing algorithm)

[0148] S1: image acquisition, set the Z-axis position at this time as the zero position;

[0149] S2: image input, the collected image is sent to the OpenCvSharp algorithm;

[0150] S3: image gradient value calculation, using Tenengrad gradient method: Sobel operator, calculate the image gradient value;

[0151] S4: image set acquisition, move the image set up and down at the Z-axis zero position with a step interval, calculate the image gradient value, draw the curve of Z-axis position and gradient value, and find the peak position;

[0152] S5: Image focusing, the platform moves to the highest point, and the Z-axis position at this time is the in-focus position of the image, achieving image focusing.

[0153] Chip inclination correction:

[0154] S1: Image acquisition, acquire the focused image;

[0155] S2: Image input, the image is sent to the trained model;

[0156] S3: Data point generation, identify the chamber, and then identify the center point of the chamber to form a data point column, each point has a coordinate (X, Y), forming a two-dimensional array;

[0157] S4: K value solving, each row array is connected and fitted to solve the average slope Slope, Slope corresponds to positive and negative, K is solved according to the slope Slope, K value is the overall inclination distance of the chamber;

[0158] S5: Correct the image according to the K value solved in step S4, move the chip platform 1, K value is positive up; K value is negative down.

[0159] Complete automatic focusing and chip inclination correction, and record the current screen position information, focusing information and correction parameters.

[0160] 1.3 Image acquisition; the image collector 5201 starts the image shooting function and shoots the current screen image.

[0161] 1.4 Image recognition

[0162] 1.4.1 ID information allocation, identify the outer contour of each chamber in the picture, mark the identified chamber with a frame diagram, and record the position information of each chamber.

[0163] S1: Image acquisition, acquire the focused image;

[0164] S2: Image input, the image is sent to the trained model;

[0165] S3: Data point generation, identify the chamber, and then identify the center point of the chamber to form a data point column, each point has a coordinate (X, Y);

[0166] S4: Minimum point traversal, traverse the minimum Y value Min of the center point,

[0167] S5: Cut line drawing, draw a cut straight line with Min±Step respectively;

[0168] S6: Same row discrimination, traverse all center points, if Y is in the range of Min±Step, it is in the same row, and is sorted by X value from small to large, and is set as the first row (odd row);

[0169] S7: Repeat the next row traversal, set as the second row (even row), and sort with the previous row by X value from large to small:

[0170] S8: Assign the data point sorting information to each corresponding chamber, and complete the ID assignment of the chamber.

[0171] At this time, the ID information assignment of each chamber is realized, including pixel coordinates, platform coordinates and sequence information.

[0172] 1.4.2 Colony discrimination, discriminate whether there is a colony in each chamber identified, and mark the chamber as 0 / 1, 0 representing no colony in the chamber, and 1 representing a colony in the chamber;

[0173] 1.4.3 Path planning, based on the greedy algorithm, the shortest path planning is performed for all chambers with value 1 in the image, and the serial number 1-N (N≤50) is marked.

[0174] 1.5 Picking; picking colonies according to the image recognition information of step 1.4, as follows:

[0175] 1.5.1 Chamber positioning, according to step 1.4.3, move the chip platform to make the chamber with serial number 1 to the light-thermal heating point;

[0176] 1.5.2 Turn on the laser 1201, and make its light-thermal light path irradiate on the chamber 1;

[0177] 1.5.3 Time setting, according to T=t0+Δt*(m-1) (where T is the time to start collecting droplets of target colonies, t0 is the basic time, Δt is the time interval, and m is the number of screens), dynamically set the collection time of droplets of target colonies;

[0178] 1.5.4 Start collecting, start the collection of colony droplets in the well plate 4 (for example, a 96-well plate), and the specific process is to switch the well position of the 96-well plate and move up and down;

[0179] 1.5.5 Repeat steps 1.5.1-1.5.4 to complete the picking of all colony chambers.

[0180] 1.6 Switch to the next screen;

[0181] 1.7 Repeat 1.2-1.6 until all image acquisition is completed.

[0182] (II) Asynchronous mode; first complete all image acquisition, then identify and pick, the specific steps are as follows:

[0183] bright field

[0184] 1. Panoramic image shooting

[0185] 1.1 Initial screen identification, based on pre-selection preparation -> positioning -> initial screen setting, based on AI technology, identify the initial screen, the specific identification algorithm is as follows (taking the upper left corner of the image as an example):

[0186] S1: Image acquisition;

[0187] S2: Image input, send the collected image into the model (the model is a model established in the early stage);

[0188] S3: Image discrimination, based on the model to discriminate whether the image is an image with a chamber;

[0189] S4-1: If yes, continue to move the chip left to the next screen, repeat steps S1-S3;

[0190] S4-2: If not, return to the previous screen;

[0191] S5: Boundary correction, set the Padding value, make the leftmost chamber edge and the image left boundary as Padding, and make the chamber symmetrically distributed in the image frame.

[0192] Similarly, complete the position identification of up and down, and finally realize the initial screen identification, and move the platform to the initial screen position.

[0193] Complete the initial screen identification, and move the platform to the initial screen position.

[0194] 1.2 Automatic image focusing and chip tilt correction, and record the relevant data of the current screen, including position information, focusing information and correction parameters, the specific focusing / correction method is as follows:

[0195] Automatic focusing (climbing algorithm)

[0196] S1: Image acquisition, set the Z-axis position as zero point position;

[0197] S2: Image input, send the collected image into OpenCvSharp algorithm;

[0198] S3: Image gradient value calculation, use Tenengrad gradient method: Sobel operator to calculate image gradient value;

[0199] S4: Image set acquisition, move up and down at the Z-axis zero point position with Step interval to collect image set, and calculate the image gradient value, draw the curve of Z-axis position and gradient value, and find the peak position (as shown in the figure);

[0200] S5: Image focusing, the platform moves to the highest peak point, and the Z-axis position at this time is the in-focus position of the image, achieving image focusing.

[0201] S1: Image acquisition, acquire the focused image;

[0202] S2: Image input, the image is sent to the trained model;

[0203] S3: Data point generation, identify the chamber, and then identify the center point of the chamber to form a data point column, each point has a coordinate (X, Y), forming a two-dimensional array;

[0204] S4: K value solving, each row array is connected and fitted to solve the average slope Slope, Slope corresponds to positive and negative, K is solved according to the slope Slope, K value is the overall tilt distance of the chamber;

[0205] S5: Correct the image, move the chip platform according to the K value solved in step S4, K value is positive up; K value is negative down.

[0206] Complete automatic focusing and chip tilt correction, and record the current screen position information, focusing information and correction parameters.

[0207] 1.3 Image acquisition; the image collector acquires images and displays them in the panoramic frame, the panoramic image is set to N*M copies according to the number of chip chambers, N is the number of rows, M is the number of columns, and the first row and first column image N1 is acquired;

[0208] 1.4 Chamber identification / data analysis, image recognition and data analysis are performed on the acquired images.

[0209] ID information allocation, identify the outer contour of each chamber in the picture, label the identified chamber with a frame diagram, and record the position information of each chamber.

[0210] S1: Image acquisition, acquire the focused image;

[0211] S2: Image input, the image is sent to the trained model;

[0212] S3: Data point generation, identify the chamber, and then identify the center point of the chamber to form a data point column, each point has a coordinate (X, Y);

[0213] S4: Minimum point traversal, traverse the minimum Y value Min of the center point,

[0214] S5: Cut line drawing, draw a cut straight line with Min±Step respectively;

[0215] S6: Same row discrimination, traverse all center points, if Y is in the range of Min±Step, it is in the same row, and is sorted by X value from small to large, and is set as the first row (odd row);

[0216] S7: Repeat the next row traversal, set as the second row (even row), and sort with the previous row by X value from large to small:

[0217] S8: Assign the data point sorting information to each corresponding chamber, and complete the ID assignment of the chamber.

[0218] At this time, the ID information assignment of each chamber is realized, including pixel coordinates, platform coordinates, and sequence information. Colony discrimination discriminates whether there is a colony in each chamber identified. The chamber is marked 0 / 1, 0 represents that there is no colony in the chamber, and 1 represents that there is a colony in the chamber;

[0219] Colony recognition, if the chamber is marked 1, further recognition of the chamber is performed based on the semantic segmentation algorithm, the colony area (ROI) is identified, and the contour is obtained; calculate the contour, OpenCV calculates the area, average gray value, axis center and other related data of the area where the contour is located. The colony area and gray value are labeled on the chamber, and a scatter plot is constructed with the area and gray value, and the range value of the area and gray value is given. This step can be separately established as a thread, and is synchronized with the next step;

[0220] 1.5 Switch to the next screen and repeat steps 1.2-1.4 until all images are collected, and panoramic image shooting is completed.

[0221] 2. Set the picking parameters; set the picking conditions, including the area, gray value, and product of the two values (concentration) of the chamber, for example, the numerical range is 0-1 (or 0-255), and the required numerical range is selected according to the displayed value.

[0222] 3. Picking

[0223] 3.1 Return to the initial screen, return to the initial position where the panoramic image starts to be shot;

[0224] 3.2 Path planning, according to the picking parameters set in step 2, the shortest path is planned for each screen image, and the serial number NM-1 to NM-P is marked, where N represents the row, M represents the column, and P represents the number of chambers under the screen image, P≤50, and is picked according to the marked serial number;

[0225] 3.3 Chamber positioning, according to step 3.2, move the chip platform to make the chamber with serial number 1 to the light-heat heating point;

[0226] 3.4 Open the optical shutter 5102, so that the light-heat light path is irradiated on the light-heat heating point of the chamber 1;

[0227] 3.5 Time setting, according to T=t0+Δt*(m-1) (wherein T is the time of starting to collect the colony droplet, t0 is the basic time, Δt is the time interval, and m is the screen number, i.e. the column number), the collection time of the colony droplet is dynamically set;

[0228] 3.6 Droplet collection, according to the set time, the collection of the droplet is started;

[0229] 3.7 Continue steps 3.2-3.6 until the current screen finishes the picking of the colonies in all the chambers;

[0230] 3.8 Switch to the next screen, repeat steps 3.2-3.7 until all the screens are finished.

[0231] It should be noted that: if a light-thermal module 51 with a spatial light modulator 5108 is used, when multiple light-thermal light paths are irradiated on the chamber containing the target colony, there is no need to move the chip platform 1 for the experiment, but directly through the image information of the chamber containing multiple target colonies, and then converting into the corresponding phase image, and through the corresponding holographic calculation formula, the synchronous irradiation of multiple light-thermal light paths can be realized, and because the distance between each chamber of the chip and the position of the outlet 204 is different, the target colony can be quickly guided out according to the corresponding order, thereby greatly improving the picking efficiency.

[0232] The embodiments described above are only preferred embodiments of the present application, and do not limit the scope of the present application, and various modifications and improvements to the technical solutions of the present application made by those skilled in the art without departing from the design spirit of the present application shall fall within the protection scope of the present application.

Claims

1. A colony picking system based on optical manipulation technology, wherein, The chip platform is provided with a chip placement part for placing an array chamber chip. The aperture plate platform is provided with an aperture plate placement part for placing an aperture plate. The light-thermal module is used for emitting a light-thermal light path. The number of the light-thermal light path is one and the light-thermal light path irradiates a light-sensitive heating part on one chamber of the chip; or the number of the light-thermal light path is multiple and the light-thermal light path irradiates multiple light-sensitive heating parts on multiple chambers of the chip in parallel. The imaging module is used for identifying the state of the colony in the chamber of the chip in the current field of view. The control module is configured to identify and mark the target colony in the chamber of the chip through the identification of the imaging module, control the light-thermal module to perform light-thermal heating on the light-sensitive heating part on the chamber where the identified and marked target colony is located, and adjust the position of the chip platform and / or the aperture plate platform to transport the target colony from the chip to the collection hole of the aperture plate. The fluorescence detection module is further included for emitting a fluorescence light path; the light-thermal module, the imaging module and the fluorescence detection module share the same microscope objective which is fixedly installed above the chip placement part. The detection and identification module is further included for detecting the colony in the chamber; the detection and identification module is at least one of a fluorescence detection module, a Raman detection module, a laser confocal detection module and a mass spectrometry detection module.

2. The colony picking system of claim 1, wherein, The light-thermal module includes a colony picking light source and a light beam shaping module; the colony picking light source is a laser light source or an LED light source; the light beam shaping module is used for shaping the light beam from the colony picking light source into single-beam or multi-beam and shrinking the light beam; and a light-thermal light path capable of irradiating the light-sensitive heating part on the chip is obtained.

3. The colony picking system of claim 1, wherein, The horizontal height of the chip placement part is higher than the horizontal height of the aperture plate placement part; the picking system further includes a detection module configured to identify whether the droplet discharged from the chip contains the target colony and judge whether the end of the droplet discharged from the chip is aligned with the collection hole of the aperture plate.

4. The colony picking system of claim 1, wherein, The detection module includes:

5. The colony picking system of claim 1, wherein, a visual camera arranged below the aperture plate platform; 6. The colony picking system of claim 5, wherein, a second light splitting prism arranged above the visual camera; an illumination light source and a sixth reflector arranged on both sides of the second light splitting prism respectively; the illumination light source emits a light beam and splits the light beam into reflected light and transmitted light through the second light splitting prism; the reflected light is used for illuminating the collection hole of the aperture plate to judge the alignment state of the end of the droplet discharged from the chip with the collection hole of the aperture plate; and the transmitted light is used for illuminating the chip for imaging through the sixth reflector. The chip platform and the aperture plate platform move along the XY axis in the horizontal direction and move relatively in the vertical direction; the visual camera is configured to identify whether the droplet discharged from the end of the chip contains the target colony, and the relative movement of the chip platform and the aperture plate platform in the vertical direction is controlled to realize the collection of the target colony. A discharge pipe is further included for connecting with the discharge port of the chip; the discharge pipe is vertically arranged at the discharge port or fixedly arranged between the chip platform and the aperture plate platform and connected with the discharge port of the chip through a flexible connecting pipe.

7. The colony picking system of claim 6, wherein, ​ 8. The colony picking system of claim 7, wherein, ​ 9. A colony picking method based on optical manipulation technology, using the colony picking system of any one of claims 1-8, the picking method comprising the steps of: placing a chip with target colonies on a chip placement part of a chip platform; and aligning a microscope objective with the chip; placing a well plate on a well plate placement part of a well plate platform; an imaging module identifying the state of the colonies in each chamber of the chip within the current field of view of the microscope objective; an optical heating module emitting one or more optical heating light paths capable of irradiating the light-sensitive heating part of one chamber of the chip, or the light-sensitive heating parts of multiple chambers; a control module identifying and marking the target colonies in the chambers of the chip through the identification of the imaging module, controlling the optical heating module to perform optical heating on the light-sensitive heating part of the chamber where the identified and marked target colony is located; and adjusting the position of the chip platform and / or the well plate platform to transport the target colony from the chip to the collection well of the well plate.

10. The selection method of claim 9, wherein, comprising: a synchronous mode and an asynchronous mode; the synchronous mode is to obtain the information of the target colonies in the chip within the current field of view, plan the picking path of the target colonies according to the greedy algorithm, and collect according to the planned path, and repeat the steps for the chips within other fields of view until the picking of the target colonies in the entire chip is completed; the asynchronous mode is to complete the image acquisition of the entire chip, identify all target colonies according to the image acquisition, plan the path of all identified target colonies according to the greedy algorithm, and complete the sequential picking.

11. The selection method of claim 9, wherein, It also includes a chip inclination correction method: collect the image information of the placed chip and identify each chamber, form a data point column through the center points of each chamber located on the same horizontal reference line, and obtain the corresponding two-dimensional array coordinates; according to the connection between adjacent arrays on each horizontal reference line to fit and solve the average slope Slope, Slope corresponds to positive and negative, according to Slope to solve K, K value is the overall inclination distance of each chamber, and according to K value to move the chip platform, K value is positive, up; K value is negative, down.

12. The selection method of claim 9, wherein, The method of the optical heating module irradiating the light-sensitive heating part on the chamber of the chip is: adjusting the position of the chip platform to form a single optical heating light path from the laser on the chamber where the target colony is located, and guiding the target colony out in an optical heating manner; or expanding, parallelizing the light beam formed by the laser, and making it enter the spatial light modulator, the spatial light modulator forms multiple optical heating light paths within the current field of view according to the image of the chamber where the target colony is located within the current field of view, and irradiates each chamber where the target colony is located, since the distance from each chamber within the current field of view to the outlet of the chip is different, all target colonies are sequentially guided out in an optical heating manner.

Images