Micro-particle manipulation method, device, system and storage medium

Abstract

The present application relates to the technical field of optical microscopic manipulation, and discloses a micro-particle manipulation method, device, system and storage medium, the method comprising: step 1, obtaining a current micro-particle image; step 2, identifying a target particle in the current micro-particle image, the target particle being a particle as a manipulation object; step 3, for a first target particle currently identified, planning a path of the first target particle based on the current micro-particle image; the first target particle being a target particle without a planned path; step 4, generating a hologram based on the currently planned path; step 5, performing light control based on the currently generated hologram to realize manipulation of the target particle with the planned path; repeating steps 1-5 until the path planning of all target particles is completed; and repeatedly executing steps 4 and 5 until all target particles move to corresponding end positions. The present application can compress cold start to the millisecond level.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical microscopic manipulation, and in particular to a microscopic particle manipulation method, device, system and storage medium. BACKGROUND

[0002] Optical tweezers use the gradient force of a highly focused laser beam to capture and manipulate microscopic particles such as cells, nanoparticles, DNA, etc. The core principles include:

[0003] Gradient force: the strong electric field gradient at the laser focus pulls the particle towards the beam center.

[0004] Scattering force: the photon momentum transfer pushes the particle to move along the light propagation direction.

[0005] Potential well formation: the balance between gradient force and scattering force forms a stable trapping region with nanometer precision.

[0006] In related technologies, the optical tweezers system mainly relies on a spatial light modulator (SLM) digital micromirror (DMD), a laser scanning galvanometer, etc. that can generate a multi-trap light field to generate a dynamic hologram to achieve multi-focus manipulation. The generation of the hologram needs to rely on the path planning result. The existing A* algorithm (also known as A-star algorithm or A* search algorithm) and its derivative algorithms mostly use global batch path planning. When processing more than 50 targets, the cold start delay is more than 500 ms, and as the number increases, the complexity and planning time required increase exponentially. Moreover, it cannot dynamically respond to environmental changes. SUMMARY

[0007] Therefore, the present application provides a microscopic particle manipulation method, device, system and storage medium to solve the problem of long cold start delay and inability to dynamically respond to environmental changes of an intelligent optical tweezers system when processing multiple targets.

[0008] In a first aspect, the present application provides a microscopic particle manipulation method, which comprises:

[0009] Step 1: obtaining a current microscopic particle image;

[0010] Step 2: identifying a target particle in the current microscopic particle image, the target particle being a particle as a manipulation object;

[0011] Step 3: for a first target particle identified at present, planning a path of the first target particle based on the current microscopic particle image; the first target particle being a target particle that has not been planned a path.

[0012] Step 4, generating hologram based on current planned path;

[0013] Step 5, performing light control based on current generated hologram to realize manipulation of the target particle on the planned path;

[0014] repeating the steps 1-5 until path planning of all the target particles is completed; repeating the steps 4 and 5 until all the target particles move to the corresponding end point positions.

[0015] In an optional embodiment, the method for manipulating micro-particles further comprises:

[0016] For a second target particle on the planned path, if it is identified that the second target particle has a collision risk with other particles, a new path is planned for the second target particle to avoid collision;

[0017] wherein the other particles include other target particles or interference particles other than the second target particle.

[0018] In an optional embodiment, the method for manipulating micro-particles further comprises:

[0019] If it is determined based on the current micro-particle image that there is a target interference particle, the target interference particle is an interference particle that will affect the movement or identification of the target particle;

[0020] a movement path is planned to move away the target interference particle; when the new hologram is generated, the new hologram is generated based on the current planned path of the target particle and the movement path of the interference particle.

[0021] In an optional embodiment, the planning of the path of the first target particle based on the current micro-particle image for the first target particle identified currently comprises:

[0022] If a plurality of target particles without planned paths are identified based on the current micro-particle image, the first target particle is determined based on the priority of the target particles; the priority of the target particles is determined based on at least one of the following: start point position, end point position, size and identification confidence;

[0023] The path of the first target particle is planned based on the current micro-particle image.

[0024] In an optional embodiment, the generation of the hologram based on the current planned path comprises:

[0025] The Fourier iterative operator encapsulated based on CUDA is used to generate the hologram corresponding to the current planned path.

[0026] In an optional embodiment, the Fourier iterative operator encapsulated based on CUDA is used to generate the hologram corresponding to the current planned path, including:

[0027] The target amplitude matrix on the CPU is copied to the GPU memory.

[0028] The Fourier iterative method is used to generate the hologram corresponding to the current planned path based on the target amplitude matrix.

[0029] The generated hologram is copied back to the CPU.

[0030] In a second aspect, the present application provides a micro-particle manipulation device, which comprises:

[0031] An image acquisition module is configured to acquire a current micro-particle image.

[0032] An identification module is configured to identify a target particle in the current micro-particle image, the target particle being a particle as a manipulation object.

[0033] A path planning module is configured to plan a path for a first target particle based on the current micro-particle image, the first target particle being a target particle without a planned path.

[0034] A generation module is configured to generate a hologram based on a current planned path.

[0035] An optical control module is configured to control light based on a current generated hologram to manipulate the target particle with a planned path.

[0036] A control module is configured to control the image acquisition module, the identification module, the path planning module, the generation module and the optical control module to repeatedly run until the path planning of all the target particles is completed, and to control the generation module and the optical control module to repeatedly run until all the target particles move to corresponding end positions.

[0037] In a third aspect, the present application provides an optical tweezer system, which comprises:

[0038] A micro-particle image acquisition assembly.

[0039] An optical assembly.

[0040] The memory and the processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the micro-particle manipulation method of the first aspect or any of the corresponding embodiments.

[0041] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer instructions for causing a computer to perform the micro-particle manipulation method of the first aspect or any of the corresponding embodiments.

[0042] In a fifth aspect, the present application provides a computer program product, which includes computer instructions for causing a computer to perform the micro-particle manipulation method of the first aspect or any of the corresponding embodiments.

[0043] The micro-particle manipulation method, device, system and storage medium provided by the embodiments of the present application adopt an incremental path planning scheme, and path planning is performed for only a single micro-particle (target particle) each time, in other words, the path of a micro-particle is planned first, and then the path of another micro-particle is planned. As long as the path of a micro-particle is planned, a hologram is generated based on the planned path and light control is performed, so that the micro-particle (target particle) on the planned path is manipulated, that is, as long as the path of a micro-particle is planned, the manipulation process is started, and therefore the cold start can be compressed to the millisecond level. The embodiments of the present application also eliminate the correlation between the path planning cold start time and the number of target particles. In the related art, when the number of target particles reaches 50, the cold start delay will exceed 500 ms, and when the number of target particles reaches 100, the cold start delay needs 1.2 s, while using the micro-particle manipulation method provided by the embodiments of the present application, even if the number of target particles reaches 100, the cold start delay will be stabilized at 100 ms. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the related art, the drawings needed in the specific embodiments or related art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0045] Figure 1 is a flowchart of the micro-particle manipulation method according to the embodiments of the present application;

[0046] Figure 2 is a schematic diagram of pushing away interfering particles according to the embodiments of the present application;

[0047] Figure 3is a schematic diagram of a Fourier iteration process under CUDA acceleration according to an embodiment of the present application;

[0048] Figure 4 is a structural block diagram of a micro-particle manipulation device according to an embodiment of the present application;

[0049] Figure 5 is a schematic diagram of a hardware structure of a light tweezer system according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] To make the objects, technical solutions and advantages of embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0051] According to an embodiment of the present application, a micro-particle manipulation method embodiment is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of executable computer instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0052] In the present embodiment, a micro-particle manipulation method is provided, which can be used in a light tweezer system, Figure 1 is a flowchart of a micro-particle manipulation method according to an embodiment of the present application, as Figure 1 shown, the flow includes the following steps:

[0053] Step 1, obtaining a current micro-particle image. The micro-particle can be a cell, a nanoparticle, DNA, etc. These micro-particles can be on a glass slide or in a microfluidic chip structure. The microfluidic chip contains microstructures, and this part of the structure is similar to a wall in reality, and the cell cannot pass through. The micro-particle image includes but is not limited to: bright field, dark field, and fluorescence and spectral imaging results obtained by a microscope.

[0054] Specifically, a high-speed CMOS camera can be used to collect micro-particle images in real time.

[0055] Step 2, identifying a target particle in the current micro-particle image, the target particle being a particle as a manipulation object.

[0056] Specifically, the target particles identified here include both target particles that have planned paths and target particles that have not planned paths. The identified parameters include the position, size, classification, speed, and other path allocation related parameters of the target particles. The identification here includes identifying the terrain, obstacles, and target manipulation, etc.

[0057] In addition, while identifying the target particles in the current micro-particle image, other particles, environmental maps (such as a map of micro-structures contained in the microfluidic chip), dirt and other obstacles also need to be identified, because when planning the moving path of the target particles, other particles in addition to the target particles need to be avoided as interference particles to avoid collisions.

[0058] Step 3, for the first target particle currently identified, planning a path for the first target particle based on the current micro-particle image; the first target particle is one of the target particles that has not planned a path.

[0059] Specifically, the algorithm used by the embodiment of the present application when planning a path can be a heuristic pathfinding algorithm-A* algorithm (dA), and specifically can be an optimized A* path planning, and the optimization part solves the problem of excessive bending of the path after optimization. The algorithm used when planning a path can also be a breadth-first search, Dijkstra algorithm (commonly known as "Dijkstra algorithm" or "Dijkstra algorithm" in Chinese), bidirectional search algorithm, etc. In the embodiment of the present application, each main loop only plans a path for one target particle, and if there are multiple target particles currently identified, the first target particle for which the path is currently planned can be selected according to the principle of priority of the center of the end position, or other types of priority principles (such as arranged from bottom to top, or arranged from top to bottom). If the principle of priority of the center of the end position is selected, specifically, the end positions corresponding to all target particles can form an area (or pattern), and for the multiple target particles currently identified, the one whose end position is closer to the center of the area is selected to plan a path. The single-particle path planning is specifically explained as follows:

[0060] If a matching particle and an end point are found, then:

[0061] Step 1, record the matching particle ID and end point coordinates;

[0062] Step 2, calculate the safety radius of the particle;

[0063] Step 3, remove the mask of the end point;

[0064] Step 4, path planning. If the path planning is successful: store the path information in the path list, i.e. path_list, and delete the matched end point from the existing end point table, i.e. end_points_now, and add the end point to the selected end point set, in preparation for the possible loss of particles caused by future interference, when the subsequent particles lose the match with the end point, the end point will be released from the selected end point set, so as to find a new particle to match with it; otherwise, mark the particle as unselectable and delete the relevant record.

[0065] In some specific embodiments, when performing path planning, a scaling factor can be determined based on the size information of the target particles and / or obstacles (such as microstructures in the microfluidic chip, interfering particles, etc.), and the map can be scaled according to the determined scaling factor. The original size of the path planning is generally 1920x1200, and the calculation amount is very large, so if the map can be scaled by a certain multiple (such as ten times, or other multiples) according to the determined scaling factor before path planning, the calculation amount can be reduced, and at the same time the maximum operation time can be limited, generally limited to 0.2 seconds. When the path planning operation time exceeds this value, it means that the end point is probably unreachable, so the path planning under the current state is abandoned, the end point is added to the set of end points that cannot be found, and after a certain number of frames (usually 30 frames), it is re-added to the set of end points that can be matched.

[0066] Since the light path will have a certain degree of scaling and translation after hologram modulation, in order to ensure the stability of the manipulation, a correction matrix is made according to the position of the light spot, and an adaptive matrix is made to ensure the accuracy of the light spot. In addition, accurate control of the position of the light spot also requires pre-allocation of light spot power to avoid possible problems such as damage, ejection, escape, etc. of the target particles caused by too large fluctuations in light spot intensity.

[0067] Regarding the correction matrix, specifically, since the laser incidence will have a certain deformation, it is not possible to accurately match by simply scaling, so scaling is performed and then the calculation of the correction matrix is performed, which acts on the target image and the transformation of the imaging, so that each light spot in the field matches the target image.

[0068] Regarding the pre-allocation of light spot power, specifically, since the laser incidence power is controlled by an attenuator, when the number of particles increases, the energy obtained by each particle will decrease, so a specified number of light spots can be reserved in the field and outside the field, and when the number of particles in the field increases, the reserved positions of the light spots outside the field are allocated into the field, so even if the power is large at the beginning, the light intensity borne by the particles is relatively uniform due to the allocation of a large number of reserved light spots, thereby reducing the probability of ejection.

[0069] If there is a planned path, then start generating hologram, the following steps need to be performed before generating hologram:

[0070] Step 1, initialize relevant variables;

[0071] Step 2, traverse all planned paths.

[0072] If the particle is still in the field of view: proceed with path correction; calculate the safety distance; proceed with obstacle avoidance processing. If the obstacle avoidance is successful, update the path information, mark the particle as being controlled, record the end position, add the path to the drawing list (path_list); otherwise: release the end point, delete the relevant records.

[0073] In the process of path planning and in the process of obstacle avoidance processing, the particle speed can be updated in real time, the future path to be passed through is sampled, and the distance is divided by the current speed, so as to obtain the time required to reach the sampling point in the future. The subsequent particle will make a prediction according to the above-mentioned time when planning, thereby reducing the probability of future conflicts.

[0074] If the particle disappears, increase the disappearance counter, and record the frame number of the particle disappearance. If the disappearance exceeds 20 frames, release the end point corresponding to the disappeared particle, and delete the relevant records.

[0075] In addition, if there is a particle being controlled, the interfering particle is processed. Specifically, when an interfering particle (such as a cell that is not being manipulated) approaches a particle being manipulated, the interfering particle is pushed away. If necessary, a leaving path of the interfering particle can be planned to avoid affecting other manipulated particles during the leaving process. For example, as shown in FIG. 8, for an interfering particle (white particle in the figure) close to a manipulated particle (black particle in the figure), the interfering particle is pushed away in a direction away from the manipulated particle (the arrow in the figure indicates the pushing direction). Two optical tweezers are placed in the direction of pushing the particle, so that the interfering particle receives a stronger gradient force, thereby quickly solving the interference. At the same time, in order to reduce the influence of other optical tweezers on the particle during movement, two optical tweezers are also placed in front of the moving direction of the particle. Therefore, when the particle passes through the end point, since only one optical tweezer is placed at the end point of the cell, there is a stronger gradient force in the front of the moving particle, thereby reducing the interference. Figure 2

[0076] The path planning in the embodiment of the application is a global optimal method, and the path planning in the embodiment of the application is sequentially planned by a single agent, and the path planning is planned frame by frame according to the current situation.

[0077] ​Step 4, generating hologram based on the current planned path. Specifically, the GS algorithm (i.e., Gerchberg-Saxton algorithm) can be used to generate the hologram.

[0078] Step 5, performing light control based on the current generated hologram to realize the manipulation of the target particle on the planned path. Specifically, the manipulation of the target particle is realized based on the light field formed by the hologram.

[0079] Repeating the steps 1-5 until the path planning of all the target particles is completed; then, repeatedly performing steps 4 and 5 until all the target particles move to the corresponding end positions.

[0080] In the related art, in the process of manipulating multiple micro-particles simultaneously using an optical tweezer system, a global batch path planning scheme is adopted, and most of the time, the path is calculated only after the micro-particles are bound, so as to avoid the movement of the micro-particles during the path calculation process, which may cause the calculated path to be invalid. Therefore, the cold start with low delay is not realized in the related art. The embodiment of the present application adopts an incremental path planning scheme, and the path is planned for only one micro-particle (target particle) at a time. In other words, the path of one micro-particle is planned first, and then the path of another micro-particle is planned. As long as the path of one micro-particle is planned, a hologram is generated based on the planned path and light control is performed to realize the manipulation of the micro-particle (target particle) on the planned path. That is, as long as the path of one micro-particle is planned, the manipulation process is started. Therefore, the cold start can be compressed to the millisecond level. The embodiment of the present application also eliminates the correlation between the cold start time of path planning and the number of target particles. In the related art, when the number of target particles reaches 50, the cold start delay will exceed 500 ms, and when the number of target particles reaches 100, the cold start delay will be 1.2 s. However, using the micro-particle manipulation method provided by the embodiment of the present application, even if the number of target particles reaches 100, the cold start delay will be stable at 100 ms.

[0081] In addition, in the embodiment of the present application, as long as the path of one micro-particle is planned, the manipulation process is started, and then the path planning of other micro-particles (other target particles whose paths have not been planned) is performed while the particles are being manipulated to move. That is, for the other micro-particles whose paths are planned later, the path is planned based on the latest particle environment (specifically, based on the latest micro-particle image). Therefore, even if the particle environment changes compared to the initial environment, it does not affect the planned path. Therefore, compared with the related art, the embodiment of the present application can dynamically respond to environmental changes, and the path planning can be performed without binding the micro-particles.

[0082] In some optional specific embodiments, the micro-particle manipulation method provided by the embodiment of the present application further comprises:

[0083] For the second target particle of the planned path, if it is identified that the second target particle has a collision risk with other particles, a new path is planned for the second target particle to avoid collision;

[0084] The other particles include other target particles or interference particles other than the second target particle.

[0085] In the embodiments of the present application, whether in the process of moving the target particle while planning the path of other micro-particles (other target particles without planned paths), or after the paths of all target particles have been planned, as long as it is identified that the second target particle of the planned path has a collision risk with other particles, a new path is planned for the second target particle to avoid collision. That is, the embodiments of the present application establish a dynamic interference suppression mechanism, and improve the generalization of the micro-particle manipulation method and the stability of the manipulation process.

[0086] In the embodiments of the present application, the process of manipulating the movement of the target particle is step by step. Specifically, in the process of manipulating the movement of the target particle, the following process needs to be repeated multiple times: generating a hologram based on the planned path, and performing light control based on the hologram to complete the one-step movement process of the target particle. In this process, a high-speed CMOS camera is also used to collect micro-particle images in real time. Therefore, in the process of manipulating the target particle, the current manipulated target particle can be determined in real time whether it has a collision risk according to the collected micro-particle images. For example, if the distance between the other particles and the target particle is less than or equal to 15 μm, it is determined that the target particle has a collision risk with other particles.

[0087] In some other optional specific embodiments, the micro-particle manipulation method provided by the embodiments of the present application further includes:

[0088] Step one, if it is determined based on the current micro-particle image that there is a target interference particle, the target interference particle is an interference particle that will affect the movement or identification of the target particle. Specifically, some target interference particles may collide with the moving target particle, i.e. affect the movement of the target particle; some particles, although they will not collide with the moving target particle, are too close to the target particle, affecting the identification of the target particle (for example, the target particle and the interference particle may be identified as a whole, i.e. identified as another kind of particle). This is generally because the interference particle did not appear when the path of the target particle was planned, it is a newly appeared interference particle, or the interference particle appeared when the path of the target particle was planned but was not identified, or the interference particle appeared when the path of the target particle was planned and was avoided when the path was planned, but the position of the interference particle changed over time. Among them, the identification of the interference particle affecting the movement of the target particle (i.e. may collide with the target particle) can also be identified by distance, i.e. if the distance between the interference particle and the target particle is less than a predetermined threshold, it is determined that the interference particle affects the movement of the target particle.

[0089] Step two, planning a moving path to move away the target interference particle; when generating a new hologram, based on the current planned path of the target particle and the moving path of the interference particle, a new hologram is generated.

[0090] In the embodiments of the present application, whether in the process of moving the particle while planning the path of other micro-particles (other target particles whose paths have not been planned) or after the paths of all target particles have been planned, as long as an interference particle that may affect the movement or identification of a target particle or several target particles is identified, a moving path is planned for the interference particle. Of course, in the process of planning the moving path of the interference particle, not only does it need to avoid affecting the original target particle that may be affected, but it also needs to avoid affecting other target particles during its movement, that is, when planning the moving path of the interference particle, it needs to avoid affecting any target particle during its movement or after its movement.

[0091] The embodiments of the present application establish a dynamic interference suppression mechanism from the perspective of moving the interference particle, improving the generalization of the micro-particle manipulation method and the stability of the manipulation process. One test data shows that the interference events are reduced by 82%, and the success rate of particle manipulation is improved from 68% to 93% (yeast cell line test data).

[0092] In some optional specific embodiments, step 3, i.e. planning a path for the first target particle identified based on the current micro-particle image, includes:

[0093] In step 31, if a plurality of target particles without planned paths are identified based on the current micro-particle image, the first target particle is determined based on a priority of the target particles; the priority of the target particles is determined based on at least one of the following: the start position, the end position, the size, and the identification confidence. The priority of the target particles indicates the degree of urgency of manipulation. As described above, if the priority of the target particles is determined based on the end position, one specific determination principle is the center priority principle.

[0094] In step 32, the path of the first target particle is planned based on the current micro-particle image.

[0095] In the embodiments of the present application, the priority of the target particles can be determined in advance based on the start position or the end position of the target particles or other factors, and then when a plurality of target particles without planned paths are identified based on the current micro-particle image, the path of which target particle is planned first is determined according to the priority determined in advance. Ideally, the path is planned one target particle after another according to the priority order determined in advance. However, if a target particle with a high priority order (which has not been planned a path) is not identified when the target particles are identified based on the current micro-particle image, the target particle with the highest priority among the target particles without planned paths identified currently can be selected to plan a path, or the path can be temporarily not planned and wait for N (N is a natural number greater than or equal to 1) new micro-particle images. If a target particle with a high priority order (which has not been planned a path) is not identified in the N new micro-particle images, the target particle with the highest priority among the target particles without planned paths identified currently is selected to plan a path.

[0096] In addition, the path planning in the embodiments of the present application is dynamically performed in real time throughout the whole process, and the path is adjusted in real time according to the state of the target particle and the surrounding environment.

[0097] In related technologies, when generating a hologram, a CPU-based Fourier iterative algorithm (such as the Gerchberg-Saxton algorithm, i.e., the GS algorithm) is used. This method is low in efficiency in generating a hologram, and the calculation time is about 2.5 seconds under a 2048x2048 resolution, which cannot meet the real-time interaction requirement. Therefore, the hologram generation method based on CUDA acceleration is used in the embodiments of the present application to break through the CPU calculation bottleneck and realize millisecond-level hologram generation. The method is described in detail as follows.

[0098] In an alternative specific implementation, step 5, i.e., the generation of the hologram based on the current planned path, includes:

[0099] The current planned path corresponds to the hologram is generated using a CUDA encapsulated Fourier iterative operator.

[0100] CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by NVIDIA, which allows developers to utilize the powerful computing capabilities of GPUs using C / C++ language. Unlike traditional CPU programming, CUDA programming requires developers to understand the parallel computing architecture of GPUs. The core concepts of CUDA programming include kernel functions, threads, thread blocks, and grids. Kernel functions are functions that run on GPUs and are executed by multiple threads simultaneously. Threads are the smallest execution units in CUDA, and multiple threads form a thread block, while multiple thread blocks form a grid. This hierarchical structure allows CUDA to flexibly handle computing tasks of different scales. In actual programming, CUDA programs usually follow the following process: first, prepare data on the CPU, then copy data from the CPU memory to the GPU memory, then launch the kernel function to let the GPU parallelize the data processing, and finally copy the results from the GPU memory back to the CPU memory. This process requires developers to pay special attention to memory management and data transfer overhead. The advantage of CUDA programming is that it can significantly accelerate compute-intensive tasks, especially those that can be decomposed into a large number of independent small tasks. However, CUDA programming also faces some challenges, such as the need to reasonably decompose problems into parallel tasks, the need to consider the memory architecture of the GPU to achieve optimal performance, and the need to handle data transfer between the CPU and the GPU.

[0101] Specifically, the core logic of generating a hologram using the Fourier iterative method is as follows:

[0102] Step 1, randomly initialize the phase;

[0103] Step 2, Fourier transform the initialized phase and amplitude;

[0104] Step 3, normalize and calculate the correlation;

[0105] Step 4, obtain the phase in Step 2;

[0106] Step 5, inverse Fourier transform the target amplitude and phase;

[0107] Repeat the above steps 2-5 until the correlation reaches the target value;

[0108] Finally, extract the phase information in Step 5 and normalize it, and the corresponding hologram can be obtained.

[0109] In an alternative embodiment, the generating the hologram corresponding to the current planned path based on the CUDA encapsulated Fourier iterative operator comprises:

[0110] copying the target amplitude matrix on the CPU to the GPU memory;

[0111] generating the hologram corresponding to the current planned path based on the target amplitude matrix using the Fourier iterative method;

[0112] copying the generated hologram back to the CPU.

[0113] Specifically, the generating the hologram corresponding to the current planned path based on the CUDA encapsulated Fourier iterative operator comprises: Figure 3 The specific process (CUDA encapsulated logic) of generating the hologram corresponding to the current planned path is shown in the following table:

[0114] 1. Initialization stage:

[0115] calculating the image size size = width × height;

[0116] allocating memory on the GPU: d_amplitude, d_g0_GS, d_phase_image;

[0117] copying the CPU data h_amplitude (i.e. target amplitude distribution data) to the GPU memory d_amplitude.

[0118] 2. Random phase initialization, i.e. form initialization phase distribution:

[0119] starting the init_random_phase kernel function;

[0120] each thread generates a random phase and initializes the complex number array d_g0_GS.

[0121] 3. Set FFT plan:

[0122] create a 2D FFT plan (cufftPlan2d).

[0123] 4. Iterative processing (loop iterations times):

[0124] a. Perform forward FFT transform (cufftExecC2C);

[0125] b. Calculate the current correlation (reserve correlation_kernel);

[0126] c. Check if the target correlation is reached: if so, print the correlation information and break the loop; i.e., meet the iteration stop condition to break the loop;

[0127] d. Normalize the amplitude (normalize_amplitude kernel function);

[0128] e. Perform the inverse FFT transform (cufftExecC2C);

[0129] f. Add random perturbation (reserve add_random_perturbation kernel function).

[0130] 5. Phase extraction:

[0131] Start the extract_phase kernel function;

[0132] Each thread extracts the phase information from the complex data, i.e., the output target phase.

[0133] 6. Result processing:

[0134] Copy the phase data from the GPU back to the CPU;

[0135] Normalize the phase data ([0, 2π] -> [0, 1]).

[0136] At this point, the hologram generation has been completed.

[0137] 7. Cleanup phase:

[0138] Destroy the FFT plan;

[0139] Release all GPU memory.

[0140] In the embodiments of the present application, the Fourier iterative operation is performed using the Fourier transform interface of CUDNN. CUDNN (CUDADeep Neural Network library) is a library built based on CUDA, which is optimized for deep learning applications. It provides a series of highly optimized routines for common operations in deep neural networks, such as convolution, pooling, activation functions, etc. Using CUDNN, developers can easily implement high-performance training and inference in their deep learning models without deep understanding of the underlying parallel algorithms or hardware details. The main advantages of CUDNN are its high performance and ease of use. Since it is designed specifically for deep learning, it is more efficient than directly using CUDA when dealing with related tasks.

[0141] In the embodiment of the present application, the GS algorithm is also used to generate the hologram, but based on CUDA acceleration, which can make the generation speed of the hologram increase by 100 times relative to the traditional hologram generation algorithm. Experimental data shows that the hologram generation time can be reduced from 2500 ms (CPU) to 22.3±1.5 ms (GPU), that is, the hologram generation speed is increased by about 112 times. The system response frequency reaches 10 Hz, supporting 50+ target particle synchronous operation.

[0142] In the micro-particle manipulation method provided by the embodiment of the present application, the target particle can be a cell, so the embodiment of the present application can be applied to the cell precision operation scene in the biomedical field, including but not limited to cell sorting, cell fusion and other applications.

[0143] In the present embodiment, a micro-particle manipulation device is also provided, which is used to realize the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0144] The present embodiment provides a micro-particle manipulation device, as shown in Figure 4 , comprising:

[0145] The image acquisition module 201 is configured to acquire a current micro-particle image.

[0146] The recognition module 202 is configured to recognize a target particle in the current micro-particle image, the target particle being a particle as a manipulation object.

[0147] The path planning module 203 is configured to plan a path of a first target particle based on the current micro-particle image, the first target particle being one of the target particles without a planned path.

[0148] The generation module 204 is configured to generate a hologram based on the currently planned path.

[0149] The light control module 205 is configured to control light based on the currently generated hologram to manipulate the target particles with the planned paths.

[0150] The control module 206 is configured to control the image acquisition module 201, the recognition module 202, the path planning module 203, the generation module 204 and the light control module 205 to repeatedly run until the path planning of all the target particles is completed, and control the generation module 204 and the light control module 205 to repeatedly run until all the target particles move to the corresponding end positions.

[0151] In some optional embodiments, the path planning module 203 is further configured to, for a second target particle of the planned path, if it is identified that the second target particle has a collision risk with other particles, plan a new path for the second target particle to avoid the collision;

[0152] wherein the other particles include other target particles or interference particles other than the second target particle.

[0153] In some optional embodiments, the path planning module 203 is further configured to, if it is determined based on the current microscopic particle image that there is a target interference particle, plan a moving path for moving away the target interference particle; the target interference particle is an interference particle that will affect the movement or identification of the target particle; and when generating a new hologram, generate the new hologram based on the currently planned path of the target particle and the moving path of the interference particle.

[0154] In some optional embodiments, the path planning module 203 includes:

[0155] A determination unit configured to, if it is identified based on the current microscopic particle image that there are a plurality of target particles that have not been planned a path, determine the first target particle based on a priority of the target particles; the priority of the target particles is determined based on at least one of a starting position, an ending position, a size, and an identification confidence.

[0156] A planning unit configured to plan a path for the first target particle based on the current microscopic particle image.

[0157] In some optional embodiments, the generation module 204 is specifically configured to generate the hologram corresponding to the currently planned path using a Fourier iterative operator encapsulated based on CUDA.

[0158] In some optional embodiments, the generation module 204 specifically includes:

[0159] A first copying unit configured to copy a target amplitude matrix on a CPU to a GPU memory;

[0160] A generation unit configured to generate a hologram corresponding to the currently planned path based on the target amplitude matrix using a Fourier iterative method;

[0161] A second copying unit configured to copy the generated hologram back to the CPU.

[0162] Further function descriptions of the above-mentioned various modules and units are the same as those of the above-mentioned corresponding embodiments, which will not be repeated here.

[0163] In this embodiment, the microparticle manipulation device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0164] This invention also provides an optical tweezers system having the above-described features. Figure 4 The microscopic particle manipulation device shown.

[0165] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an optical tweezers system provided in an optional embodiment of the present invention, as shown below. Figure 5 As shown, the optical tweezers system includes one or more processors 10, a memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the optical tweezers system, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Figure 5 Take a processor 10 as an example.

[0166] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0167] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0168] The memory 20 can include a program storage area that can store an operating system, at least one application program required by a function, and a data storage area that can store data created according to use of the optical tweezer system, etc. In addition, the memory 20 can include a high-speed random access memory and can also include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include memory that is remotely located relative to the processor 10, which can be connected to the optical tweezer system through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The memory 20 can include a volatile memory such as a random access memory, and can also include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk, and can also include a combination of the above types of memory.

[0170] The optical tweezer system also includes a micro-particle image acquisition assembly 50 and an optical assembly 60. Specifically, the micro-particle image acquisition assembly can be a high-speed CMOS camera. The optical assembly can be a phase-type liquid crystal on silicon - spatial light modulator (LCOS-SLM) (3840x2160, 60Hz refresh rate).

[0171] The optical tweezer system also includes a graphics processing unit (GPU), which can be, for example, a NVIDIA Ada Lovelace architecture GPU (RTX 4090D).

[0172] The embodiments of the present application provide an efficient optical tweezer system combining GPU parallel computing and intelligent path planning.

[0173] The optical tweezer system also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30, and the output device 40 can be connected through a bus or other means, Figure 5 For example, by way of a bus connection.

[0174] The input device 30 can receive inputted digital or character information, and generate key signal inputs related to user settings and function controls of the optical tweezer system, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), a haptic feedback device (e.g., a vibration motor), etc. The display device includes, but is not limited to, a liquid crystal display, a light emitting diode, a display, and a plasma display. In some alternative embodiments, the display device can be a touch screen.

[0175] The optical tweezer system further includes a communication interface for the optical tweezer system to communicate with other devices or communication networks.

[0176] The embodiments of the present application also provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded to a local storage medium through network, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, which, when accessed and executed by the computer, the processor, or the hardware, implements the method shown in the above embodiments.

[0177] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be invoked or provided. Those skilled in the art should understand that the form of computer program instructions in a computer readable medium includes but is not limited to source files, executable files, installation package files, etc. Correspondingly, the way of computer program instructions executed by a computer includes but is not limited to: the computer directly executes the instructions, or the computer executes the corresponding compiled program after compiling the instructions, or the computer reads and executes the instructions, or the computer executes the corresponding installed program after reading and installing the instructions. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0178] While embodiments of the application have been described in connection with the preferred embodiments of the various figures, those of ordinary skill in the art will appreciate that various modifications and variations of the preferred embodiments can be employed without departing from the spirit and scope of the application.

Claims

1. A method of manipulating micro-objects, comprising: The method comprises: Step 1, obtaining a current micro-particle image; Step 2, identifying a target particle in the current micro-particle image, the target particle being a particle as a manipulation object; Step 3, for a first target particle currently identified, planning a path of the first target particle based on the current micro-particle image; the first target particle being a target particle without a planned path; Step 4, generating a hologram based on the currently planned path; Step 5, performing light control based on the currently generated hologram to realize manipulation of the target particle with the planned path; Repeat steps 1-5 until the path planning of all target particles is completed; repeat steps 4 and 5 until all target particles move to the corresponding end position; Wherein, for the first target particle currently identified, planning a path of the first target particle based on the current micro-particle image comprises: If multiple target particles without a planned path are identified based on the current micro-particle image, the first target particle is determined based on the priority of the target particle; the priority of the target particle is determined based on at least one of the following: starting position, end position, size and identification confidence; Planning a path of the first target particle based on the current micro-particle image.

2. The method of claim 1, wherein, Further comprising: For a second target particle with a planned path, if it is identified that the second target particle has a collision risk with other particles, a new path is planned for the second target particle to avoid collision; Wherein, the other particles include other target particles or interference particles other than the second target particle.

3. The method of claim 1, wherein, Further comprising: If it is determined based on the current micro-particle image that there is a target interference particle, a moving path is planned to move away the target interference particle; the target interference particle is an interference particle that will affect the movement or identification of the target particle; when generating a new hologram, a new hologram is generated based on the currently planned path of the target particle and the moving path of the interference particle.

4. The method of claim 1, wherein, The generation of the hologram based on the currently planned path comprises: Using a Fourier iteration operator encapsulated based on CUDA to generate the hologram corresponding to the currently planned path.

5. The method of claim 4, wherein, The generation of the hologram corresponding to the currently planned path using the Fourier iteration operator encapsulated based on CUDA comprises: Copying a target amplitude matrix on a CPU to a GPU memory; Using a Fourier iteration method to generate a hologram corresponding to the currently planned path based on the target amplitude matrix; Copying the generated hologram back to the CPU.

6. A micro-particle manipulation device, characterized by comprising: The device comprises: An image acquisition module for acquiring a current micro-particle image; An identification module for identifying a target particle in the current micro-particle image, the target particle being a particle as a manipulation object; A path planning module for planning a path of a first target particle currently identified based on the current micro-particle image; the first target particle being a target particle without a planned path; generating a hologram based on a currently planned path; controlling light based on the currently generated hologram to achieve manipulation of the target particles on the planned path; controlling the image acquisition module, the identification module, the path planning module, the generating module, and the light control module to repeatedly run until path planning for all the target particles is completed, and controlling the generating module and the light control module to repeatedly run until all the target particles move to corresponding end point positions; wherein the path planning module comprises: a determination unit configured to determine the first target particle based on a priority of the target particle if a plurality of target particles without planned paths are identified based on the current microscopic particle image, and the priority of the target particle is determined based on at least one of a start point position, an end point position, a size, and an identification confidence; a planning unit configured to plan a path for the first target particle based on the current microscopic particle image.

7. A optical tweezer system characterized by, comprise: a microscopic particle image acquisition assembly; an optical assembly; a memory and a processor, which are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the microscopic particle manipulation method in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to perform the microscopic particle manipulation method in any one of claims 1 to 5.

9. A computer program product, characterised in that, The computer instructions are used to cause a computer to perform the microscopic particle manipulation method in any one of claims 1 to 5. The computer instructions are used to cause a computer to perform the microscopic particle manipulation method in any one of claims 1 to 5.

Images