Microscopic particle manipulation method, device and system and storage medium

Abstract

The invention relates to the technical field of optical micromanipulation, and discloses a microscopic particle manipulation method, device and system and a storage medium, and the method comprises the steps: 1, obtaining a current microscopic particle image; step 2, identifying target particles in the current microscopic particle image, wherein the target particles are particles serving as manipulated objects; 3, planning the path of the first target particle according to the currently identified first target particle based on the current microscopic particle image; the first target particle is a target particle of which the path is not planned; step 4, generating a hologram based on the currently planned path; step 5, performing light control based on the currently generated hologram to realize manipulation of the target particle with the planned path; repeating the steps 1-5 until the path planning of all target particles is completed; and repeating the step 4 and the step 5 until all the target particles move to the corresponding final positions. According to the invention, cold start can be compressed to a millisecond level.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical microscopic manipulation, and in particular to a microscopic particle manipulation method, device, system and storage medium. BACKGROUND

[0002] Optical tweezers use the gradient force of a highly focused laser beam to capture and manipulate microscopic particles such as cells, nanoparticles, DNA, etc. The core principles include: Gradient force: The strong electric field gradient at the laser focus pulls the particle towards the beam center.

[0003] Scattering force: The photon momentum transfer pushes the particle to move along the light propagation direction.

[0004] Potential well formation: The balance between the gradient force and the scattering force forms a stable trapping region with nanometer precision.

[0005] In related technologies, the optical tweezers system mainly relies on a spatial light modulator (SLM) digital micromirror (DMD), a laser scanning galvanometer, etc. to generate a multi-trap light field, and generates a dynamic hologram to achieve multi-focus manipulation. The generation of the hologram needs to rely on the path planning result. The existing A* algorithm (also known as A-star algorithm or A* search algorithm) and its derivative algorithms mostly use global batch path planning. When processing more than 50 targets, the cold start delay is more than 500 ms, and as the number increases, the complexity and planning time required increase exponentially. Moreover, it cannot dynamically respond to environmental changes. SUMMARY

[0006] Therefore, the present application provides a microscopic particle manipulation method, device, system and storage medium to solve the problem that the intelligent optical tweezers system has a long cold start delay when processing multiple targets and cannot dynamically respond to environmental changes.

[0007] In a first aspect, the present application provides a microscopic particle manipulation method, which comprises: Step 1: obtaining a current microscopic particle image; Step 2: identifying a target particle in the current microscopic particle image, the target particle being a particle as a manipulation object; Step 3: for a first target particle identified at present, planning a path of the first target particle based on the current microscopic particle image; the first target particle being a target particle that has not been planned a path; Step 4: generating a hologram based on the currently planned path; Step 5, light control based on the current generated hologram, to achieve the manipulation of the target particles on the planned path; Repeat steps 1-5 until all the target particles path planning is completed; repeat steps 4 and 5 until all the target particles move to the corresponding end position.

[0008] In an alternative embodiment, the micro-particle manipulation method further comprises: For the second target particle on the planned path, if it is identified that the second target particle has a collision risk with other particles, a new path is planned for the second target particle to avoid collision; Wherein, the other particles include other target particles or interference particles other than the second target particle.

[0009] In an alternative embodiment, the micro-particle manipulation method further comprises: If it is determined based on the current micro-particle image that there is a target interference particle, the target interference particle is an interference particle that will affect the movement or identification of the target particle; A movement path is planned to move away the target interference particle; when generating a new hologram, a new hologram is generated based on the currently planned path of the target particle and the movement path of the interference particle.

[0010] In an alternative embodiment, the planning of the path of the first target particle based on the current micro-particle image for the currently identified first target particle comprises: If multiple target particles without planned paths are identified based on the current micro-particle image, the first target particle is determined based on the priority of the target particles; the priority of the target particles is determined based on at least one of the following: starting position, end position, size and identification confidence; The path of the first target particle is planned based on the current micro-particle image.

[0011] In an alternative embodiment, the generation of a hologram based on the currently planned path comprises: The Fourier iterative operator encapsulated based on CUDA is used to generate the hologram corresponding to the currently planned path.

[0012] In an alternative embodiment, the generation of the hologram corresponding to the currently planned path using the Fourier iterative operator encapsulated based on CUDA comprises: Copy the target amplitude matrix on the CPU to the GPU memory; generate a hologram corresponding to the current planned path based on the target amplitude matrix using a Fourier iteration method; copy the generated hologram back to the CPU.

[0013] In a second aspect, the present application provides a micro-particle manipulation device, the device comprising: an image acquisition module configured to acquire a current micro-particle image; an identification module configured to identify a target particle in the current micro-particle image, the target particle being a particle as a manipulation object; a path planning module configured to plan a path for a first target particle based on the current micro-particle image, the first target particle being one of the target particles that has not been planned a path; a generation module configured to generate a hologram based on the current planned path; an optical control module configured to perform optical control based on the current generated hologram to manipulate the target particles that have been planned a path; a control module configured to control the image acquisition module, the identification module, the path planning module, the generation module and the optical control module to repeatedly run until the path planning for all the target particles is completed, and to control the generation module and the optical control module to repeatedly run until all the target particles move to corresponding end positions.

[0014] In a third aspect, the present application provides an optical tweezer system, comprising: a micro-particle image acquisition assembly; an optical assembly; a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the micro-particle manipulation method of the first aspect or any of the corresponding embodiments thereof.

[0015] In a fourth aspect, the present application provides a computer readable storage medium, the computer readable storage medium storing computer instructions, the computer instructions being used to make a computer execute the micro-particle manipulation method of the first aspect or any of the corresponding embodiments thereof.

[0016] In a fifth aspect, the present application provides a computer program product, the computer program product comprising computer instructions, the computer instructions being used to make a computer execute the micro-particle manipulation method of the first aspect or any of the corresponding embodiments thereof.

[0017] The micro-particle manipulation method, device, system and storage medium provided by the embodiment of the present application adopt an incremental path planning scheme, and path planning is performed for only a single micro-particle (target particle) each time, in other words, the path of a micro-particle is planned first, and then the path of another micro-particle is planned. As long as the path of a micro-particle is planned, a hologram is generated based on the planned path and light control is performed, so that the micro-particle (target particle) on the planned path is manipulated, that is, as long as the path of a micro-particle is planned, the manipulation process is started, and therefore the cold start can be compressed to the millisecond level. The embodiment of the present application also eliminates the correlation between the path planning cold start time and the number of target particles. In the related art, when the number of target particles reaches 50, the cold start delay will exceed 500 ms, and when the number of target particles reaches 100, the cold start delay needs 1.2 s. However, using the micro-particle manipulation method provided by the embodiment of the present application, even if the number of target particles reaches 100, the cold start delay will be stabilized at 100 ms. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the specific embodiments or the related art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the related art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0019] Figure 1 is a flowchart of a micro-particle manipulation method according to an embodiment of the present application; Figure 2 is a schematic diagram of pushing away interfering particles according to an embodiment of the present application; Figure 3 is a flowchart of Fourier iteration under CUDA acceleration according to an embodiment of the present application; Figure 4 is a structural block diagram of a micro-particle manipulation device according to an embodiment of the present application; Figure 5 is a hardware structure diagram of an optical tweezer system according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort are within the scope of protection of the present application.

[0021] According to an embodiment of the present application, a micro-particle manipulation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of executable computer instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0022] In this embodiment, a micro-particle manipulation method is provided, which can be used in an optical tweezer system, Figure 1 is a flowchart of a micro-particle manipulation method according to an embodiment of the present application, as Figure 1 shown, the flow includes the following steps: Step 1, obtaining a current micro-particle image. The micro-particle can be a cell, a nanoparticle, DNA, etc. These micro-particles can be on a glass slide or in a microfluidic chip structure. The microfluidic chip contains microstructures, and the structure is similar to a wall in reality, which cannot be passed through by cells. The micro-particle image includes but is not limited to bright field, dark field, and fluorescence and spectral imaging results obtained by a microscope.

[0023] Specifically, a high-speed CMOS camera can be used to collect micro-particle images in real time.

[0024] Step 2, identifying a target particle in the current micro-particle image, the target particle being a particle as a manipulation object.

[0025] Specifically, the target particle identified here includes both a target particle that has been planned a path and a target particle that has not been planned a path. The identified parameters include the position, size, classification, speed, etc. of the target particle, and the path allocation related parameters. The identification here includes identifying the terrain, obstacles, and control targets, etc.

[0026] In addition, while identifying the target particle in the current micro-particle image, other particles, environmental maps (such as maps of microstructures contained in a microfluidic chip), dirt, and other obstacles are also identified, because when planning the moving path of the target particle, other particles except the target particle are needed as interference particles to be avoided to avoid collision.

[0027] Step 3, for a first target particle identified in the current, planning a path for the first target particle based on the current micro-particle image; the first target particle being a target particle that has not been planned a path.

[0028] Specifically, the algorithm used by the embodiment of the present application when performing path planning can be a heuristic pathfinding algorithm, A* algorithm (dA), and specifically can be an optimized A* path planning, the optimization part being in a cost function, which solves the problem of excessive detours after optimization. The algorithm used when performing path planning can also be a breadth-first search, Dijkstra algorithm (commonly known as "Dijkstra algorithm" or "Dijkstra algorithm" in Chinese), bidirectional search algorithm, etc. In the embodiment of the present application, each main loop only plans the path of one target particle. If multiple target particles are currently identified, the first target particle to be planned can be selected according to the principle of priority of the center of the end position, or other types of priority principles (such as arranged from bottom to top, or from top to bottom). If the principle of priority of the center of the end position is selected, specifically, the end positions corresponding to all target particles can form an area (or pattern), and for multiple target particles currently identified, who has an end position closer to the center of the area is preferentially selected to perform path planning. The single-particle path planning is specifically described below: If a matched particle and an end point are found, then: Step 1, record the matched particle ID and end point coordinates; Step 2, calculate the safety radius of the particle; Step 3, remove the mask of the end point; Step 4, perform path planning. If the path planning is successful: store the path information in the path list, i.e. path_list, and delete the matched end point from the existing end point table, i.e. end_points_now, and add the end point to the selected end point set, to prepare for the particle loss caused by possible future interference, when the subsequent particle loses the match with the end point, the end point will be released from the selected end point, so as to find a new particle to match with it again; otherwise, mark the particle as unselectable and delete the related record.

[0029] In some specific embodiments, when performing path planning, a scaling factor can be determined based on the size information of the target particles and / or obstacles (e.g., microstructures in the microfluidic chip, interfering particles, etc.), and the map can be scaled according to the determined scaling factor. The original size of the path planning is generally 1920x1200, and the calculation amount is very large. Therefore, if the map can be scaled by a certain multiple (e.g., ten times, or other multiples) according to the determined scaling factor before path planning, the calculation amount can be reduced, and at the same time, the longest operation time can be limited, generally limited to 0.2 seconds. When the path planning operation time exceeds this value, it means that the end point is probably unreachable, so the path planning under the current state is abandoned, and the end point is added to the set of end points that cannot be discovered. After a certain number of frames (usually 30 frames), it is re-added to the set of end points that can be matched.

[0030] Since the result of the light path modulation of the hologram will have a certain degree of scaling and translation, in order to ensure the stability of the manipulation, a correction matrix will be made according to the position of the light spot, and an adaptive matrix will be made to ensure the accuracy of the light spot. In addition, accurate control of the position of the light spot also requires pre-allocating light spot power to avoid possible problems such as damage, ejection, escape of target particles caused by too large fluctuations in light spot intensity.

[0031] Regarding the correction matrix, specifically, since the laser incidence will have a certain deformation, it is not possible to accurately match by simply scaling, so scaling will be performed and then the calculation of the correction matrix will be performed. The correction matrix acts on the target image and the transformation of the imaging, so that each light spot in the field matches the target image.

[0032] Regarding the pre-allocation of light spot power, specifically, since the laser incidence power is controlled by an attenuator, when the number of particles increases, the energy obtained by each particle will decrease, so a specified number of light spots can be reserved in the field and outside the field. When the number of particles in the field increases, the reserved positions of the light spots outside the field are allocated into the field, so even if the power is large at the beginning, the light intensity borne by the particles is relatively uniform due to the allocation of a large number of reserved light spots, thereby reducing the probability of ejection.

[0033] If there is a planned path, then start generating the hologram. The following steps need to be performed before generating the hologram: Step 1, initialize related variables; Step 2, traverse all planned paths.

[0034] If the particle is still in the field of view: perform path correction; calculate the safety distance; perform obstacle avoidance processing. If the obstacle avoidance is successful, update the path information, mark the particle as being controlled, record the end point position, add the path to the drawing list (path_list); otherwise: release the end point, delete the related records.

[0035] In the path planning process and in the process of obstacle avoidance, the particle speed can be updated in real time, the path to be passed through in the future is sampled, and the distance is divided by the current speed to obtain the time required to reach the sampling point in the future. In the subsequent planning, a prediction is made according to the above-mentioned time, thereby reducing the probability of future conflicts.

[0036] If the particle disappears, the disappearance counter is increased to record the number of frames in which the particle disappears. If the disappearance exceeds 20 frames, the end point corresponding to the disappearing particle is released, and the relevant record is deleted.

[0037] In addition, if there is a particle being controlled, the interfering particle is processed. Specifically, when an interfering particle (such as a cell not being manipulated) approaches a particle being manipulated, the interfering particle is pushed away. If necessary, a path for the interfering particle to leave can be planned to avoid affecting other manipulated particles during the leaving process. For example, as shown in FIG. 6, for an interfering particle (white particle in the figure) approaching a particle being manipulated (black particle in the figure), the interfering particle is pushed away in a direction away from the manipulated particle (the arrow in the figure indicates the pushing direction). Two optical tweezers are placed in the direction in which the interfering particle is pushed away, so that the interfering particle is subjected to a stronger gradient force, thereby quickly solving the interference. At the same time, in order to reduce the influence of other optical tweezers on the particle during movement, two optical tweezers are also placed in front of the moving direction of the particle. Therefore, when the particle passes through the end point, since only one optical tweezer is placed at the end point of the cell, there is a stronger gradient force in front of the moving particle, thereby reducing the interference. Figure 2

[0038] The path planning in the embodiment of the present application is a method of following the global optimum, and the path planning in the embodiment of the present application is sequentially planned by a single agent, and specifically, the path is planned frame by frame according to the current situation.

[0039] Step 4, generating a hologram based on the currently planned path. Specifically, a GS algorithm (Gerchberg-Saxton algorithm) can be used to generate a hologram.

[0040] Step 5, performing light control based on the currently generated hologram to manipulate the target particles of the planned path. Specifically, the light field is formed based on the hologram to manipulate the target particles.

[0041] The steps 1 to 5 are repeated until the path planning of all the target particles is completed, and then the steps 4 and 5 are repeatedly executed until all the target particles move to the corresponding end point positions.

[0042] ​In the related art, in the process of manipulating multiple micro-particles simultaneously using an optical tweezer system, a global batch path planning scheme is adopted, and most of the time, the path is calculated only after the micro-particle is bound, so as to avoid the movement of the micro-particle during the path calculation process, which may cause the path calculated to be invalid. Therefore, the low-delay cold start is not realized in the related art. The embodiment of the present application adopts an incremental path planning scheme, and the path is planned for only one micro-particle (target particle) each time. In other words, the path of one micro-particle is planned first, and then the path of another micro-particle is planned. As long as the path of one micro-particle is planned, a hologram is generated based on the planned path and optical control is performed, so as to realize the manipulation of the micro-particle (i.e., the target particle) on the planned path. That is, as long as the path of one micro-particle is planned, the manipulation process is started. Therefore, the cold start can be compressed to the millisecond level. The embodiment of the present application also eliminates the correlation between the path planning cold start time and the number of target particles. In the related art, when the number of target particles reaches 50, the cold start delay will exceed 500 ms, and when the number of target particles reaches 100, the cold start delay needs 1.2 s. However, using the micro-particle manipulation method provided by the embodiment of the present application, even if the number of target particles reaches 100, the cold start delay will be stabilized at 100 ms.

[0043] In addition, in the embodiment of the present application, as long as the path of one micro-particle is planned, the manipulation process is started. Subsequently, the path planning of other micro-particles (other target particles whose paths have not been planned) is performed while the particles are being manipulated to move. That is, for the other micro-particles whose paths are planned later, the path is planned based on the latest particle environment (specifically, based on the latest micro-particle image). Therefore, even if the particle environment changes compared to the initial environment, it does not affect the planned path. Therefore, compared with the related art, the embodiment of the present application can dynamically respond to environmental changes, and the path planning can be performed without binding the micro-particles.

[0044] In some optional specific embodiments, the micro-particle manipulation method provided by the embodiment of the present application further includes: For the second target particle whose path has been planned, if it is identified that the second target particle has a collision risk with other particles, a new path is planned for the second target particle to avoid the collision. The other particles include other target particles or interference particles other than the second target particle.

[0045] In the embodiment of the present application, no matter in the process of moving the particle while planning the path of other micro-particles (other target particles which have not been planned a path) or after the paths of all target particles have been planned, as long as it is identified that the second target particle which has been planned a path exists a collision risk with other particles, a new path will be planned for the second target particle to avoid the collision. That is, the embodiment of the present application establishes a dynamic interference suppression mechanism, and improves the generalization of the micro-particle manipulation method and the stability of the manipulation process.

[0046] In the embodiment of the present application, the process of manipulating the movement of the target particle is step by step. Specifically, in the process of manipulating the movement of the target particle, the following process needs to be repeated multiple times: generating a hologram based on the planned path, and completing the one-step movement process of the target particle based on the light control of the hologram. In this process, a high-speed CMOS camera is also used to collect micro-particle images in real time. Therefore, in the process of manipulating the target particle, it can be determined in real time whether the currently manipulated target particle has a collision risk according to the collected micro-particle images. For example, if the distance between other particles and the target particle is less than or equal to 15 μm, it is determined that the target particle has a risk of collision with other particles.

[0047] In some other optional specific embodiments, the micro-particle manipulation method provided by the embodiment of the present application further includes: Step one, if it is determined based on the current micro-particle image that there is a target interference particle, the target interference particle is an interference particle that will affect the movement or identification of the target particle. Specifically, some target interference particles may collide with the moving target particle, that is, they will affect the movement of the target particle; some particles, although they will not collide with the moving target particle, are too close to the target particle to affect the identification of the target particle (for example, they may identify the target particle and the interference particle as a whole, that is, identify them as another kind of particle). This situation is generally because the interference particle did not exist when the path of the target particle was planned, it is a new interference particle that appeared later, or the interference particle existed when the path of the target particle was planned, but it was not identified, or the interference particle existed when the path of the target particle was planned, and it was avoided during path planning, but the position of the interference particle changed over time. Among them, the identification of the interference particle affecting the movement of the target particle (that is, it may collide with the target particle) can also be identified by distance, that is, if the distance between the interference particle and the target particle is less than a preset threshold, it is determined that the interference particle affects the movement of the target particle.

[0048] Step two, planning a moving path for the target interfering particle; when generating a new hologram, based on the current planned path of the target particle and the moving path of the interfering particle, a new hologram is generated.

[0049] In the embodiments of the present application, whether in the process of planning the path of other micro-particles (other target particles whose paths have not been planned) while manipulating the movement of particles, or after the paths of all target particles have been planned, as long as an interfering particle that may affect the movement or identification of a target particle or a few target particles is identified, a moving path is planned for the interfering particle. Of course, in the process of planning the moving path of the interfering particle, not only does it need to avoid affecting the original target particle that may be affected, but it also needs to avoid affecting other target particles during its movement. That is, when planning the moving path of the interfering particle, it needs to avoid affecting any target particle during its movement or after its movement.

[0050] From the perspective of moving the interfering particle, the embodiments of the present application establish a dynamic interference suppression mechanism, improving the generalization of the micro-particle manipulation method and the stability of the manipulation process. One test data shows that the interference events are reduced by 82%, and the success rate of particle manipulation is improved from 68% to 93% (yeast cell line test data).

[0051] In some optional specific embodiments, step 3, i.e., planning a path for the first target particle identified based on the current micro-particle image, includes: Step 31, if a plurality of target particles whose paths have not been planned are identified based on the current micro-particle image, the first target particle is determined based on the priority of the target particles; the priority of the target particles is determined based on at least one of the following: starting position, ending position, size, and identification confidence. The priority of the target particles indicates the degree of urgency of manipulation. As described above, if the priority of the target particles is determined based on the ending position, one of the specific determination principles is the center priority principle.

[0052] Step 32, planning a path for the first target particle based on the current micro-particle image.

[0053] In the embodiments of the present application, the priority of the target particle can be determined in advance based on the start position or end position of the target particle or other factors, and then when multiple target particles that have not been planned a path are identified based on the current microscopic particle image, the path of which target particle is planned first can be determined according to the priority determined in advance. Ideally, the path is planned one target particle after another according to the priority order determined in advance. However, if a target particle with a high priority order is not identified (the path of the target particle has not been planned) when the target particle is identified based on the current microscopic particle image, the target particle with the highest priority among the target particles that have not been planned a path can be selected to plan a path, or the path can be temporarily not planned, and the new microscopic particle image can be waited for N (N is a natural number greater than or equal to 1) frames. If a target particle with a high priority order is not identified (the path of the target particle has not been planned) in the new microscopic particle image for N frames, the target particle with the highest priority among the target particles that have not been planned a path can be selected to plan a path.

[0054] In addition, the path planning in the embodiments of the present application is dynamically performed in real time throughout the whole process, and the path can be adjusted in real time according to the state of the current target particle and the surrounding environment.

[0055] In related technologies, when a hologram is generated, a Fourier iterative algorithm based on CPU (such as the Gerchberg-Saxton algorithm, i.e., the GS algorithm) is used. This method has low efficiency in generating a hologram, and the calculation time is about 2.5 seconds under a 2048x2048 resolution, which cannot meet the real-time interaction requirement. Therefore, the hologram generation method based on CUDA acceleration is used in the embodiments of the present application to break through the CPU calculation bottleneck and realize millisecond-level hologram generation. The method is described in detail as follows.

[0056] In an alternative specific embodiment, step 5, i.e., generating a hologram based on the current planned path, includes: The Fourier iterative operator based on CUDA encapsulation is used to generate the hologram corresponding to the current planned path.

[0057] CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by NVIDIA, which allows developers to utilize the powerful computing capabilities of GPUs using C / C++ language. Unlike traditional CPU programming, CUDA programming requires developers to understand the parallel computing architecture of GPUs. The core concepts of CUDA programming include kernel functions, threads, thread blocks, and grids. Kernel functions are functions that run on GPUs and are executed by multiple threads simultaneously. Threads are the smallest execution units in CUDA, and multiple threads form a thread block, while multiple thread blocks form a grid. This hierarchical structure allows CUDA to flexibly handle computing tasks of different scales. In practical programming, a CUDA program usually follows the following process: first, data is prepared on the CPU, then the data is copied from the CPU memory to the GPU memory, then the kernel function is launched to allow the GPU to process the data in parallel, and finally the results are copied back from the GPU memory to the CPU memory. Developers need to pay special attention to memory management and data transfer overheads. The advantage of CUDA programming is that it can significantly accelerate compute-intensive tasks, especially those that can be decomposed into a large number of independent small tasks. However, CUDA programming also faces some challenges, such as the need to reasonably decompose problems into parallel tasks, the need to consider the memory architecture of the GPU to achieve optimal performance, and the need to handle data transfer between the CPU and the GPU.

[0058] Specifically, the core logic of generating a hologram using the Fourier iteration method is as follows: Step 1, randomly initialize the phase; Step 2, perform Fourier transform on the initialized phase and amplitude; Step 3, normalize and calculate the correlation; Step 4, obtain the phase in Step 2; Step 5, perform inverse Fourier transform on the target amplitude and phase; Repeat the above steps 2-5 until the correlation reaches the target value; Finally, extract the phase information in Step 5 and normalize it, and the corresponding hologram can be obtained.

[0059] In an alternative embodiment, the Fourier iteration operator based on CUDA encapsulation is used to generate the hologram corresponding to the current planned path, which specifically includes: Copy the target amplitude matrix on the CPU to the GPU memory; Using the Fourier iteration method, generate the hologram corresponding to the current planned path based on the target amplitude matrix; Copy the generated hologram back to the CPU.

[0060] Specifically, a CUDA encapsulated based Fourier iterative operator is used to generate the hologram corresponding to the current planned path, as shown in the following: Figure 3 The specific process (CUDA encapsulated logic) for generating the hologram corresponding to the current planned path is as follows: 1. Initialization stage: Calculate the image size size = width × height; Allocate memory on the GPU: d_amplitude, d_g0_GS, d_phase_image; Copy CPU data h_amplitude (i.e., target amplitude distribution data) to GPU memory d_amplitude.

[0061] 2. Random phase initialization, i.e., form initialization phase distribution: Launch the init_random_phase kernel function; Each thread generates a random phase and initializes the complex number array d_g0_GS.

[0062] 3. Set FFT plan: Create a 2D FFT plan (cufftPlan2d).

[0063] 4. Iterative processing (loop iterations times): a. Perform forward FFT transform (cufftExecC2C); b. Calculate the current correlation (reserve correlation_kernel); c. Check if the target correlation is reached: if so, print the relevant information and break out of the loop; i.e., if the iteration stop condition is met, break out of the loop; d. Normalize the amplitude (normalize_amplitude kernel function); e. Perform inverse FFT transform (cufftExecC2C); f. Add random perturbation (reserve add_random_perturbation kernel function).

[0064] 5. Phase extraction: Launch the extract_phase kernel function; Each thread extracts phase information from the complex number data, i.e., outputs the target phase.

[0065] 6. Result processing: Copy phase data from GPU back to CPU; Normalize phase data ([0, 2π] -> [0, 1]).

[0066] At this point, the hologram has been generated.

[0067] 7. Cleanup phase: Destroy the FFT plan; Free all GPU memory.

[0068] In the embodiment of the application, the Fourier iterative operation is performed using the Fourier transform interface of CUDNN. CUDNN (CUDADeep Neural Network library) is a library built based on CUDA, which is optimized for deep learning applications. It provides a series of highly optimized routines for common operations in deep neural networks, such as convolution, pooling, activation functions, etc. Using CUDNN, developers can easily implement high-performance training and inference in their deep learning models without delving into the details of underlying parallel algorithms or hardware. The main advantages of CUDNN are its high performance and ease of use. Since it is specifically designed for deep learning, it is more efficient than directly using CUDA when dealing with related tasks.

[0069] In the embodiment of the application, the GS algorithm is also used to generate the hologram, but based on CUDA acceleration, which can make the generation speed of the hologram increase by 100 times compared with the traditional hologram generation algorithm. One experimental data shows that the hologram generation time can be reduced from 2500ms (CPU) to 22.3±1.5ms (GPU), i.e. the hologram generation speed is increased by about 112 times. The system response frequency reaches 10Hz, supporting 50+ target particle synchronous operation.

[0070] In the micro-particle manipulation method provided by the embodiment of the application, the target particle can be a cell, so the embodiment of the application can be applied to the cell precision operation scene in the biomedical field, including but not limited to cell sorting, cell fusion and other applications.

[0071] In the embodiment, a micro-particle manipulation device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0072] The embodiment provides a micro-particle manipulation device, as shown in Figure 4 includes: An image acquisition module 201 is configured to acquire a current micro-particle image. an identification module 202, configured to identify a target particle in the current micro-particle image, the target particle being a particle as a manipulation object; a path planning module 203, configured to plan a path for a first target particle in the current micro-particle image, the first target particle being one of the target particles without a planned path; a generation module 204, configured to generate a hologram based on the currently planned path; a light control module 205, configured to perform light control based on the currently generated hologram, to realize manipulation of the target particles with the planned paths; a control module 206, configured to control the image acquisition module 201, the identification module 202, the path planning module 203, the generation module 204, and the light control module 205 to repeatedly run until path planning for all the target particles is completed, and control the generation module 204 and the light control module 205 to repeatedly run until all the target particles move to corresponding end positions.

[0073] In some optional embodiments, the path planning module 203 is further configured to, for a second target particle with a planned path, plan a new path for the second target particle to avoid collision if it is identified that the second target particle has a collision risk with other particles. The other particles include other target particles or interference particles other than the second target particle.

[0074] In some optional embodiments, the path planning module 203 is further configured to, if it is determined based on the current micro-particle image that there is a target interference particle, plan a moving path for moving away the target interference particle, the target interference particle being an interference particle that affects movement or identification of the target particles, and generate a new hologram based on the currently planned path of the target particles and the moving path of the interference particles when generating the new hologram.

[0075] In some optional embodiments, the path planning module 203 includes: a determination unit, configured to, if a plurality of target particles without a planned path are identified based on the current micro-particle image, determine the first target particle based on a priority of the target particles, the priority of the target particles being determined based on at least one of a start position, an end position, a size, and an identification confidence. a planning unit, configured to plan a path for the first target particle based on the current micro-particle image.

[0076] In some optional implementations, the generating module 204 is specifically configured to generate the hologram corresponding to the currently planned path using a Fourier iteration operator based on a CUDA package.

[0077] In some optional implementations, the generating module 204 specifically includes: A first copy unit is used to copy the target amplitude matrix on the CPU to the GPU memory; a generating unit, configured to generate a hologram corresponding to the currently planned path based on the target amplitude matrix using a Fourier iteration method; The second copying unit is used to copy the generated hologram back to the CPU.

[0078] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0079] The microscopic particle manipulation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0080] The embodiment of the present invention also provides an optical tweezers system having the above Figure 4 The microscopic particle manipulation device shown.

[0081] See also Figure 5 , Figure 5 FIG. 1 is a schematic structural diagram of an optical tweezers system provided by an optional embodiment of the present invention. Figure 5 As shown, the optical tweezers system includes one or more processors 10, a memory 20, and interfaces for connecting the various components, including high-speed and low-speed interfaces. The various components communicate with each other using different buses and can be installed on a common motherboard or in other ways as needed. The processor can process instructions executed within the optical tweezers system, including instructions stored in or on the memory for displaying graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if desired, multiple processors and / or multiple buses can be used with multiple memories and multiple memories. Figure 5 A processor 10 is taken as an example.

[0082] The processor 10 can be a central processor, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic gate array, a generic array logic, or any combination thereof.

[0083] The memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform the methods illustrated in the above embodiments.

[0084] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system and application programs required by at least one function. The data storage area can store data created according to the use of the optical tweezer system, etc. In addition, the memory 20 can include a high-speed random access memory and can further include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some optional embodiments, the memory 20 can optionally include a memory disposed remotely with respect to the processor 10, and these remote memories can be connected to the optical tweezer system through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0085] The memory 20 can include a volatile memory, such as a random access memory, and can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state disk. The memory 20 can further include a combination of the above types of memories.

[0086] The optical tweezer system further includes a micro-particle image acquisition assembly 50 and an optical assembly 60. Specifically, the micro-particle image acquisition assembly can be a high-speed CMOS camera. The optical assembly can be a phase-type liquid crystal on silicon - spatial light modulator (LCOS-SLM) (3840x2160, 60Hz refresh rate).

[0087] The optical tweezer system further includes a graphics processing unit (GPU), which can be, for example, a NVIDIA Ada Lovelace architecture GPU (RTX 4090D).

[0088] The embodiments of the present application provide an efficient optical tweezer system combining GPU parallel computing and intelligent path planning.

[0089] The optical tweezer system also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 can be connected by a bus or other means, Figure 5 The bus connection is taken as an example.

[0090] The input device 30 can receive inputted digital or character information, and generate key signal input related to user settings and function control of the optical tweezer system, such as touch screen, keypad, mouse, trackpad, touchpad, pointing stick, one or more mouse buttons, trackball, joystick, etc. The output device 40 can include display device, auxiliary lighting device (e.g. LED), and tactile feedback device (e.g. vibration motor), etc. The display device includes but is not limited to liquid crystal display, light emitting diode, display and plasma display. In some optional embodiments, the display device can be a touch screen.

[0091] The optical tweezer system also includes a communication interface for the optical tweezer system to communicate with other devices or communication networks.

[0092] The embodiments of the present application also provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded from a network and stored in a local storage medium by computer code, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor or programmable or special purpose hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk or a solid state disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that the computer, processor, microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor or hardware, the method shown in the above embodiments is implemented.

[0093] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, can invoke or provide the method and / or technical solutions according to the present application. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source files, executable files, installation package files and the like, and accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0094] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method for manipulating microscopic particles, characterized in that: The method comprises: Step 1: Obtain the current microscopic particle image; Step 2: Identify target particles in the current microscopic particle image, where the target particles are particles to be manipulated; Step 3: for the first target particle currently identified, planning a path for the first target particle based on the current microscopic particle image; the first target particle is a target particle for which a path has not been planned; Step 4: Generate a hologram based on the currently planned path; Step 5: Performing light control based on the currently generated hologram to manipulate the target particles along the planned path; Repeat steps 1 to 5 until all the target particles have completed path planning; and repeat steps 4 and 5 until all the target particles have moved to their corresponding end positions.

2. The method according to claim 1, characterized in that Also includes: For a second target particle with a planned path, if it is identified that the second target particle has a collision risk with other particles, a new path is planned for the second target particle to avoid collision; The other particles include other target particles or interfering particles other than the second target particles.

3. The method according to claim 1, characterized in that Also includes: If, based on the current microscopic particle image, it is determined that there are target interfering particles, a moving path is planned to remove the target interfering particles; the target interfering particles are interfering particles that will affect the movement or identification of the target particles; when generating a new hologram, the new hologram is generated based on the currently planned path of the target particles and the moving path of the interfering particles.

4. The method according to claim 1, wherein The step of planning a path of the first target particle currently identified based on the current microscopic particle image includes: If a plurality of target particles that are not included in the planned path are identified based on the current microscopic particle image, the first target particle is determined based on a priority of the target particles; the priority of the target particle is determined based on at least one of the following: a starting position, an end position, a size, and a recognition confidence; Based on the current microscopic particle image, a path of the first target particle is planned.

5. The method according to claim 1, wherein The generating of a hologram based on the currently planned path includes: The hologram corresponding to the currently planned path is generated by using a Fourier iteration operator based on CUDA encapsulation.

6. The method according to claim 5, characterized in that The generating the hologram corresponding to the currently planned path by using a Fourier iteration operator based on a CUDA package includes: Copy the target amplitude matrix on the CPU to the GPU memory; Generate a hologram corresponding to the currently planned path based on the target amplitude matrix using a Fourier iteration method; The generated hologram is copied back to the CPU.

7. A microscopic particle manipulation device, characterized in that: The device comprises: An image acquisition module is used to acquire the current microscopic particle image; an identification module, configured to identify target particles in the current microscopic particle image, wherein the target particles are particles to be manipulated; a path planning module, configured to plan a path for a currently identified first target particle based on the current microscopic particle image; the first target particle being a target particle for which a path has not been planned; A generation module, configured to generate a hologram based on the currently planned path; a light control module, configured to perform light control based on the currently generated hologram to manipulate the target particles along the planned path; The control module is used to control the image acquisition module, the recognition module, the path planning module, the generation module and the light control module to repeatedly operate until the path planning of all the target particles is completed; and to control the generation module and the light control module to repeatedly operate until all the target particles move to the corresponding end positions.

8. An optical tweezers system, characterized in that: include: Microscopic particle image acquisition component; Optical components; A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the microscopic particle manipulation method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the microscopic particle manipulation method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the microscopic particle manipulation method according to any one of claims 1 to 6.

Images