Photoelectric tweezers microparticle array system and method

Abstract

The present invention provides a photoelectric tweezers microparticle arraying system and method, and the system includes: the microparticle arraying system includes a visualization module, a memory communication module, an arraying main control module, a microparticle detection submodule, an arraying pattern submodule, a target allocation submodule, a path planning submodule and a light projection module. The photoelectric tweezers microparticle arraying system, which integrates arraying pattern generation, microparticle detection and path planning, can generate various desired patterns according to user needs, greatly simplifying the arraying data preparation process. Then, an efficient detection algorithm is used to detect the position of the microparticles in the photoelectric tweezers chip, the starting position and the target position are matched from a global optimal angle, and a motion strategy is generated according to the RVO path planning method, and then the photoelectric tweezers are controlled to transport the microparticles to the target position, thereby forming a target arraying pattern, which optimizes the response time of the arraying system as a whole, and ensures the real-time performance of device control and data processing when the number of microparticles is large.

Description

Technical Field

[0001] The invention relates to the technical field of photoelectric tweezers control, and in particular to a photoelectric tweezers particle arraying system and method. Background Art

[0002] The existing monoclonal cell line development technologies, including limiting dilution and flow sorting, can obtain single cells, but they often damage the cells and interfere with the normal state of the cells. Therefore, the proportion of clones formed is very low. The photoelectric tweezers single-cell screening device is currently the only comprehensive system that can directly and non-destructively manipulate single cells and perform fully automatic culture, detection, cloning, export and in-depth research. However, the photoelectric tweezers single-cell screening device is a new device. The existing photoelectric tweezers array manipulation technology requires manual preparation of array data in advance, which is cumbersome and time-consuming. The number of particles that can be manipulated at the same time is small and most of them are manually manipulated. The particle manipulation speed is slow, making the photoelectric tweezers array manipulation inconvenient and inefficient.

[0003] Existing patent: CN111908421A A micro-nano self-assembly operating system and method based on light-induced dielectrophoresis. By making an ODEP (Optically Induced Dielectrophoresis) chip, a liquid sample containing micro-nano particles is injected into the solution layer of the prepared ODEP chip; the image of the micro-nano particles injected into the channel of the ODEP chip is obtained, and the image of the micro-nano particles is collected in real time; the real-time collected image is processed and the target is identified to obtain the information of the identified micro-nano particles; an external electric field is added to the two layers of ITO glass in the ODEP chip, and the drawn optical patterns are sequentially projected onto the positions of the identified micro-nano particles to be operated, and the movement of the optical pattern is controlled to achieve the setting operation of the micro-nano particles.

[0004] The above-mentioned existing patent detection method focuses on processing the image itself, and uses image processing methods such as image enhancement and morphological processing to improve the detection effect. However, there are too many processing steps, which causes the detection to take too long.

[0005] Moreover, in the above-mentioned existing patent, the matching method of the starting position of the micro-nano particle and the target position is: indexing the target position in sequence, and then selecting the micro-nano particle with the closest distance for matching. This method is inefficient and only considers the local optimum.

[0006] In addition, in the above-mentioned existing patent, the micro-nano particles are moved in the following manner: first, the optical pattern is moved according to the horizontal coordinate until the horizontal coordinates of all matching particle positions reach the horizontal coordinate of the target position, and then the vertical coordinate is changed at the same time using the above steps, resulting in the following problems in the existing patent:

[0007] The moving speed is slow, and the straight line between two points is the shortest. For example, if the target position of a micro-nano particle is in the upper right corner, the technology needs to move horizontally to the right first, and then move vertically upward, and it cannot move diagonally directly to the target position.

[0008] It is easy to produce collisions and conflicts, and is only applicable to a small number of particles. If the number of micro-nano particles is large, or multiple nanoparticles are located in the same horizontal or vertical direction, collisions are very likely to occur. It is impossible to make autonomous adjustments and decisions based on the particle situation and environment. Therefore, the present invention proposes a photoelectric tweezers particle array system and method. Summary of the invention

[0009] The present invention provides a photoelectric tweezers microparticle arraying system and method to solve the above problems.

[0010] The present invention provides a photoelectric tweezers microparticle array system, comprising:

[0011] A memory communication module, used for receiving image data from the photoelectric tweezers single-cell screening device and transmitting the image data to the visualization module and the arrayed main control module respectively;

[0012] Visualization module, used to visualize image data;

[0013] The arrayed main control module is used to control the particle detection submodule, the arrayed pattern submodule, the target allocation submodule and the path planning submodule to work together to obtain the arrayed image and the next planned position of the particles to be projected;

[0014] The light projection module is used to generate a light projection adaptation map of the photoelectric tweezers single-cell screening device based on the arrayed image and the next planned position of the particles to be projected, and send the light projection adaptation map to the photoelectric tweezers single-cell screening device to generate corresponding light projection.

[0015] Preferably, in an optoelectronic tweezers microparticle arraying system, the arraying main control module comprises:

[0016] The particle detection submodule is used to perform particle detection on the received image data and obtain particle information;

[0017] An arrayed pattern submodule, for generating an image according to a text description input by a user through an image generation model, processing the generated image based on a grid sampling method to obtain an arrayed image, and determining coordinate data of each target point from the arrayed image;

[0018] A target allocation submodule, used for performing one-to-one position allocation between effective particles and target points in the arrayed image;

[0019] The path planning submodule is used to determine the particle to be projected according to the position allocation result, the particle coordinates in the particle information and the coordinate data of the target point, and calculate the next planned position of the particle to be projected according to the RVO algorithm.

[0020] Preferably, in an optoelectronic tweezers microparticle array system, the microparticle detection submodule comprises:

[0021] An image data processing unit, used to first convert the image data into a grayscale image and threshold the grayscale image, and perform particle contour recognition based on the thresholded grayscale image;

[0022] A particle contour processing unit, used for generating a minimum circumscribed circle of a corresponding particle based on a particle contour recognition result;

[0023] A particle screening processing unit is used to compare the particle morphology with the minimum circumscribed circle morphology, and screen the radius and circularity of the particles in the grayscale image according to the comparison result to determine the valid particles on the grayscale image;

[0024] The particle information collection unit is used to collect the particle coordinates and radius of the effective particles and generate particle information.

[0025] Preferably, in an optoelectronic tweezers microparticle arraying system, the arraying pattern submodule comprises:

[0026] An image generation unit, used to perform semantic recognition on the text description input by the user based on the image generation model, and generate a corresponding image according to the semantic recognition result;

[0027] An image arraying unit, used for performing binarization processing on the generated image to obtain a binarized image, and generating a grid on the binarized image based on an arbitrarily adjustable grid spacing to obtain an arrayed image;

[0028] The target point screening unit is used to screen each pixel point in the grid on the arrayed image one by one. When the value corresponding to the pixel point meets the preset conditions, the pixel point is regarded as a valid particle that needs to be arrayed and the coordinate data of the valid particle that needs to be arrayed is collected.

[0029] Preferably, in an optoelectronic tweezers microparticle array system, the target allocation submodule comprises:

[0030] A cost matrix building unit, used to build a movement cost matrix corresponding to each valid particle based on particle information and coordinate data of arrayed target points;

[0031] A position allocation determination unit, for determining the optimal allocation position of each effective particle that needs array manipulation in the array image based on the movement cost matrix and using the Jonker-Volgenant algorithm;

[0032] The array position allocation unit is used to establish a one-to-one correspondence between the effective particles that need to be arrayed and the target points of the arrayed image based on the optimal allocation positions of the effective particles that need to be arrayed.

[0033] The present invention provides a method for forming an array of photoelectric tweezer particles, comprising:

[0034] Initialize and reset each functional module, and control the memory communication module to wait for receiving image data;

[0035] After the photoelectric tweezers single-cell screening device is started, it automatically captures images in real time, packages the image data and transmits it to the memory for storage, and synchronously sends an event ready signal to the memory communication module;

[0036] After receiving the event ready signal, the memory communication module starts to continuously read the memory, obtains the image data packet, unpacks and parses the image data into image data, and sends it to the arrayed main control module;

[0037] After receiving the image data, the arraying main control module performs an arraying operation on the image data to obtain an arrayed image and the next planned position of the particles to be projected;

[0038] The light projection module generates a light projection adaptation map for the optoelectronic tweezers single-cell screening device based on the arrayed image and the next planned position of the particles to be projected, and sends the light projection adaptation map to the optoelectronic tweezers single-cell screening device to generate corresponding light projection.

[0039] Preferably, in a method for arraying photoelectric tweezer particles, after receiving an image, the arraying main control module performs an arraying operation on the image to obtain the arrayed image and the next planned position of the particles to be projected, including:

[0040] After receiving the image data, the array main control module calls the particle detection submodule to perform particle detection on the received image data and obtain particle information;

[0041] The arraying main control module calls the arraying pattern submodule to generate an arraying image, and determines the coordinate data of each target point from the arraying image;

[0042] The arraying main control module calls the target allocation submodule to perform one-to-one position allocation between the effective particles and the target points in the arraying image;

[0043] The arraying main control module calls the path planning submodule to determine the particles to be projected according to the position allocation results, the particle coordinates in the particle information and the coordinate data of the target point, and calculates the next planned position of the particles to be projected according to the RVO algorithm.

[0044] Preferably, in a method for arraying photoelectric tweezers particles, after receiving the image data, the arraying main control module calls the particle detection submodule to perform particle detection on the received image data and obtain particle information, including:

[0045] First, the image data is converted into a grayscale image and the grayscale image is thresholded, and particle contours are identified based on the thresholded grayscale image;

[0046] Generate the minimum circumscribed circle of the corresponding particle based on the particle contour recognition result;

[0047] The particle morphology is compared with the minimum circumscribed circle morphology, and the radius and roundness of the particles in the grayscale image are screened according to the comparison results to determine the valid particles on the grayscale image;

[0048] The particle coordinates and radius of the effective particles are collected to generate particle information.

[0049] Preferably, in a method for arraying photoelectric tweezers microparticles, the arraying main control module calls the arraying pattern submodule to generate an arrayed image, and determines the coordinate data of each target point from the arrayed image, including:

[0050] Based on the image generation model, semantic recognition is performed on the text description input by the user, and the corresponding image is generated according to the semantic recognition result;

[0051] Binarizing the generated image to obtain a binary image, and generating a grid on the binary image based on an arbitrarily adjustable grid spacing to obtain an arrayed image;

[0052] Each pixel point in the grid on the arrayed image is screened one by one. When the value corresponding to the pixel point meets the preset condition, the pixel point is regarded as a valid particle that needs to be arrayed and manipulated, and the coordinate data of the valid particle that needs to be arrayed is collected.

[0053] Preferably, in a method for arraying photoelectric tweezers microparticles, the arraying main control module calls the target allocation submodule to perform one-to-one position allocation between the effective microparticles to be arrayed and the target points in the arrayed image, including:

[0054] A cost matrix building unit, used for building a movement cost matrix based on particle information and coordinate data of arrayed target points;

[0055] Based on the moving cost matrix, the Jonker-Volgenant algorithm is used to determine the optimal allocation position of each effective particle that needs to be arrayed and manipulated in the arrayed image.

[0056] Based on the optimal allocation positions of the effective microparticles that need to be manipulated in an arrayed manner, a one-to-one correspondence is established between the effective microparticles that need to be manipulated in an arrayed manner and the target points of the arrayed image.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The image data is received from the optoelectronic tweezers single-cell screening device through the memory communication module and passed to the visualization module and the arraying main control module; then the arraying main control module completes the process control of the arraying process, calls the particle detection submodule, arraying pattern submodule, target allocation submodule and path planning submodule to complete each link of the arraying; and the image data is visualized through the visualization module. The arraying main control module controls the particle detection module to detect the input image data and obtain the particle position and radius data; at the same time, the arraying main control module controls the arraying pattern module to generate an image according to the user text description through the image generation model, and then uses grid sampling to obtain the pattern dot coordinate data; then the target allocation module is called to perform a one-to-one allocation based on the initial position of the particle detection and the target position of the arraying pattern; finally, the path planning module is called to perform path planning and calculate the next moving position based on the allocation result. The light projection module will convert the image data into a light projection pattern that can be recognized by the optoelectronic tweezers single-cell screening device based on the next moving position and transmit it to the optoelectronic tweezers single-cell screening device to realize the manipulation of optoelectronic tweezers particles. The present invention is an optoelectronic tweezers microparticle arraying system that integrates array pattern generation, particle detection, and path planning. It can generate various desired patterns according to user needs, and extract the component coordinates of the pattern as the array coordinates of the array operation, which greatly simplifies the array data preparation process. Then, an efficient detection algorithm is used to detect the position of the particles in the optoelectronic tweezers chip, and the starting position and the target position are matched from the global optimal angle. Then, a motion strategy is generated according to the RVO path planning method, and then the optoelectronic tweezers are controlled to transport the particles to the target position, thereby forming a target array pattern. The response time of the array system is optimized as a whole, ensuring the real-time performance of device control and data processing when the number of particles is large (more than 50).

[0059] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures specifically pointed out in this application document.

[0060] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0062] Figure 1 It is a structural schematic diagram of a photoelectric tweezers microparticle array system of the present invention;

[0063] Figure 2 This is a schematic structural diagram of a particle detection submodule of an optoelectronic tweezers particle array system of the present invention;

[0064] Figure 3 It is a structural schematic diagram of an arraying pattern submodule of an optoelectronic tweezers microparticle arraying system of the present invention;

[0065] Figure 4 A schematic diagram of the positions of target points of the arrayed image of the present invention;

[0066] Figure 5 This is a schematic structural diagram of a target allocation submodule of an optoelectronic tweezers microparticle arraying system of the present invention;

[0067] Figure 6 The present invention is a flow chart of a method for photoelectric tweezers microparticle arraying. DETAILED DESCRIPTION

[0068] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0069] Embodiment 1:

[0070] The present invention provides a photoelectric tweezers microparticle array system, such as Figure 1 As shown, including:

[0071] A memory communication module, used for receiving image data from the photoelectric tweezers single-cell screening device and transmitting the image data to the visualization module and the arrayed main control module respectively;

[0072] Visualization module, used to visualize image data;

[0073] The arrayed main control module is used to control the particle detection submodule, the arrayed pattern submodule, the target allocation submodule and the path planning submodule to work together to obtain the arrayed image and the next planned position of the particles to be projected;

[0074] A light projection module, for generating a light projection adaptation map of the photoelectric tweezers single-cell screening device based on the arrayed image and the next planned position of the particles to be projected, and sending the light projection adaptation map to the photoelectric tweezers single-cell screening device to generate a corresponding light projection;

[0075] The arrayed main control module includes:

[0076] The particle detection submodule is used to perform particle detection on the received image data and obtain particle information;

[0077] An arrayed pattern submodule, for generating an image according to a text description input by a user through an image generation model, processing the generated image based on a grid sampling method to obtain an arrayed image, and determining coordinate data of each target point from the arrayed image;

[0078] A target allocation submodule, used for performing one-to-one position allocation between effective particles and target points in the arrayed image;

[0079] The path planning submodule is used to determine the particle to be projected according to the position allocation result, the particle coordinates in the particle information and the coordinate data of the target point, and calculate the next planned position of the particle to be projected according to the RVO algorithm.

[0080] In this embodiment, the light projection module is used to generate a light projection adaptation map of the photoelectric tweezers single cell screening device based on the arrayed image and the next planned position of the particle to be projected, and send the light projection adaptation map to the photoelectric tweezers single cell screening device. In the process of generating the corresponding light projection, the circular spot radius of the particles in the adapted projection image is adjusted according to the parameters and state of the device instead of using a fixed radius. Then, the force on the particles is intelligently adjusted, the mutual influence between multiple particles is reduced, and the automation and intelligence of the system are improved.

[0081] In this embodiment, communication with the memory can also be completed through HTTP(S), TCP / IP, MQTT, CoAP, Bluetooth, Wifi, etc.

[0082] In this embodiment, the grid adopting method may also adopt a uniform sampling method, a random sampling method, or directly use a text generation model to generate coordinate point data through text description for replacement.

[0083] The beneficial effects of the above technical solution are as follows: the image data is received from the optoelectronic tweezers single cell screening device through the memory communication module and passed to the visualization module and the arraying main control module; then the arraying main control module completes the process control of the arraying process, calls the particle detection submodule, the arraying pattern submodule, the target allocation submodule and the path planning submodule to complete each link of the arraying; and the image data is visualized through the visualization module. The arraying main control module controls the particle detection module to detect the input image data and obtain the particle position and radius data; at the same time, the arraying main control module controls the arraying pattern module to generate an image according to the user text description through the image generation model, and then uses grid sampling to obtain the pattern lattice coordinate data; then the target allocation module is called to perform a one-to-one allocation based on the initial position of the particle detection and the target position of the arraying pattern; finally, the path planning module is called to perform path planning and calculate the next moving position based on the allocation result. The light projection module will convert the image data into a light projection pattern that can be recognized by the optoelectronic tweezers single cell screening device based on the next moving position and transmit it to the optoelectronic tweezers single cell screening device to realize the manipulation of optoelectronic tweezers particles. The present invention is an optoelectronic tweezers microparticle arraying system that integrates array pattern generation, particle detection, and path planning. It can generate various desired patterns according to user needs, and extract the component coordinates of the pattern as the array coordinates of the array operation, which greatly simplifies the array data preparation process. Then, an efficient detection algorithm is used to detect the position of the particles in the optoelectronic tweezers chip, and the starting position and the target position are matched from the global optimal angle. Then, a motion strategy is generated according to the RVO path planning method, and then the optoelectronic tweezers are controlled to transport the particles to the target position, thereby forming a target array pattern. The response time of the array system is optimized as a whole, ensuring the real-time performance of device control and data processing when the number of particles is large (more than 50).

[0084] Embodiment 2:

[0085] Based on Example 1, the particle detection submodule, such as Figure 2 As shown, including:

[0086] An image data processing unit, used to first convert the image data into a grayscale image and threshold the grayscale image, and perform particle contour recognition based on the thresholded grayscale image;

[0087] A particle contour processing unit, used for generating a minimum circumscribed circle of a corresponding particle based on a particle contour recognition result;

[0088] A particle screening processing unit is used to compare the particle morphology with the minimum circumscribed circle morphology, and screen the radius and circularity of the particles in the grayscale image according to the comparison result to determine the valid particles on the grayscale image;

[0089] The particle information collection unit is used to collect the particle coordinates and radius of the effective particles and generate particle information.

[0090] The beneficial effects of the above technical solution: the present invention simplifies the whole image processing steps and only needs to perform simple grayscale processing to enter the particle detection, reducing the operation of processing the whole image. At the same time, the minimum circumscribed circle, radius, and roundness screening steps are added in the detection process to ensure the detection effect, which reduces the detection time as a whole and is more suitable for high-throughput large-image particle detection.

[0091] Embodiment 3:

[0092] On the basis of Example 1, the arrayed pattern submodule, such as Figure 3 As shown, including:

[0093] An image generation unit, used to perform semantic recognition on the text description input by the user based on the image generation model, and generate a corresponding image according to the semantic recognition result;

[0094] An image arraying unit, used for performing binarization processing on the generated image to obtain a binarized image, and generating a grid on the binarized image based on an arbitrarily adjustable grid spacing to obtain an arrayed image;

[0095] The target point screening unit is used to screen each pixel point in the grid on the arrayed image one by one. When the value corresponding to the pixel point meets the preset conditions, the pixel point is regarded as a valid particle that needs to be arrayed and the coordinate data of the valid particle that needs to be arrayed is collected.

[0096] In this embodiment, the grid point coordinates are defined as in and are the indices of the grid points in the horizontal and vertical directions, is the spacing of the grid, which can be adjusted arbitrarily;

[0097] The grid width, height and number are defined as in is the total size of the image (width times height), and determines the sampling function of the valid particles in the arrayed image that need to be arrayed. as follows:

[0098]

[0099] in is the image at point The value at is the set sampling threshold; It refers to the sampling threshold of the effective particle points that need to be manipulated by array.

[0100] By sampling function , that is, all target points in the arrayed image, such as Figure 4 The points shown in the circles.

[0101] The beneficial effects of the above technical solution are as follows: the present invention generates an image of the target pattern based on a simple text description of the target pattern using an image generation model, and there is no need to extract and select points manually one by one. The generated pattern is processed using a grid method to generate an arrayed pattern and the generated pattern is automatically processed and determined using a grid method, thereby avoiding the tedious operation of manual marking by the user, helping the user to immediately obtain the required arrayed pattern, simplifying the process and improving efficiency.

[0102] Embodiment 4:

[0103] Based on Example 1, the target allocation submodule, such as Figure 5 As shown, including:

[0104] A cost matrix building unit is used to build a movement cost matrix corresponding to each valid particle based on particle information and coordinate data of arrayed target points;

[0105] A position allocation determination unit, for determining the optimal allocation position of each effective particle that needs array manipulation in the array image based on the movement cost matrix and using the Jonker-Volgenant algorithm;

[0106] The array position allocation unit is used to establish a one-to-one correspondence between the effective particles that need to be arrayed and the target points of the arrayed image based on the optimal allocation positions of the effective particles that need to be arrayed.

[0107] In this embodiment, a movement cost matrix is ​​established based on the particle information and the coordinate data of the arrayed target points. The cost matrix is ​​constructed as follows:

[0108]

[0109] is the cost matrix Line The element of the column indicates that the starting point With target point The total cost when matching, the starting point refers to the valid particles that need to be arrayed; is the distance weight; is the weight of the radius difference;

[0110] The distance part uses Minkowski distance, which is a measurement method that can take into account both Euclidean distance and Manhattan distance. is the parameter of the Minkowski distance, when When , it represents the Manhattan distance, which is suitable for special microfluidic chips or confined spaces with obstacles. When , it indicates the Euclidean distance, which is suitable for barrier-free environments;

[0111] Is the starting point In the dimensional coordinates, It is the target point In the dimensional coordinates; n represents the dimension of the coordinate axis;

[0112] Is the starting point The radius of It is the target point The radius difference uses a logarithm to reduce the excessive impact of large radius values ​​on the cost calculation, making the impact of radius differences on the cost matrix smoother.

[0113] In this embodiment, based on the movement cost matrix, the Jonker-Volgenant algorithm is used to determine the optimal allocation position of each effective particle that needs array manipulation in the array image as follows:

[0114]

[0115] in, Indicates the starting point The minimum moving cost is Figure 4 As shown by the white spots on the black particles;

[0116] At the starting point A one-to-one correspondence is established between the target points at the optimal allocation position corresponding to the minimum moving cost;

[0117] Among them, each target point in the arrayed image can only correspond to one valid particle. When a target point at the optimal allocation position corresponding to a small movement cost corresponding to an effective particle that needs arrayed manipulation has been allocated with a valid particle, the effective particle that needs arrayed manipulation can only find the target point corresponding to the minimum movement cost among the remaining target points as the optimal allocation position.

[0118] Beneficial effects of the above technical solution: The present invention uses the Jonker-Volgenant algorithm to complete the optimal allocation of the starting position and the target position of the particles according to the cost matrix, minimize the total distance required for global movement, and reduce the time consumption of arraying.

[0119] Embodiment 5:

[0120] The present invention provides a method for forming an array of photoelectric tweezer particles, such as Figure 6 As shown, including:

[0121] Initialize and reset each functional module, and control the memory communication module to wait for receiving image data;

[0122] After the photoelectric tweezers single-cell screening device is started, it automatically captures images in real time, packages the image data and transmits it to the memory for storage, and synchronously sends an event ready signal to the memory communication module;

[0123] After receiving the event ready signal, the memory communication module starts to continuously read the memory, obtains the image data packet, unpacks and parses the image data into image data, and sends it to the arrayed main control module;

[0124] After receiving the image data, the arraying main control module performs an arraying operation on the image data to obtain an arrayed image and the next planned position of the particles to be projected;

[0125] The light projection module generates a light projection adaptation map for the optoelectronic tweezers single-cell screening device based on the arrayed image and the next planned position of the particles to be projected, and sends the light projection adaptation map to the optoelectronic tweezers single-cell screening device to generate corresponding light projection.

[0126] After receiving the image, the arraying main control module performs an arraying operation on the image to obtain the arrayed image and the next planned position of the particles to be projected, including:

[0127] After receiving the image data, the array main control module calls the particle detection submodule to perform particle detection on the received image data and obtain particle information;

[0128] The arraying main control module calls the arraying pattern submodule to generate an arraying image, and determines the coordinate data of each target point from the arraying image;

[0129] The arraying main control module calls the target allocation submodule to perform one-to-one position allocation between the effective particles and the target points in the arraying image;

[0130] The arraying main control module calls the path planning submodule to determine the particles to be projected according to the position allocation results, the particle coordinates in the particle information and the coordinate data of the target point, and calculates the next planned position of the particles to be projected according to the RVO algorithm.

[0131] In this embodiment, the light projection module is used to generate a light projection adaptation map of the photoelectric tweezers single cell screening device based on the arrayed image and the next planned position of the particle to be projected, and send the light projection adaptation map to the photoelectric tweezers single cell screening device. In the process of generating the corresponding light projection, the circular spot radius of the particles in the adapted projection image is adjusted according to the parameters and state of the device instead of using a fixed radius. Then, the force on the particles is intelligently adjusted, the mutual influence between multiple particles is reduced, and the automation and intelligence of the system are improved.

[0132] In this embodiment, communication with the memory can also be completed through HTTP(S), TCP / IP, MQTT, CoAP, Bluetooth, Wifi, etc.

[0133] In this embodiment, the grid adopting method may also adopt a uniform sampling method, a random sampling method, or directly use a text generation model to generate coordinate point data through text description for replacement.

[0134] The beneficial effects of the above technical solution are as follows: the image data is received from the optoelectronic tweezers single cell screening device through the memory communication module and passed to the visualization module and the arraying main control module; then the arraying main control module completes the process control of the arraying process, calls the particle detection submodule, the arraying pattern submodule, the target allocation submodule and the path planning submodule to complete each link of the arraying; and the image data is visualized through the visualization module. The arraying main control module controls the particle detection module to detect the input image data and obtain the particle position and radius data; at the same time, the arraying main control module controls the arraying pattern module to generate an image according to the user text description through the image generation model, and then uses grid sampling to obtain the pattern lattice coordinate data; then the target allocation module is called to perform a one-to-one allocation based on the initial position of the particle detection and the target position of the arraying pattern; finally, the path planning module is called to perform path planning and calculate the next moving position based on the allocation result. The light projection module will convert the image data into a light projection pattern that can be recognized by the optoelectronic tweezers single cell screening device based on the next moving position and transmit it to the optoelectronic tweezers single cell screening device to realize the manipulation of optoelectronic tweezers particles. The present invention is an optoelectronic tweezers microparticle arraying system that integrates array pattern generation, particle detection, and path planning. It can generate various desired patterns according to user needs, and extract the component coordinates of the pattern as the array coordinates of the array operation, which greatly simplifies the array data preparation process. Then, an efficient detection algorithm is used to detect the position of the particles in the optoelectronic tweezers chip, and the starting position and the target position are matched from the global optimal angle. Then, a motion strategy is generated according to the RVO path planning method, and then the optoelectronic tweezers are controlled to transport the particles to the target position, thereby forming a target array pattern. The response time of the array system is optimized as a whole, ensuring the real-time performance of device control and data processing when the number of particles is large (more than 50).

[0135] Embodiment 6:

[0136] On the basis of Example 5, after receiving the image data, the arrayed main control module calls the particle detection submodule to perform particle detection on the received image data and obtain particle information, including:

[0137] First, the image data is converted into a grayscale image and the grayscale image is thresholded, and particle contours are identified based on the thresholded grayscale image;

[0138] Generate the minimum circumscribed circle of the corresponding particle based on the particle contour recognition result;

[0139] The particle morphology is compared with the minimum circumscribed circle morphology, and the radius and roundness of the particles in the grayscale image are screened according to the comparison results to determine the valid particles on the grayscale image;

[0140] The particle coordinates and radius of the effective particles are collected to generate particle information.

[0141] The beneficial effects of the above technical solution: the present invention simplifies the whole image processing steps and only needs to perform simple grayscale processing to enter the particle detection, reducing the operation of processing the whole image. At the same time, the minimum circumscribed circle, radius, and roundness screening steps are added in the detection process to ensure the detection effect, which reduces the detection time as a whole and is more suitable for high-throughput large-image particle detection.

[0142] Embodiment 7:

[0143] On the basis of Example 6, the arraying main control module calls the arraying pattern submodule to generate an arraying image, and determines the coordinate data of each target point from the arraying image, including:

[0144] Based on the image generation model, semantic recognition is performed on the text description input by the user, and the corresponding image is generated according to the semantic recognition result;

[0145] Binarizing the generated image to obtain a binary image, and generating a grid on the binary image based on an arbitrarily adjustable grid spacing to obtain an arrayed image;

[0146] Each pixel point in the grid on the arrayed image is screened one by one. When the value corresponding to the pixel point meets the preset condition, the pixel point is regarded as a valid particle that needs to be arrayed and manipulated, and the coordinate data of the valid particle that needs to be arrayed is collected.

[0147] In this embodiment, the grid point coordinates are defined as in and are the indices of the grid points in the horizontal and vertical directions, is the spacing of the grid, which can be adjusted arbitrarily;

[0148] The grid width, height and number are defined as in is the total size of the image (width times height), and determines the sampling function of the valid particles in the arrayed image that need to be arrayed. as follows:

[0149]

[0150] in is the image at point The value at is the set sampling threshold; It refers to the sampling threshold of the effective particle points that need to be manipulated in array. , that is, the effective particles in the arrayed image that need to be arrayed and manipulated, such as Figure 4 The points shown in the circles.

[0151] The beneficial effects of the above technical solution are as follows: the present invention generates an image of the target pattern based on a simple text description of the target pattern using an image generation model, and there is no need to extract and select points manually one by one. The generated pattern is processed using a grid method to generate an arrayed pattern and automatically determine the effective particles that need to be arrayed, avoiding the tedious operation of manual marking by the user, helping the user to immediately obtain the required arrayed pattern, simplifying the process and improving efficiency.

[0152] Embodiment 8:

[0153] On the basis of Example 6, the arraying main control module calls the target allocation submodule to perform one-to-one position allocation between the effective particles and the target points in the arrayed image, including:

[0154] A cost matrix building unit is used to build a movement cost matrix corresponding to each valid particle based on particle information and coordinate data of arrayed target points;

[0155] A position allocation determination unit, for determining the optimal allocation position of each effective particle that needs array manipulation in the array image based on the movement cost matrix and using the Jonker-Volgenant algorithm;

[0156] The array position allocation unit is used to establish a one-to-one correspondence between the effective particles that need to be arrayed and the target points of the arrayed image based on the optimal allocation positions of the effective particles that need to be arrayed.

[0157] In this embodiment, a movement cost matrix is ​​established based on the particle information and the coordinate data of the arrayed target points. The cost matrix is ​​constructed as follows:

[0158]

[0159] is the cost matrix Line The element of the column indicates that the starting point With target point The total cost when matching, the starting point refers to the valid particles that need to be arrayed; is the distance weight; is the weight of the radius difference;

[0160] The distance part uses Minkowski distance, which is a measurement method that can take into account both Euclidean distance and Manhattan distance. is the parameter of the Minkowski distance, when When , it represents the Manhattan distance, which is suitable for special microfluidic chips or confined spaces with obstacles. When , it indicates the Euclidean distance, which is suitable for barrier-free environments;

[0161] Is the starting point In the dimensional coordinates, It is the target point In the dimensional coordinates; n represents the dimension of the coordinate axis;

[0162] Is the starting point The radius of It is the target point The radius difference uses a logarithm to reduce the excessive impact of large radius values ​​on the cost calculation, making the impact of radius differences on the cost matrix smoother.

[0163] In this embodiment, based on the movement cost matrix, the Jonker-Volgenant algorithm is used to determine the optimal allocation position of each effective particle that needs array manipulation in the array image as follows:

[0164]

[0165] in, Indicates the starting point The minimum moving cost is Figure 4 As shown by the white spots on the black particles;

[0166] At the starting point A one-to-one correspondence is established between the target points at the optimal allocation position corresponding to the minimum moving cost;

[0167] Among them, each target point in the arrayed image can only correspond to one valid particle. When a target point at the optimal allocation position corresponding to a small movement cost corresponding to an effective particle that needs arrayed manipulation has been allocated with a valid particle, the effective particle that needs arrayed manipulation can only find the target point corresponding to the minimum movement cost among the remaining target points as the optimal allocation position.

[0168] Beneficial effects of the above technical solution: The present invention uses the Jonker-Volgenant algorithm to complete the optimal allocation of the starting position and the target position of the particles according to the cost matrix, minimize the total distance required for global movement, and reduce the time consumption of arraying.

[0169] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A photoelectric tweezers microparticle array system, It is characterized in that include: A memory communication module, used for receiving image data from the photoelectric tweezers single-cell screening device and transmitting the image data to the visualization module and the arrayed main control module respectively; Visualization module, used to visualize image data; The arrayed main control module is used to control the particle detection submodule, the arrayed pattern submodule, the target allocation submodule and the path planning submodule to work together to obtain the arrayed image and the next planned position of the particles to be projected; A light projection module, for generating a light projection adaptation map of the photoelectric tweezers single-cell screening device based on the arrayed image and the next planned position of the particles to be projected, and sending the light projection adaptation map to the photoelectric tweezers single-cell screening device to generate a corresponding light projection; The arrayed main control module includes: The particle detection submodule is used to perform particle detection on the received image data and obtain particle information; An arrayed pattern submodule, for generating an image according to a text description input by a user through an image generation model, processing the generated image based on a grid sampling method to obtain an arrayed image, and determining coordinate data of each target point from the arrayed image; A target allocation submodule, used for performing one-to-one position allocation between effective particles and target points in the arrayed image; The path planning submodule is used to determine the particle to be projected according to the position allocation result, the particle coordinates in the particle information and the coordinate data of the target point, and calculate the next planned position of the particle to be projected according to the RVO algorithm; Among them, the particle detection submodule includes: An image data processing unit, used to first convert the image data into a grayscale image and threshold the grayscale image, and perform particle contour recognition based on the thresholded grayscale image; A particle contour processing unit, used for generating a minimum circumscribed circle of a corresponding particle based on a particle contour recognition result; A particle screening processing unit is used to compare the particle morphology with the minimum circumscribed circle morphology, and screen the radius and circularity of the particles in the grayscale image according to the comparison result to determine the valid particles on the grayscale image; A particle information collection unit, used to collect particle coordinates and radius of effective particles and generate particle information; The arrayed pattern submodule includes: An image generation unit, used to perform semantic recognition on the text description input by the user based on the image generation model, and generate a corresponding image according to the semantic recognition result; An image arraying unit, used for performing binarization processing on the generated image to obtain a binarized image, and generating a grid on the binarized image based on an arbitrarily adjustable grid spacing to obtain an arrayed image; The grid point coordinates are defined as ,in and are the indices of the grid points in the horizontal and vertical directions, is the spacing of the grid, which can be adjusted arbitrarily; The number of grids is defined as ,in, It is the total size of the image, i.e., width multiplied by height, and determines the sampling function of the effective particles in the arrayed image that need to be arrayed. as follows: in is the image at point The value at is the set sampling threshold; It refers to the sampling threshold of the effective particle points that need to be manipulated in array; , all target points in the arrayed image can be obtained; Among them, the target allocation submodule includes: A cost matrix building unit, used for building a movement cost matrix based on particle information and coordinate data of arrayed target points; A position allocation determination unit, for determining the optimal allocation position of each effective particle that needs array manipulation in the array image based on the movement cost matrix and using the Jonker-Volgenant algorithm; An array position allocation unit, for establishing a one-to-one correspondence between the effective particles that need to be arrayed and the target points of the arrayed image based on the optimal allocation positions of the effective particles that need to be arrayed; Among them, based on the particle information and the coordinate data of the arrayed target points, a movement cost matrix is ​​established, and the cost matrix is ​​constructed as follows: is the cost matrix Line The element of the column indicates that the starting point With target point The total cost when matching, the starting point refers to the valid particles that need to be arrayed; is the distance weight; is the weight of the radius difference; The distance part uses Minkowski distance, which is a measurement method that can take into account both Euclidean distance and Manhattan distance. is the parameter of the Minkowski distance, when When , it represents the Manhattan distance, which is suitable for special microfluidic chips or confined spaces with obstacles. When , it indicates the Euclidean distance, which is suitable for barrier-free environments; Is the starting point In the dimensional coordinates, It is the target point In the dimensional coordinates; n represents the dimension of the coordinate axis; Is the starting point The radius of It is the target point The radius difference uses logarithm to reduce the excessive impact of large radius values ​​on cost calculation, making the impact of radius difference on the cost matrix smoother; Among them, based on the movement cost matrix, the Jonker-Volgenant algorithm is used to determine the optimal allocation position of each effective particle that needs array manipulation in the array image as follows: in, Indicates the starting point The minimum moving cost; At the starting point A one-to-one correspondence is established between the target points at the optimal allocation position corresponding to the minimum moving cost; Among them, each target point in the arrayed image can only correspond to one valid particle. When a target point at the optimal allocation position corresponding to the minimum movement cost of a valid particle that needs arrayed manipulation has been allocated with a valid particle, the valid particle that needs arrayed manipulation can only find the target point corresponding to the minimum movement cost among the remaining target points as the optimal allocation position.

2. A method for arraying photoelectric tweezers particles, It is characterized in that include: Initialize and reset each functional module, and control the memory communication module to wait for receiving image data; After the photoelectric tweezers single-cell screening device is started, it automatically captures images in real time, packages the image data and transmits it to the memory for storage, and synchronously sends an event ready signal to the memory communication module; After receiving the event ready signal, the memory communication module starts to continuously read the memory, obtains the image data packet, unpacks and parses the image data into image data, and sends it to the arrayed main control module; After receiving the image data, the arraying main control module performs an arraying operation on the image data to obtain an arrayed image and the next planned position of the particles to be projected; Generate a light projection adaptation map of the optoelectronic tweezers single-cell screening device based on the arrayed image and the next planned position of the particles to be projected through the light projection module, and send the light projection adaptation map to the optoelectronic tweezers single-cell screening device to generate a corresponding light projection; After receiving the image, the arraying main control module performs an arraying operation on the image to obtain the arrayed image and the next planned position of the particles to be projected, including: After receiving the image data, the array main control module calls the particle detection submodule to perform particle detection on the received image data and obtain particle information; The arraying main control module calls the arraying pattern submodule to generate an arraying image, and determines the coordinate data of each target point from the arraying image; The arraying main control module calls the target allocation submodule to perform one-to-one position allocation between the effective particles and the target points in the arraying image; The array main control module calls the path planning submodule to determine the particles to be projected according to the position allocation result, the particle coordinates in the particle information and the coordinate data of the target point, and calculates the next planned position of the particles to be projected according to the RVO algorithm; Among them, after receiving the image data, the arrayed main control module calls the particle detection submodule to perform particle detection on the received image data and obtain particle information, including: First, the image data is converted into a grayscale image and the grayscale image is thresholded, and particle contours are identified based on the thresholded grayscale image; Generate the minimum circumscribed circle of the corresponding particle based on the particle contour recognition result; The particle morphology is compared with the minimum circumscribed circle morphology, and the radius and roundness of the particles in the grayscale image are screened according to the comparison results to determine the valid particles on the grayscale image; Collect particle coordinates and radius of effective particles to generate particle information; The arraying main control module calls the arraying pattern submodule to generate an arraying image, and determines the coordinate data of each target point from the arraying image, including: Based on the image generation model, semantic recognition is performed on the text description input by the user, and the corresponding image is generated according to the semantic recognition result; Binarizing the generated image to obtain a binary image, and generating a grid on the binary image based on an arbitrarily adjustable grid spacing to obtain an arrayed image; Screening each pixel point in the grid on the arrayed image one by one, when the value corresponding to the pixel point meets the preset condition, the pixel point is regarded as a valid particle that needs to be arrayed and manipulated, and the coordinate data of the valid particle that needs to be arrayed is collected; The grid point coordinates are defined as ,in and are the indices of the grid points in the horizontal and vertical directions, is the spacing of the grid, which can be adjusted arbitrarily; The number of grids is defined as ,in It is the total size of the image, i.e., width multiplied by height, and determines the sampling function of the effective particles in the arrayed image that need to be arrayed. as follows: in is the image at point The value at is the set sampling threshold; It refers to the sampling threshold of the effective particle points that need to be manipulated in array; , the effective particles that need array manipulation in the arrayed image can be obtained; The array main control module calls the target allocation submodule to perform one-to-one position allocation between the effective particles and the target points in the array image, including: A cost matrix building unit, used for building a movement cost matrix based on particle information and coordinate data of arrayed target points; A position allocation determination unit, for determining the optimal allocation position of each effective particle that needs array manipulation in the array image based on the movement cost matrix and using the Jonker-Volgenant algorithm; An array position allocation unit, for establishing a one-to-one correspondence between the effective particles that need to be arrayed and the target points of the arrayed image based on the optimal allocation positions of the effective particles that need to be arrayed; Among them, based on the particle information and the coordinate data of the arrayed target points, a movement cost matrix is ​​established, and the cost matrix is ​​constructed as follows: is the cost matrix Line The element of the column indicates that the starting point With target point The total cost when matching, the starting point refers to the valid particles that need to be arrayed; is the distance weight; is the weight of the radius difference; The distance part uses Minkowski distance, which is a measurement method that can take into account both Euclidean distance and Manhattan distance. is the parameter of the Minkowski distance, when When , it represents the Manhattan distance, which is suitable for special microfluidic chips or confined spaces with obstacles. When , it indicates the Euclidean distance, which is suitable for barrier-free environments; Is the starting point In the dimensional coordinates, It is the target point In the dimensional coordinates; n represents the dimension of the coordinate axis; Is the starting point The radius of It is the target point The radius difference uses logarithm to reduce the excessive impact of large radius values ​​on cost calculation, making the impact of radius difference on the cost matrix smoother; Among them, based on the movement cost matrix, the Jonker-Volgenant algorithm is used to determine the optimal allocation position of each effective particle that needs array manipulation in the array image as follows: in, Indicates the starting point The minimum moving cost; At the starting point A one-to-one correspondence is established between the target points at the optimal allocation position corresponding to the minimum moving cost; Among them, each target point in the arrayed image can only correspond to one valid particle. When a target point at the optimal allocation position corresponding to the minimum movement cost of a valid particle that needs arrayed manipulation has been allocated with a valid particle, the valid particle that needs arrayed manipulation can only find the target point corresponding to the minimum movement cost among the remaining target points as the optimal allocation position.

Images