Full-autonomous cell operation system based on body intelligence

Abstract

The invention provides a full-autonomous cell operation system based on body intelligence, and relates to the field of cell operation, the system comprises a system hardware architecture and a system software architecture; the system hardware architecture comprises a micromanipulation execution system, a precision positioning system, a multi-mode sensing system and an intelligent calculation and control center. The micromanipulation execution system is used for cell environment interaction, the precision positioning system is used for sample bearing and environment control, the multi-modal sensing system is used for providing key information, and the intelligent calculation and control center is used for operation decision making; the system software architecture comprises a sensing layer, a planning layer, an execution layer and an interaction layer. According to the application, cell recognition, tool selection, path planning, operation execution and result feedback can be automatically completed without manual intervention; through tool autonomous switching and strategy dynamic adjustment, various operations such as cell clamping, injection, controllable deformation and micro-assembly can be completed, and the problems that traditional cell operation depends on manual work, efficiency is low, and consistency is poor are solved.

Description

Technical Field

[0001] This application relates to the field of cell manipulation technology, specifically to a fully autonomous cell operating system based on embodied intelligence. Background Technology

[0002] Cell manipulation refers to a series of processing actions performed on cells with the aid of microscopes and micro / nano-systems. Cell manipulation mainly includes three types: cell injection, cell scanning, and cell clamping. It requires observation using tools such as optical microscopes, atomic force microscopes, or fluorescence inverted microscopes. In specific operations, appropriate observation tools and manipulation methods must be selected based on the actual application. Embodied intelligence refers to intelligence possessing a physical body, where the machine brain can assist in decision-making, thereby controlling the limbs to react quickly to changes in the external environment. Its core lies in the deep integration of physical devices and intelligent decision-making. As a cutting-edge direction in the field of artificial intelligence, it aims to break the traditional model of separation between body and brain in machine intelligence. Based on embodied intelligence, cell manipulation can be improved and optimized.

[0003] In related technologies, cell manipulation mainly includes traditional manual micromanipulation systems and semi-automatic cell manipulation systems. Traditional manual micromanipulation systems rely on operators observing cell states under a microscope for extended periods and manually adjusting the position, orientation, and movement of the micromanipulation arm to perform actions such as cell clamping, injection, and transfer. These systems primarily consist of a microscope, a manual micromanipulation arm, and an operating table. Semi-automatic cell manipulation systems possess certain automation functions, such as automatic cell positioning through image recognition technology, but key operational steps, such as clamping force control and injection depth adjustment, still require manual intervention. They consist of a microscopic imaging module, a micromanipulation arm, a sensing module, and a control terminal. These modules are connected via wired connections for data transmission and control. The operator locates the target cell using the microscopic imaging module, manually adjusts the micromanipulation arm to its approximate position, and then activates the automatic mode. The system completes clamping and movement actions according to preset parameters, and the operator can intervene in real-time through the control terminal during the process.

[0004] Traditional manual micromanipulation systems rely entirely on the operator's skill level, experience, and mental state for precision, resulting in low automation. Each operation has a long cycle and cannot process multiple samples in parallel, making it difficult to complete tasks within a reasonable timeframe, especially in scenarios requiring large numbers of cells, such as single-cell sequencing sample preparation and high-throughput drug screening. Furthermore, differences in operator habits and force control, as well as variations in the same operator's work at different times, lead to low repeatability and consistency of experimental results.

[0005] Semi-automatic cell operating systems have limited automation, can only execute preset actions, and cannot dynamically adjust strategies according to changes in cell morphology (such as cell drift and deformation), requiring frequent manual intervention; their sensing capabilities are weak, the sensors are not precise enough, and they cannot distinguish between force signals from cell membrane contact and excessive clamping, resulting in a high cell damage rate; they rely on human experience, and key steps such as cell recognition and operation point selection still require operator judgment, resulting in poor consistency of results.

[0006] Therefore, in traditional manual and semi-automatic cell operating systems, the lack of a complete closed-loop control mechanism results in poor coordination between different parts, which greatly limits the accuracy and efficiency of operation and makes it difficult to meet the needs of high-precision and high-throughput cell operations. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this application provides a fully autonomous cell operating system based on embodied intelligence, which solves the problem that current cell operations suffer from poor coordination between different parts, making it difficult to meet the requirements of high-precision and high-throughput cell operations.

[0008] To achieve the above objectives, this application provides the following technical solution:

[0009] This application provides a fully autonomous cellular operating system based on embodied intelligence. This operating system includes a system hardware architecture and a system software architecture. The system hardware architecture includes a micromanipulation execution system, a precision positioning system, a multimodal sensing system, and an intelligent computing and control center. The micromanipulation execution system is used for cellular environment interaction, the precision positioning system is used for sample carrying and environmental control, the multimodal sensing system is used to provide key information, and the intelligent computing and control center is used for computational decision-making. The system software architecture includes a perception layer, a planning layer, an execution layer, and an interaction layer.

[0010] The system software architecture is used for: image preprocessing, target localization and tracking, semantic mapping of the operating environment, semantic understanding of the operating scene, outputting system status information and transmitting it to the planning layer through the perception layer; receiving system status information and high-level task instructions from the user interaction layer interface to clarify the operation goals, objects and constraints through the planning layer; performing instruction parsing, feasibility verification, autonomous task decomposition and sub-task optimization, autonomous tool selection, path planning, interaction strategy generation, multi-objective optimization, risk assessment and replanning through the planning layer, and generating an executable sequence of target instructions; receiving and parsing the sequence of target instructions through the execution layer, converting the parsed instructions into actual control signals to drive hardware devices, calculating low-level action instructions based on imitation learning and reinforcement learning, performing real-time monitoring and adjustment, generating feedback and status updates, and building an execution safety mechanism; and receiving user-input task instructions through the interaction layer, displaying microscopic images and system status, providing system configuration, operation tool selection, and task scheduling functions to achieve system monitoring and management. Specifically, system initialization and task definition are required when the system starts; and system reset is performed when the system shuts down.

[0011] This application provides a fully autonomous cellular operating system based on embodied intelligence. Compared with existing technologies, it has the following advantages:

[0012] This application proposes a fully automated cell manipulation system, incorporating embodied intelligence technology to design specific system hardware and software architectures. The hardware architecture utilizes a micromanipulation execution system for cell-environment interaction, a precision positioning system for sample handling and environmental control, a multimodal sensing system for providing key information, and an intelligent computing and control center for computational decision-making. Furthermore, the software architecture constructs a closed-loop "perception-planning-execution" architecture through a perception layer, planning layer, execution layer, and interaction layer, achieving full-process autonomy. Compared to the limitations of traditional manual systems requiring full operator intervention and semi-automatic systems only capable of executing preset actions, this application's system can automatically complete cell identification, tool selection, path planning, operation execution, and result feedback without human intervention. Through autonomous tool switching and dynamic strategy adjustment, this system can perform various operations such as cell clamping, injection, controlled deformation, and micro-assembly. By integrating multimodal sensing, intelligent planning, and precision control technologies, it addresses the pain points of traditional cell manipulation, such as reliance on manual labor, low efficiency, and poor consistency. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is an exemplary operation flow diagram of a system software architecture provided in an embodiment of this application;

[0015] Figure 2 This is an exemplary architecture diagram of a fully autonomous cellular operating system based on embodied intelligence, provided in an embodiment of this application.

[0016] Figure 3 This is an exemplary architecture diagram of a perception layer provided in an embodiment of this application;

[0017] Figure 4 This is an exemplary architecture diagram of a planning layer provided in an embodiment of this application;

[0018] Figure 5 This is an exemplary architecture diagram of an execution layer provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0021] This application provides a fully autonomous cell operating system based on embodied intelligence, which solves the problem that current cell operations are difficult to meet the requirements of high-precision and high-throughput cell operations due to poor coordination between various parts.

[0022] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0023] Cell manipulation refers to a series of processing actions performed on cells with the aid of microscopes and micro / nano-systems. Cell manipulation mainly includes three types: cell injection, cell scanning, and cell clamping. It requires observation using tools such as optical microscopes, atomic force microscopes, or fluorescence inverted microscopes. In specific operations, appropriate observation tools and manipulation methods must be selected based on the actual application. Embodied intelligence refers to intelligence possessing a physical body, where the machine brain can assist in decision-making, thereby controlling the limbs to react quickly to changes in the external environment. Its core lies in the deep integration of physical devices and intelligent decision-making. As a cutting-edge direction in the field of artificial intelligence, it aims to break the traditional model of separation between body and brain in machine intelligence. Based on embodied intelligence, cell manipulation can be improved and optimized.

[0024] In related technologies, cell manipulation mainly includes traditional manual micromanipulation systems and semi-automatic cell manipulation systems. Traditional manual micromanipulation systems rely on operators observing cell states under a microscope for extended periods and manually adjusting the position, orientation, and movement of the micromanipulation arm to perform actions such as cell clamping, injection, and transfer. These systems primarily consist of a microscope, a manual micromanipulation arm, and an operating table. Semi-automatic cell manipulation systems possess certain automation functions, such as automatic cell positioning through image recognition technology, but key operational steps, such as clamping force control and injection depth adjustment, still require manual intervention. They consist of a microscopic imaging module, a micromanipulation arm, a sensing module, and a control terminal. These modules are connected via wired connections for data transmission and control. The operator locates the target cell using the microscopic imaging module, manually adjusts the micromanipulation arm to its approximate position, and then activates the automatic mode. The system completes clamping and movement actions according to preset parameters, and the operator can intervene in real-time through the control terminal during the process.

[0025] Traditional manual micromanipulation systems rely entirely on the operator's skill level, experience, and mental state for precision, resulting in low automation. Each operation has a long cycle and cannot process multiple samples in parallel, making it difficult to complete tasks within a reasonable timeframe, especially in scenarios requiring large numbers of cells, such as single-cell sequencing sample preparation and high-throughput drug screening. Furthermore, differences in operator habits and force control, as well as variations in the same operator's work at different times, lead to low repeatability and consistency of experimental results.

[0026] Semi-automatic cell operating systems have limited automation, can only execute preset actions, and cannot dynamically adjust strategies according to changes in cell morphology (such as cell drift and deformation), requiring frequent manual intervention; their sensing capabilities are weak, the sensors are not precise enough, and they cannot distinguish between force signals from cell membrane contact and excessive clamping, resulting in a high cell damage rate; they rely on human experience, and key steps such as cell recognition and operation point selection still require operator judgment, resulting in poor consistency of results.

[0027] In traditional manual and semi-automatic cell manipulation systems, the microscope acquires cell images and presents them to the operator or a simple image recognition module. The manual operating arm or the actuators of the semi-automatic system then perform actions based on the operator's instructions or preliminary automatic positioning information. However, due to the lack of a robust closed-loop control mechanism, the coordination between the various parts is poor, and the operational accuracy and efficiency are greatly limited, making it difficult to meet the demands of high-precision, high-throughput cell manipulation.

[0028] Specifically, existing cell manipulation technologies and products have the following significant defects in terms of process and performance: (1) Low degree of automation and reliance on manual operation: Traditional cell manipulation systems are mostly semi-automatic or manual control modes, requiring operators to observe the cell state through a microscope and manually operate the micromanipulation arm to complete actions such as clamping and injection. When operating manually, the accuracy of operation depends entirely on the experience and proficiency of the personnel. The success rate of manual operation is significantly affected by fatigue and psychological state. The operation deviation of different operators or the same person at different times is large, and long-term operation is prone to errors due to visual fatigue. The semi-automatic operating system can only achieve preset path movement or simple positioning and cannot dynamically adapt to environmental changes. When cells drift or change morphology due to the flow of culture medium, the system cannot adjust the operation strategy autonomously and requires manual recalibration, interrupting the operation process.

[0029] (2) Low operational efficiency and difficulty in meeting high-throughput requirements: Manual operation has a long single processing cycle and cannot process multiple samples in parallel. For scenarios such as single-cell sequencing sample preparation and high-throughput drug screening that require processing hundreds or thousands of cells, traditional technology is difficult to complete the task within a reasonable time, which seriously restricts the progress of research and application.

[0030] (3) Poor consistency and repeatability of results: There are significant individual differences in the strength control, operation rhythm and visual judgment of operators, and the operation stability of the same operator will also decrease at different times (such as fatigue state, changes in attention concentration), resulting in drastic fluctuations in experimental data and making it difficult to standardize.

[0031] (4) Lack of precise sensing and feedback mechanisms: Most existing technologies do not integrate high-sensitivity force sensors or multimodal sensing modules, making it impossible to monitor key information such as contact force, clamping force, and cell deformation during cell manipulation in real time. For example, when clamping cells, the inability to sense the force can easily lead to cell rupture due to excessive force or cell detachment due to insufficient force, resulting in poor operational reliability.

[0032] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0033] The following describes a fully autonomous cellular operating system based on embodied intelligence provided in the embodiments of this application.

[0034] This application provides a flowchart of a fully autonomous cellular operating system based on embodied intelligence, which may include a system hardware architecture 100 and a system software architecture 200.

[0035] The system hardware architecture 100 includes a micromanipulation execution system, a precision positioning system, a multimodal sensing system, and an intelligent computing and control center. The micromanipulation execution system is used for cell-environment interaction, the precision positioning system is used for sample carrying and environmental control, the multimodal sensing system provides key information, and the intelligent computing and control center performs computational decisions. Please refer to [the relevant documentation / reference]. Figure 1 and Figure 2 The system software architecture 200 includes a perception layer, a planning layer, an execution layer, and an interaction layer; the aforementioned system software architecture 200 is used for:

[0036] S210: Image preprocessing, target localization and tracking, semantic mapping of the operating environment, semantic understanding of the operating scene, output system status information and transmit it to the planning layer through the perception layer.

[0037] S220: Receive system status information through the planning layer and receive high-level task instructions from the user interaction layer interface to clarify the operation objectives, objects, and constraints.

[0038] S230, through the planning layer, performs instruction parsing, feasibility verification, autonomous task decomposition and sub-task optimization, autonomous tool selection, path planning, interaction strategy generation, multi-objective optimization, risk assessment and replanning, and generates an executable sequence of target instructions.

[0039] S240: Receive and parse the target instruction sequence through the execution layer, convert the parsed instructions into actual control signals to drive the hardware device, calculate low-level action instructions based on imitation learning and reinforcement learning, perform real-time monitoring and adjustment, generate feedback and state updates, and build an execution safety mechanism.

[0040] S250 receives user-input task commands through the interaction layer, displays microscopic images and system status, and provides system configuration, operation tool selection, and task scheduling functions to achieve system monitoring and management. Among these, system initialization and task definition are required when the system starts up, and system reset is required when the system shuts down.

[0041] The above describes a specific implementation of a fully autonomous cell operating system based on embodied intelligence, as provided in this application. It is understood that this application, for fully automated cell manipulation, introduces embodied intelligence technology to design a specific system hardware and software architecture. In the system hardware architecture, a micromanipulation execution system is used for cell environment interaction, a precision positioning system for sample carrying and environmental control, a multimodal perception system for providing key information, and an intelligent computing and control center for computational decision-making. Furthermore, the system software architecture constructs a closed-loop "perception-planning-execution" architecture through a perception layer, planning layer, execution layer, and interaction layer to achieve full-process autonomy. Compared to the limitations of traditional manual systems requiring full operator intervention and semi-automatic systems only capable of executing preset actions, the system of this application can automatically complete cell identification, tool selection, path planning, operation execution, and result feedback without human intervention.

[0042] Based on this, this system, through autonomous tool switching and dynamic strategy adjustment, can perform various operations such as cell clamping, injection, controlled deformation, and micro-assembly. It is adaptable to different sample types, including HeLa cells, embryonic stem cells, and neurons, as well as various carriers such as culture dishes and microfluidic chips, meeting the needs of multiple fields such as life science research and precision medicine. This application is mainly applied in life science research, precision medicine, and drug development, enabling a series of high-precision operations such as cell clamping, injection, controlled deformation, and micro-assembly. By integrating multimodal sensing, intelligent planning, and precision control technologies, it solves the pain points of traditional cell manipulation, such as reliance on manual labor, low efficiency, and poor consistency.

[0043] In some embodiments, the micromanipulation execution system includes a micromanipulator, a micromanipulation end effector, and a force sensing device; the micromanipulator includes two micromanipulator arms, which have submicron-level positioning accuracy and a range of motion that can cover the cell manipulation workspace; the micromanipulator arms can perform operations including cell gripping, injection, controlled deformation, gentle pushing, and microassembly through autonomous collaboration; the micromanipulation end effector can autonomously switch according to specific tasks and interact directly with cells; the micromanipulation end effector includes a microgripper and a microinjection needle, and is made of biocompatible materials and integrates a micro-actuation mechanism to achieve precise manipulation.

[0044] In addition, force sensing devices are integrated into the end-effectors of micromanipulation devices to acquire force feedback information in real time. This allows the sensing layer to use the force feedback information to perceive the mechanical properties of cells and determine the contact state, ensuring low damage to fragile cells and enabling non-invasive cell manipulation. Non-invasive cell manipulation includes adaptive adjustment of clamping force and control of injection depth and speed.

[0045] In the embodiments of this application, it is understood that the system is configured with two high-precision micromanipulation arms to achieve flexible spatial movement and fine posture adjustment. These manipulators have sub-micron level positioning accuracy and high stability, and can cover the entire cell manipulation workspace. Through the autonomous collaboration of multiple manipulator arms, complex operations such as cell clamping, injection, controlled deformation, gentle pushing, and microassembly can be performed.

[0046] Micromanipulation end effectors, such as microgrippers and microneedles, are the parts that directly interact with cells and can be autonomously switched according to specific tasks. Force sensing devices are key to achieving fully autonomous and compliant interaction. Highly sensitive force sensors are integrated into the end effector or tool to provide real-time feedback on minute forces applied during the manipulation process. The sensing layer uses this force feedback information to perceive the mechanical properties of cells and determine the contact state, while the planning and control layer uses this information to achieve autonomous force control, ensuring minimal damage to fragile cells and enabling non-invasive cell manipulations such as adaptive adjustment of gripping force and control of injection depth and speed.

[0047] In some embodiments, the precision positioning system includes a precision positioning platform on which cell manipulation is performed. The surface of the precision positioning platform is integrated with an automated fixation device adapted to different sample containers. The precision positioning platform has multiple autonomously controlled degrees of freedom of motion, so as to adjust the position and orientation of the sample container through fine-tuning of each degree of freedom, and cooperate with the micromanipulation execution system to complete cell manipulation tasks in a large area or a specific region.

[0048] In this embodiment, it is understood that the entire cell manipulation process is performed on this precision positioning platform. This platform possesses multiple autonomously controlled degrees of freedom (translation along the X, Y, and Z axes, and possible rotation), enabling fine-tuning of each degree of freedom. This allows for precise adjustment of the position and orientation of sample containers such as culture dishes and microfluidic chips, facilitating large-scale or specific-area cell manipulation tasks in conjunction with a micromanipulation system. This precision positioning platform exhibits high stability and high positioning accuracy, providing a reliable benchmark for autonomous operation. Furthermore, its surface integrates automated fixation devices adapted to different sample containers.

[0049] In one example, the multimodal sensing system includes a microscopic imaging system, a microscopic vision motion platform, and an environmental state sensor. The microscopic imaging system includes an optical microscope equipped with a high-resolution microscopic camera (containing multiple imaging modes such as bright field, phase contrast, and fluorescence) for cell identification, localization, tracking, and state assessment. The microscopic imaging system can be mounted on an autonomously controlled multi-degree-of-freedom microscopic vision motion platform to dynamically expand the sensing range of the vision system, autonomously scan the entire sample area, and automatically track cells or manipulation tools that have moved out of the current field of view, in order to adapt to large-size samples or high-throughput screening tasks.

[0050] The environmental state sensors include temperature and pH sensors, which are used to construct a multi-dimensional monitoring network for the cell environment to comprehensively perceive the state of the cell culture microenvironment. The intelligent computing and control hub includes a graphics workstation, server, and high-performance industrial computer to run perception algorithms, planning algorithms, and control algorithms. The perception algorithms are used for image processing, pattern recognition, and scene understanding; the planning algorithms are used for task planning, motion planning, and decision making; and the control algorithms are used for servo control and force control. The intelligent computing and control hub also includes a data acquisition interface, which is equipped with a high-speed image acquisition card and a multi-channel sensor interface card for the transmission of various types of perception data.

[0051] In some embodiments, the raw image quality is poor due to low contrast, noise, uneven illumination, and possible image distortion, which are common in microscopic imaging, especially live-cell imaging. Image preprocessing aims to suppress noise, enhance the features of cells and manipulators, and correct distortions, thereby improving visual quality and providing higher-quality input for subsequent analysis, identification, and understanding.

[0052] Please refer to Figure 3 The aforementioned image preprocessing, target localization and tracking, semantic mapping of the operating environment, semantic understanding of the operating scene, output of system state information and transmission to the planning layer through the perception layer, i.e., the aforementioned S210 includes:

[0053] S310. Remove random noise and background impurities while preserving edge details of cells and manipulation tools during denoising. For example, the process of removing random noise and background impurities can employ the Non-Local Means (NL-Means) algorithm. It is understood that NL-Means is an image denoising method. Unlike traditional local filtering algorithms (such as mean filtering and Gaussian filtering) that only utilize pixel neighborhood information, NL-Means performs denoising by globally searching for similar regions, thus better preserving image details (such as edges and textures).

[0054] S320: Enhance the local contrast of the image, making cell outlines, internal structures, and features of the manipulation tools clearer, while suppressing excessive noise amplification. For example, step S320 can utilize contrast-limited adaptive histogram equalization to enhance the local contrast of the image.

[0055] S330. When the contrast exceeds a preset first contrast threshold, automatic thresholding is performed to distinguish the target from the background. In complex scenes where cells are adhered and the contrast is below a preset second contrast threshold, semantic segmentation is performed to extract pixel-level contours of cells and manipulation tools. For example, in cases where the contrast exceeds the preset first contrast threshold, automatic thresholding can be performed using the Otsu method; in complex scenes, semantic segmentation can be performed using a deep learning-based U-Net network. It is understood that the Otsu method is an algorithm for image thresholding. Its core idea is to find an optimal threshold to divide pixels in an image into foreground (target) and background parts, maximizing the inter-class variance (or minimizing the intra-class variance) between the two parts, thereby achieving effective image segmentation. The U-Net network is a convolutional neural network (CNN) architecture, originally used for biomedical image segmentation tasks. Its name comes from its "U"-shaped network structure, consisting of a contraction path (downsampling) and an expansion path (upsampling), which effectively combines low-level detail information with high-level semantic information.

[0056] S340 identifies the cell membrane boundaries and the edges of the manipulator, and performs image preprocessing through denoising, enhancement, segmentation, and edge detection to optimize the quality of the microscopic image and extract the contour features of the cells and the manipulator. For example, step S340 can use the Canny edge detection operator to clearly identify the cell membrane boundaries and the edges of the manipulator, providing crucial geometric information for localization and pose estimation. It is understood that Canny edge detection is a multi-stage edge detection algorithm; the Canny edge detection operator is not a simple single operator, but rather achieves edge detection through a series of image processing steps, featuring low error rate, high localization accuracy, and a single edge response.

[0057] S350. For tool parts with relatively fixed shapes, perform rapid localization to assist in tracking static features or tool markers in the field of view, and combine the contour information obtained from image segmentation for shape analysis. For example, in step S350, for tool parts with relatively fixed shapes, template matching based on normalized cross-correlation can be used for rapid localization, or ORB feature points can be used for detection and matching to assist in tracking static features or tool markers in the field of view. It can be understood that ORB feature points are unique and stable local regions in an image (such as corner points, edge endpoints, etc.), and are the foundation for tasks such as image matching, object recognition, and motion estimation in computer vision. ORB (Oriented FAST and Rotated BRIEF) is an efficient feature point detection and description algorithm that combines the advantages of FAST corner detection and BRIEF descriptors, and solves its rotation invariance problem.

[0058] S360 employs deep learning-based methods for object detection, instance segmentation, pose estimation, and object tracking. Understandably, to acquire the position, pose, and state of the target cell, surrounding cells, and manipulation tools in real-time and accurately during dynamic cell manipulation, the perception layer comprehensively utilizes various computer vision technologies. Furthermore, the system primarily employs the following deep learning-based methods for accurate object detection, segmentation, pose estimation, and tracking.

[0059] In the object detection process, the YOLO series network is used to quickly locate and select all objects of interest in the microscopic image and classify them. These objects include target cells, surrounding cells, and manipulation tools. In the instance segmentation process, the Mask R-CNN network is used to locate the target and segment the precise contours of each cell and tool at the pixel level. It is understood that the YOLO series network is a single-stage object detection algorithm with good flexibility and ease of use, applicable to computer vision tasks such as object detection, instance segmentation, and pose estimation. The Mask R-CNN network is a deep learning-based instance segmentation algorithm that extends Faster R-CNN with a branch for predicting object masks, enabling simultaneous object detection (localization and classification) and instance segmentation (pixel-level annotation) tasks with high accuracy and flexibility.

[0060] In the pose estimation process, the 3D pose of the cell manipulation tool is estimated by combining the results of target detection and instance segmentation. Specifically, the pose estimation involves two aspects. First, for rigid cell manipulation tools (such as manipulation needles), the 6-DOF (X, Y, Z translation + rotation around X, Y, Z axes) pose estimation is completed by using the deep learning-based PVNet method. Taking advantage of its accuracy in pose regression of rigid objects, the precise 3D pose of the tool is obtained. Secondly, for non-rigid degree-of-freedom cellular manipulation tools with opening and closing angles, such as grippers, a lightweight joint angle regression network is introduced: based on the gripper jaw contour obtained from instance segmentation, geometric features such as the distance between the jaw tips and the opening and closing gap are extracted; an improved CNN regression model (with MobileNetv3 as the backbone) is used to directly predict the jaw opening and closing angle, and the clamping force data fed back by the force sensor is used for cross-validation to correct the angle estimation error; the pose of the six-degree-of-freedom rigid body is fused with the degree-of-freedom parameters of the flexible joints to construct a complete N-degree-of-freedom posture description model of the manipulation tool (N≥6), and the posture parameters are calibrated in real time through the interaction force feedback between the end effector and the cell to ensure the dynamic consistency of the posture estimation during the operation.

[0061] During target tracking, the output of target detection and instance segmentation is combined with the DeepSORT tracking algorithm to achieve stable and robust tracking of target cells and manipulators across consecutive frames.

[0062] In the embodiments of this application, it can be understood that DeepSORT (Deep Simple Online and Realtime Tracking) is a multi-object tracking algorithm based on deep learning. By introducing a deep feature extraction network, it solves the problem of insufficient appearance features in SORT and significantly improves the tracking robustness in scenarios such as target occlusion and temporary disappearance.

[0063] Furthermore, the perception layer, as the entry point for the system's interaction with the physical world, is responsible for understanding the current operating environment and object state, extracting key information from the complex environment, and supporting subsequent autonomous decision-making and precise execution. The system utilizes images acquired by the microscopic imaging system to automatically locate target cells, segment images, and identify cells that meet the task requirements. Through force sensors, the system monitors in real time the contact and interaction forces between the end of the micromanipulation tool and the environment or cells, and determines the current contact state (e.g., no contact, slight contact, clamping, piercing, etc.) based on the magnitude and changes in force. Combining information from camera visual positioning and platform encoders, the system estimates the precise pose of the tool and target cells in a unified coordinate system and updates environmental semantics, such as identifying petri dish boundaries, manipulated areas, or other potential obstacles, forming a dynamic understanding of the operating scenario.

[0064] Please refer to Figure 3 In some embodiments, the aforementioned image preprocessing, target localization and tracking, semantic mapping of the operating environment, semantic understanding of the operating scene, output of system state information and transmission to the planning layer through the perception layer, that is, the aforementioned S210 may also include the following steps:

[0065] S410. Construct an operational environment map with semantic information to provide the planning module with global environmental awareness, supporting path planning, obstacle avoidance, and multi-objective task allocation.

[0066] S420: Visual odometry is used to estimate the motion of the camera or stage through a continuous image sequence; image feature points are extracted and inter-frame matching is performed, and pose optimization is performed through bundle adjustment; semantic information obtained from detection and segmentation is fused into the mapping process to generate an octree map with semantic labels; loop closure detection is implemented to correct accumulated errors and ensure accuracy.

[0067] S430. Extract low-level features through the visual layer, represent object attributes and instantaneous states through the object interaction layer, and determine the understanding of the operation process through the conceptual layer; perform semantic understanding of the operation scenario.

[0068] S440: Uses a Transformer network to process time series information, identify the current operation stage, understand the relationship between tools and cells, identify abnormal situations, and predict the next operation or identify abnormalities.

[0069] S450. Output the system state information obtained by the perception layer to the planning layer in a structured form. The system state information includes: the position, attitude, state, environment map, interaction state, and scene semantics of cells or operating tools.

[0070] Specifically, system status information may include: the precise location, orientation, outline, ID, and status of the target cell and related cells; the precise location, orientation, and end-effector status of the cell manipulation tool; the location / outline of obstacles and the boundary of the work area in the environmental information; the interaction status, including contact, grasp, and injection status; and the semantics of the operation scenario, such as the current operation stage and risk warnings.

[0071] Operational scenario semantic understanding involves: the interaction state between the tool and the cell, the progress stage of the operation, and potential risks. This is the key to connecting the perception layer and the planning layer, and is divided into the following three levels:

[0072] (1) Visual layer: Extracts low-level features, such as cell color, texture, and shape, and tool outline and edge. It is mainly used to accurately obtain the position and orientation of cells and tools (especially end effectors, such as gripper tips and needle tips); and utilizes the precise position, orientation, outline, color, texture and other information obtained from preprocessing and segmentation.

[0073] (2) Object Interaction Layer: Reflects object attributes and instantaneous states. For example: the opening and closing angle of the gripper, whether the cell is clamped (this can be determined by visually observing cell deformation or the relative position of the tool and the cell, and can be verified by correlating with force sensor data), whether the needle tip contacts / pierces the cell membrane, the translational / rotational speed of the cell, and the health status of the cell. By combining pose information and force sensor data, the opening and closing degree of the gripper, whether the clamping is successful, and the contact / piercing state of the needle tip are determined.

[0074] (3) Concept / Task Layer: This layer most closely resembles human understanding of the operational process. For example, it identifies the current operational stage, such as "approaching the target cell," "attempting to grasp," "successfully grasping," "moving the cell," "preparing for injection," "injection completed," and "releasing the cell." It understands the relationship between the tool and the cell (e.g., "the gripper has grasped cell A," "the needle tip has inserted into cell B"); predicts the next possible operation; identifies abnormal situations (e.g., "grabbing failed," "cell ruptured," "target lost"); and uses a Transformer network to process time-series information (continuous image features, state judgment results), identifies the current operational stage (e.g., "approaching," "grabbing," "moving"), understands the relationship between the tool and the cell, and predicts the next operation or identifies abnormalities.

[0075] Please refer to Figure 4 In some embodiments, the aforementioned process of instruction parsing, feasibility verification, autonomous task decomposition and sub-task optimization, autonomous tool selection, path planning, interaction strategy generation, multi-objective optimization, risk assessment and replanning through the planning layer, and generating an executable sequence of target instructions, specifically includes the following steps:

[0076] S510: Parse instructions to accurately understand task intent, and verify task feasibility based on the current state of the perception layer, so as to avoid the system attempting to perform operations that are physically or logically impossible.

[0077] Understandably, the planning layer first receives high-level task instructions from the user interaction layer interface, thus clarifying the operation goals, objects, and constraints. Simultaneously, the planning layer also receives comprehensive, real-time system status information from the perception layer, such as the position, posture, and state of cells or manipulation tools, the environment map, interaction states, and scene semantics. Subsequently, it parses the instructions to accurately understand the task intent and, based on the current state of the perception layer, verifies the feasibility of the task. Feasibility verification is a necessary preliminary step, designed to prevent the system from attempting operations that are physically or logically impossible, such as trying to grasp a non-existent or inaccessible cell.

[0078] S520: The complex task given by the user is automatically decomposed into a series of logically coherent, smaller, and more specific executable subtasks, i.e., atomic operation sequences, using the Hierarchical Task Network (HTN) planner. In essence, HTN is an automated planning method based on task decomposition. By breaking down complex tasks into simpler subtasks, it ultimately forms a directly executable sequence of actions, thereby enabling planning for dynamic environments or complex goals. HTN can effectively represent task knowledge with clear steps and hierarchical structures, such as cellular operations. The planning layer utilizes a predefined HTN method library, containing rules for high-level tasks such as grasping, moving, and injecting, and their decomposition into basic actions such as moving, adjusting posture, and closing grippers. Based on the current state and the goal, it searches and generates an ordered sequence of subtasks.

[0079] For example, "grabbing cell A" may be broken down into a series of fine steps: calculating the pre-grabbing pose, moving the tool to the predetermined point, fine-tuning the tool's posture to match the cell, slowly approaching the cell, performing a closing gripper action, confirming successful grasping through sensors (vision / force), and lifting the tool; at the same time, during the decomposition process, the system will perform preliminary optimization, such as selecting the grasping pre-grabbing point with the lowest energy consumption or the shortest path.

[0080] S530. During the decomposition process, the system will perform preliminary optimization, such as selecting the grabbing point with the lowest energy consumption or the shortest path.

[0081] S540. When the system is equipped with a variety of operating tools, including grippers and microneedles, the rule-based decision-making system queries the preset rule base to select the most suitable tool based on the sub-task type, the target cell size measurement value provided by the perception layer, and the current availability status of each tool.

[0082] S550: After the tool selection is completed, based on the preset two-stage planning strategy, a motion trajectory from the current position to the target position is planned for the selected operating tool. The trajectory must be collision-free, smooth and safe.

[0083] S560 employs a hybrid learning strategy that combines imitation learning (IL) and reinforcement learning (RL) to plan the interaction strategy when the tool interacts directly with the cell in order to control the fine action sequence and parameters and reduce physical damage to fragile cells.

[0084] S570. Multi-objective optimization is achieved by defining a comprehensive cost function, which assigns different weights to different objectives based on their importance and then sums them up. Understandably, at each stage of planning, it is necessary to balance multiple potentially conflicting objectives, such as reducing total operation time, minimizing tool travel distance, reducing energy consumption, maximizing cell viability, and minimizing physical damage to cells. The system achieves optimization by defining a comprehensive cost function, which assigns different weights to different objectives based on their importance and then sums them up. For example, in the precision operation stage, the weight of cell damage may be much higher than that of operation time. This multi-objective optimization is necessary because it ensures that planning decisions are not one-sided, but rather comprehensively consider multiple dimensions such as efficiency, safety, and operational quality, ultimately generating a balanced solution that better meets actual needs.

[0085] S580. When generating the initial plan, the planning layer evaluates each key step based on a model or historical data. Understandably, the planning layer possesses the ability to predict and respond to potential risks and unforeseen circumstances during execution, ensuring operational robustness. When generating the initial plan, the probability of failure events such as collisions, slippage during grasping, and cell rupture is evaluated based on a model or historical data for each key step.

[0086] S590. When the perception layer detects a significant change in the environment, the planning mechanism can automatically trigger a replanning mechanism. Starting from the current failed or unexpectedly interrupted state, it quickly invokes the relevant functions of path planning and strategy generation in the relevant planning modules to generate a new follow-up plan adapted to the current situation. It is understood that significant changes in the environment can include: target movement, new obstacles blocking the way, abnormal tool slippage, or execution errors reported by the execution layer. This dynamic evaluation and rapid response capability of this application is crucial for successfully completing tasks in the delicate and uncertain environment of cellular manipulation.

[0087] S591. The plan obtained through decomposition, selection, pathing, strategy, optimization, and risk assessment is converted into a sequence of target instructions that the underlying controller can directly understand and execute. The sequence of target instructions includes: target pose sequence, motion parameters, force / torque control targets, and discrete event instructions. For example, the target pose sequence includes the precise three-dimensional position and posture that the operating tool should achieve, the motion parameters include the velocity and acceleration curve along the path, and the force / torque control targets include the gripping force / injection force or torque value that needs to be applied or maintained during the interaction.

[0088] In the embodiments of this application, it can be understood that the planning layer receives real-time understanding of the environment and system status from the perception layer, and combines the high-level task instructions issued by the user in the interaction layer. Based on the perceived current status and the parsed task objectives, the system first performs autonomous task decomposition, breaking down the high-level task into a series of logical sub-tasks (such as: scanning, identification, selection, approach, grasping, moving, and releasing), and optimizes the execution order of these sub-tasks.

[0089] Furthermore, the planning layer autonomously selects tools based on the requirements of the current subtask. For each subtask, the system performs path planning, calculating a safe, collision-free, and smooth trajectory for the micromanipulator and / or positioning platform from its current position to the target position. Simultaneously, the system generates a refined interaction strategy, defining key parameters during the operation, such as approach speed, grasping force, holding force, injection depth, and rate. The planning process often involves multiple objectives (such as accuracy, speed, cell damage risk, and force limitations), and the system employs a multi-objective optimization method to weigh these factors and generate an optimal comprehensive strategy. The system also possesses risk assessment and replanning capabilities; it predicts potential execution risks (such as cell drift, collision, and excessive force) and adjusts the plan or initiates a completely new planning process when necessary to address unforeseen circumstances. The final output of the planning layer is a detailed sequence of commands that can be directly executed by the action module.

[0090] In one example, the aforementioned two-stage planning strategy may include the following steps: performing global path planning in a relatively static environment model, based on the map constructed by the perception layer or the current view snapshot, using the RRT* algorithm for global path planning; and during trajectory execution, combining real-time updated perception information, using the Dynamic Window Method (DWA) for local path adjustment and velocity planning.

[0091] Understandably, the RRT* algorithm is a sampling-based probabilistic complete path planning algorithm, an improved version of the Fast Exploratory Random Tree (RRT) algorithm. The RRT* algorithm introduces an optimization mechanism on top of the RRT algorithm, enabling it to efficiently search for near-optimal paths in complex environments. Dynamic Window Method (DWA) is a local path planning algorithm primarily used for real-time obstacle avoidance and trajectory generation for mobile robots in dynamic environments (such as those with sudden obstacles). The Dynamic Window Method samples feasible velocity combinations in the robot's velocity space, simulates the short-term trajectories corresponding to these velocities, and selects the optimal trajectory based on an evaluation function, thereby achieving safe and efficient motion control.

[0092] During path planning, the RRT* algorithm is used for macro-level global path planning, while the Dynamic Window Method (DWA) is used to give the system real-time responsiveness, enabling the system to flexibly avoid suddenly appearing floating debris obstacles or make immediate adjustments to minor changes in the environment, ensuring smoothness and safety of movement.

[0093] In one example, the interaction strategy includes a grasping and placement strategy and an injection strategy. In the grasping and placement strategy, based on the precise cell contour and pose provided by the perception layer, a fine-grained final approach trajectory and gripper end pose are planned, and control parameters are generated using a hybrid learning strategy based on imitation learning (IL) and reinforcement learning (RL). In the injection strategy, the puncture position and entry angle are planned. The hybrid learning strategy is used to determine the basic puncture action pattern through imitation learning and to fine-tune the force feedback control and velocity curve during the puncture process through reinforcement learning, so as to adapt to the toughness of different cell membranes, perform rapid, accurate, and low-damage punctures, and precisely control the injection volume and rate.

[0094] In one example, the aforementioned method of generating control parameters using a hybrid learning strategy based on imitation learning (IL) and reinforcement learning (RL) may specifically include the following steps: learning successful grasping techniques from teaching data of human experts through imitation learning to quickly obtain a reliable initial strategy; using reinforcement learning to explore and optimize in a simulated or real environment to further improve the grasping success rate and robustness, and to learn to adaptively adjust the grasping force and speed according to subtle differences in cell size, shape, or hardness to minimize damage; and using force sensor feedback as a reward signal.

[0095] Please refer to Figure 5 In some embodiments, the aforementioned process of receiving and parsing the target instruction sequence through the execution layer, converting the parsed instructions into actual control signals to drive the hardware device, calculating low-level action instructions based on imitation learning and reinforcement learning, performing real-time monitoring and adjustment, generating feedback and state updates, and constructing an execution safety mechanism, specifically includes the following steps in S240:

[0096] S610 receives the target instruction sequence from the planning layer, performs automatic parsing to understand the specific actions and parameter settings to be executed at each time step, and prepares to distribute them to the corresponding underlying controller.

[0097] The S620, based on motion control strategy, force / interaction control strategy and visual servo control strategy, converts the parsed instructions into actual control signals that drive the hardware devices.

[0098] S630 generates an interactive policy network through imitation learning and reinforcement learning. A neural network inference engine is used as a dedicated actuator to receive the current state, including visual features and force feedback, as input to the interactive policy network, and calculates low-level motion commands, including joint speed, gripper closing rate, and applied force increment.

[0099] S640: Pass low-level motion commands to the motion or force controller for execution. During the execution of low-level motion commands, continuously monitor the system status and external environment, and perform fine-tuning or anomaly handling.

[0100] The S650 continuously reads and processes real-time data from various sensors to handle small disturbances and errors at the lower level; it also monitors execution errors such as deviations between the desired and actual states, whether force feedback exceeds limits, and whether visual targets are lost. For example, real-time data from various sensors includes: joint position / velocity from the encoder, contact force from the force sensor, and target position / shape and tool position from the vision system.

[0101] S660. If a deviation less than the preset first allowable threshold is detected, the controller will compensate automatically. If a deviation exceeding the preset second allowable threshold or an execution failure is detected, the execution layer will immediately stop the current action and send a feedback signal of execution failure and abnormal status to the planning layer, triggering the replanning process of the planning layer.

[0102] S670. The execution layer feeds back the results of the action execution and the latest information acquired during the process to the perception layer, reports the execution results of each sub-task or key step to the planning layer, updates environmental data, and transmits the latest sensor data acquired during the execution of the action to the perception layer. For example, the execution results of each sub-task or key step include success, failure, exception codes, etc., and the latest sensor data includes the precise final position of the tool, the actual applied force, and the new position or morphological changes of cells after the operation.

[0103] S680 implements protective measures at both the hardware and control software levels through the planning layer to prevent damage to cells, equipment, or the environment; it limits the movement speed and acceleration of the manipulator and restricts the range of motion of the tool by setting maximum force / torque output limit thresholds in the controller. It is understood that the planning layer implements protective measures at both the hardware and control software levels to prevent damage to cells, equipment, or the environment. By setting strict maximum force / torque output limits in the controller; limiting the maximum movement speed and acceleration of the manipulator; and limiting the range of motion of the tool in software to prevent it from colliding with microscope objectives, culture dish edges, etc., it provides a rapid and reliable emergency stop function.

[0104] In this embodiment, the execution layer receives the target instruction sequence from the planning layer, transforms the planned instructions into actual physical operations, and directly manipulates manipulators such as cell holders and microinjection needles to complete cell manipulation tasks in a real environment. Through visual servo control, real-time image feedback guides the manipulator to precisely approach or track moving target cells. When interaction with cells or the environment is required (e.g., grasping, placing, or injecting), the system precisely controls the tool's position and applied force, employing a high-level force-position hybrid control strategy to ensure accuracy and safety. The system can autonomously execute stored expert operation strategies, which may contain highly detailed and complex action sequences, mimicking the skilled techniques of human experts. During execution, the system tracks the progress of the task and the system's actual state (e.g., tool position, sensor readings) in real time, feeding this information back to the planning module for monitoring and possible replanning. The action layer then feeds this updated environmental data back to the perception layer, forming a closed loop to ensure the system's perception of the environment remains consistent with reality.

[0105] In some embodiments, the motion control strategy includes: driving the micromanipulator to track a collision-free path generated by the planning layer; employing a strategy combining model-based feedforward control and PID feedback control, wherein the feedforward control uses the manipulator's dynamic model to predict the required driving force / torque to compensate for inertia and gravity effects and improve tracking accuracy; and using a PID controller for feedback adjustment based on real-time position / attitude errors from sensors to suppress disturbances and model uncertainties. This combined control strategy aims to achieve fast and smooth trajectory tracking, ensuring that the tool moves precisely along the planned path.

[0106] As is understandable, feedforward control is a proactive, preventative control method that anticipates potential disturbances or desired output changes in the system and takes control action before deviations occur, thereby reducing or eliminating the impact of disturbances on the system. PID feedback control is the most widely used feedback control algorithm in industrial control. Through the coordinated action of proportional (P), integral (I), and derivative (D) components, it adjusts the control input in real time based on the deviation between the system output and the target value (target value - actual output), stabilizing the system at the target state. Its core advantages lie in its simple structure, strong robustness (insensitivity to changes in system parameters), and its ability to effectively handle most linear and slightly nonlinear systems.

[0107] In some embodiments, the force / interaction control strategy includes: precisely controlling the contact force or fine displacement of the tool tip when the tool makes direct contact and interacts with the cell, executing an interaction strategy generated by the planning layer; the execution layer employs a force / position hybrid control framework to allow control of position / velocity in some degrees of freedom and control of force in the contact direction degree of freedom; for example, during grasping, controlling the closed position of the gripper while monitoring and limiting the grasping force to not exceed a set threshold to prevent damage to the cell. Real-time force feedback is provided via force sensors (integrated into the gripper tip or the wrist of the manipulator). This is crucial for manipulating delicate and fragile cells because it actively controls the interaction force, avoiding excessive force or operational failure that could result from pure position control.

[0108] In some embodiments, the visual servoing control strategy includes: the execution layer directly adjusting the pose of the tool using real-time visual information from the perception layer; using image-based visual servoing (IBVS) to directly define the error (e.g., the pixel distance between tool feature points and target cell feature points) on the image plane, and calculating the required tool motion through the controller to make the image error approach zero.

[0109] Understandably, the advantage of visual servoing lies in its ability to compensate for calibration errors and minor environmental changes. It directly performs closed-loop control based on what is "seen," achieving extremely high relative positioning accuracy, making it ideal for final alignment before cell grasping or needle tip positioning during microinjection. Visual servoing is a technology that uses visual information (such as camera images) to control robot movement in real time. Its core is to analyze changes in the target's position in the image to calculate the robot's control quantities (such as displacement and rotation) so that the target reaches the desired state (such as remaining centered in the image). Image-based visual servoing (IBVS) is one of the most commonly used branches, directly using image features (such as the target's pixel coordinates, area, and angle) as feedback to construct a control closed loop without needing to estimate the target's three-dimensional spatial position.

[0110] In some embodiments, when the system starts, system initialization and task definition include:

[0111] S710 performs a comprehensive self-test: automatically detects and confirms physical connections and basic communication with hardware interfaces including microscopes, micromanipulators, positioning platforms, and force sensors.

[0112] The S720 system loads and initializes all core software modules, including the image processing algorithm library, motion control engine, planning decision unit, and force feedback processing unit.

[0113] S730: To ensure operational accuracy, the system automatically calibrates the micromanipulation tools, setting their tip position, opening and closing range, and zero-point force.

[0114] S740: Activate the microscopic imaging system and micro-braking platform, establish a unified operating coordinate system, and prepare for subsequent sensing and action.

[0115] The S750 system receives task instructions input by the user through an interactive interface. These instructions can be high-level, near-natural language descriptions. After receiving the instructions, the system's built-in semantic understanding module parses these natural language instructions and converts them into structured, machine-understandable task objectives and parameters, laying the foundation for subsequent autonomous planning and execution.

[0116] When the system is shut down, a system reset includes the following steps:

[0117] S810: The system autonomously confirms that the task has been completed as expected based on the task objectives.

[0118] S820: Clear the current subtask queue, reset the internal state machine, perform tool state reset, and move the operator to the preset safe position.

[0119] S830 saves all log files and important results data from this run for subsequent analysis, tracing, or system performance evaluation. When needed, the system reset process can also shut down individual modules and hardware according to a set procedure.

[0120] Based on the foregoing, it should be noted that traditional manual operation typically has a positioning error of 5-10 μm, and the clamping force control relies entirely on feel, which can easily lead to cell damage. This system achieves a positioning accuracy of ±0.5 μm through visual servo control, with a force sensor feedback accuracy of ±10 nN. It can automatically adjust the clamping force according to cell type (e.g., 50-150 nN for embryonic cells and 200-300 nN for HeLa cells). In embryonic stem cell transfer experiments, the average positional deviation of manual operation is 8.3 μm, while the deviation of this system is only 0.8 μm, improving accuracy by over 90%. Traditional manual operation of a single cell takes an average of 30-60 seconds and cannot be parallelized; this system has an average single-cell operation time of only 8.2 seconds and supports parallel operation of multiple manipulators. Through standardized operating procedures and algorithmic control, this system reduces the CV value to 3.5% and improves the consistency of operations within the same batch by 85%. The system automatically records the operational parameters (force, position, time, etc.), image data, and result status of each cell, forming a complete log that allows for the traceability of every operational detail, providing a reliable basis for experimental replication and data analysis; while the coefficient of variation (CV value) of results obtained by manual operation is usually 15-25%, and the deviation of results from different operators can reach 30%.

[0121] In summary, this application aims to overcome the shortcomings of existing cell manipulation technologies, such as low automation, reliance on highly skilled manual labor, low operational efficiency, poor reliability of results, high cell damage rate, and lack of dynamic feedback and adaptive adjustment capabilities. To this end, this invention proposes a fully autonomous cell operating system based on embodied intelligence, with the following objectives: (1) to provide an automated cell operating system with autonomous perception, planning, and execution capabilities, realizing the automation and intelligence of operations such as cell clamping, injection, controllable deformation, and micro-assembly, so as to achieve the goal of highly intelligent and high-precision fully automated cell manipulation and get rid of the dependence of cell manipulation on highly skilled manual labor; (2) to improve the precision of cell manipulation, reduce the cell damage rate, and ensure cell viability; (3) to support high-throughput parallel operation, improve task processing efficiency, and meet the needs of scenarios such as single-cell sequencing and drug screening; (4) to achieve standardization and traceability of the operation process, ensure the consistency and reliability of experimental results, and provide intelligent and standardized cell manipulation solutions for life science research, precision medicine, drug development, and other fields.

[0122] This application adopts a fully closed-loop control architecture of "perception-planning-execution": multimodal data is collected through the hardware interface layer (image, micromanipulation, positioning platform, tactile module), fused and processed by the perception layer to form environmental cognition, dynamic strategies are generated by the planning layer, and finally executed and fed back by the action layer, achieving full-process autonomy without human intervention. This application also employs multimodal perception and high-precision environmental modeling technology: integrating high-resolution microscopic imaging, force sensing, and environmental sensors, it extracts cell and tool contour features through algorithms such as NL-Means denoising, U-Net semantic segmentation, and YOLOv8 target detection; and constructs an octree map with semantic labels to achieve dynamic cell tracking, tool 3D pose estimation, and global cognition of the operating environment, providing multi-dimensional data support for precise operation. Furthermore, this application proposes an intelligent planning and adaptive decision-making mechanism: a hierarchical task network (HTN) is used to decompose complex tasks into atomic operation sequences, and collision-free motion is ensured through global path planning (RRT*) and local obstacle avoidance (DWA) using the dynamic window method; an innovative integration of imitation learning and reinforcement learning is used to generate interactive strategies, which can adaptively adjust clamping force, injection speed, etc. according to cell morphology and mechanical properties to achieve low-damage operation.

[0123] This application achieves significant breakthroughs in operational precision, efficiency, cell safety, and standardization compared to traditional manual and semi-automatic systems through the deep integration of embodied intelligence technology and cell manipulation technology.

[0124] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fully autonomous cellular operating system based on embodied intelligence, characterized in that, This includes system hardware architecture and system software architecture; The system hardware architecture includes a micromanipulation execution system, a precision positioning system, a multimodal sensing system, and an intelligent computing and control center; wherein, the micromanipulation execution system is used for cell-environment interaction, the precision positioning system is used for sample carrying and environmental control, the multimodal sensing system is used to provide key information, and the intelligent computing and control center is used for computational decision-making; The system software architecture includes a perception layer, a planning layer, an execution layer, and an interaction layer; the system software architecture is used for: The perception layer performs image preprocessing, target localization and tracking, semantic mapping of the operating environment, semantic understanding of the operating scene, outputs system status information and transmits it to the planning layer. The system status information is received through the planning layer, and high-level task instructions from the user interaction layer interface are received to clarify the operation objectives, objects, and constraints. The planning layer performs instruction parsing, feasibility verification, autonomous task decomposition and sub-task optimization, autonomous tool selection, path planning, interaction strategy generation, multi-objective optimization, risk assessment and replanning, and generates an executable sequence of target instructions. The execution layer receives and parses the target instruction sequence, transforms the parsed instructions into actual control signals to drive hardware devices, calculates low-level action instructions based on imitation learning and reinforcement learning, performs real-time monitoring and adjustment, generates feedback and state updates, and constructs an execution safety mechanism. The system monitors and manages itself by receiving user-input task commands through an interactive layer, displaying microscopic images and system status, providing system configuration, operation tool selection, and task scheduling functions. Specifically, system initialization and task definition are required when the system starts up, and system reset is required when the system shuts down.

2. The fully autonomous cellular operating system based on embodied intelligence as described in claim 1, characterized in that, The micromanipulation system includes a micromanipulator, a micromanipulation end tool, and a force sensing device. The micromanipulator includes two micromanipulation arms, which have submicron-level positioning accuracy and a range of motion that can cover the cell manipulation workspace. The micromanipulation arms can perform operations including cell clamping, injection, controlled deformation, gentle pushing, and microassembly through autonomous collaboration. The micromanipulation end effector can be autonomously switched according to the specific task and interact directly with the cells; the micromanipulation end effector includes a microgripper and a microinjection needle, and is made of biocompatible materials and integrates a micro-drive mechanism to achieve precise operation; The force sensing device is integrated into the micromanipulation end tool to acquire force feedback information in real time. This allows the sensing layer to use the force feedback information to sense the mechanical properties of cells and determine the contact state, ensuring low damage to fragile cells and enabling non-invasive cell manipulation. The non-invasive cell manipulation includes adaptive adjustment of clamping force and control of injection depth and speed. The precision positioning system includes a precision positioning platform on which cell manipulation is performed. The surface of the precision positioning platform is integrated with an automated fixation device that adapts to different sample containers. The precision positioning platform has multiple autonomously controlled degrees of freedom of motion, which can be finely adjusted through each degree of freedom to adjust the position and orientation of the sample container, and cooperate with the micromanipulation execution system to complete cell manipulation tasks in a large area or a specific region.

3. The fully autonomous cellular operating system based on embodied intelligence as described in claim 1, characterized in that, The multimodal sensing system includes a microscopic imaging system, a microscopic visual motion platform, and an environmental state sensor; The microscopic imaging system includes an optical microscope equipped with a high-resolution microscopic camera for cell identification, localization, tracking, and state assessment. The microscopic imaging system can be installed on the autonomously controlled multi-degree-of-freedom microscopic vision motion platform to dynamically expand the perception range of the vision system, autonomously scan the entire sample area, and automatically track cells or manipulation tools that have moved out of the current field of view, so as to adapt to large-size samples or high-throughput screening tasks. The environmental state sensor includes a temperature sensor and a pH sensor. The environmental state sensor is used to construct a multi-dimensional monitoring network of the cell environment to comprehensively perceive the state of the cell culture microenvironment. The intelligent computing and control hub includes a graphics workstation, a server, and a high-performance industrial computer to run perception algorithms, planning algorithms, and control algorithms; the perception algorithms are used for image processing, pattern recognition, and scene understanding; the planning algorithms are used for task planning, motion planning, and decision making; and the control algorithms are used for servo control and force control. The intelligent computing and control center also includes a data acquisition interface, which is equipped with a high-speed image acquisition card and a multi-channel sensor interface card for the transmission of various types of sensor data.

4. The fully autonomous cellular operating system based on embodied intelligence as described in claim 1, characterized in that, The process of image preprocessing, target localization and tracking, semantic mapping of the operating environment, semantic understanding of the operating scene, outputting system state information and transmitting it to the planning layer through the perception layer includes: Remove random noise and background impurities while preserving edge details of cells and manipulation tools during denoising; Enhance local contrast in the image to make cell outlines, internal structures, and features of the manipulation tools clearer, while suppressing excessive noise amplification. When the contrast exceeds the preset first contrast threshold, automatic threshold segmentation is performed to distinguish the target from the background; in complex scenes where there is cell adhesion and the contrast is lower than the preset second contrast threshold, semantic segmentation is performed to extract the pixel-level contours of cells and manipulation tools. The cell membrane boundary and the edge of the manipulation tool are identified, and the image preprocessing is performed through denoising, enhancement, segmentation and edge detection to optimize the quality of the microscopic image and extract the contour features of the cell and the manipulation tool. For tool parts with relatively fixed shapes, rapid localization is performed to assist in tracking static features or tool markers in the field of view, and shape analysis is performed by combining the contour information obtained from image segmentation. Deep learning-based methods for object detection, instance segmentation, pose estimation, and object tracking; In the target detection process, the YOLO series network is used to quickly locate and select all target objects of interest in the microscopic image and classify them; wherein the target objects include target cells, surrounding other cells and manipulation tools; During instance segmentation, the target is located and the precise outline of each cell and tool is segmented pixel by pixel; In the pose estimation process, the 3D pose of the cell manipulation tool is estimated by combining the results of target detection and instance segmentation. During target tracking, the output of target detection and instance segmentation is combined with the DeepSORT tracking algorithm to achieve stable and robust tracking of target cells and manipulators across consecutive frames.

5. The fully autonomous cellular operating system based on embodied intelligence as described in claim 4, characterized in that, The process of image preprocessing, target localization and tracking, semantic mapping of the operating environment, semantic understanding of the operating scene, outputting system status information and transmitting it to the planning layer through the perception layer also includes: Construct an operational environment map with semantic information to provide the planning module with global environmental awareness, supporting path planning, obstacle avoidance, and multi-objective task allocation; Visual odometry is used to estimate the motion of the camera or stage through continuous image sequences; image feature points are extracted and inter-frame matching is performed, and pose optimization is performed through bundle adjustment; semantic information obtained from detection and segmentation is fused into the mapping process to generate an octree map with semantic labels; loop closure detection is implemented to correct accumulated errors and ensure accuracy. The system extracts low-level features through a visual layer, represents object attributes and instantaneous states through an object interaction layer, and determines the understanding of the operation process through a conceptual layer; it also performs semantic understanding of the operation scenario. The Transformer network is used to process time series information, identify the current operation stage, understand the relationship between the tool and the cell, identify abnormal situations, and predict the next operation or identify abnormalities. The system state information obtained from the perception layer is output to the planning layer in a structured form. The system state information includes: the position, posture, state, environment map, interaction state, and scene semantics of cells or operating tools.

6. The fully autonomous cellular operating system based on embodied intelligence as described in claim 1, characterized in that, The process involves instruction parsing, feasibility verification, autonomous task decomposition and sub-task optimization, autonomous tool selection, path planning, interaction strategy generation, multi-objective optimization, risk assessment and replanning through the planning layer, and generating an executable sequence of target instructions, including: Parse instructions to accurately understand task intent, and verify task feasibility based on the current state of the perception layer, so as to avoid the system attempting to perform operations that are physically or logically impossible. The complex tasks given by the user are automatically decomposed into a series of logically coherent, smaller, and more specific executable subtasks, namely atomic operation sequences, using the Hierarchical Task Network (HTN) planner. During the decomposition process, the system will perform preliminary optimizations, such as selecting the grabbing point with the lowest energy consumption or the shortest path. When the system is equipped with a variety of manipulation tools, including grippers and microneedles, the rule-based decision-making system queries the preset rule base to select the most suitable tool based on the sub-task type, the target cell size measurement value provided by the perception layer, and the current availability status of each tool. After the tool selection is completed, a motion trajectory from the current position to the target position is planned for the selected operation tool based on the preset two-stage planning strategy. The trajectory must be collision-free, smooth and safe. A hybrid learning strategy combining imitation learning (IL) and reinforcement learning (RL) is adopted to plan the interaction strategy when the tool interacts directly with the cell in order to control the fine action sequence and parameters and reduce physical damage to fragile cells. Multi-objective optimization is achieved by defining a comprehensive cost function, which assigns different weights to different objectives according to their importance and then sums them up in a weighted manner. When generating the initial plan, the planning layer evaluates each key step based on models or historical data; When the perception layer detects a significant change in the environment, the planning can automatically trigger a replanning mechanism; starting from the current failed or unexpectedly interrupted state, it can quickly call the relevant functions of path planning and strategy generation in the relevant planning modules to generate a new follow-up plan that adapts to the current situation; The plan, obtained through decomposition, selection, pathing, strategy, optimization, and risk assessment, is converted into a sequence of target instructions that the underlying controller can directly understand and execute. The sequence of target instructions includes: target pose sequence, motion parameters, force / torque control targets, and discrete event instructions.

7. The fully autonomous cellular operating system based on embodied intelligence as described in claim 6, characterized in that, The two-stage planning strategy includes: Global path planning is performed in a relatively static environment model, based on the map constructed by the perception layer or the current view snapshot, using the RRT* algorithm. During trajectory execution, dynamic window method (DWA) is used to adjust local paths and plan speeds, based on real-time updated perception information. In the path planning process, the RRT* algorithm is used for macro-global path planning, and the Dynamic Window Method (DWA) is used to give the system real-time responsiveness, enabling the system to flexibly avoid suddenly appearing floating debris obstacles or make immediate adjustments to minor changes in the environment, ensuring the smoothness and safety of the movement. The interaction strategies include a grab-and-place strategy and an injection strategy; In the grasping and placement strategy, based on the precise cell contours and poses provided by the perception layer, a fine final approach trajectory and gripper end pose are planned, and control parameters are generated using a hybrid learning strategy based on imitation learning (IL) and reinforcement learning (RL). In the injection strategy, the puncture position and entry angle are planned; and the hybrid learning strategy is used to determine the basic puncture action pattern through imitation learning, and to fine-tune the force feedback control and speed curve during the puncture process through reinforcement learning, so as to adapt to the toughness of different cell membranes, perform rapid, accurate, and low-damage punctures, and precisely control the injection volume and rate.

8. The fully autonomous cellular operating system based on embodied intelligence as described in claim 1, characterized in that, The process of receiving and parsing the target instruction sequence through the execution layer, converting the parsed instructions into actual control signals to drive the hardware device, calculating low-level action instructions based on imitation learning and reinforcement learning, performing real-time monitoring and adjustment, generating feedback and state updates, and constructing an execution safety mechanism includes: The system receives the target instruction sequence from the planning layer, performs automatic parsing to understand the specific actions and parameter settings to be executed at each time step, and prepares to distribute them to the corresponding underlying controllers. Based on motion control strategy, force / interaction control strategy and visual servo control strategy, the parsed instructions are converted into actual control signals to drive the hardware device; An interactive policy network is generated through imitation learning and reinforcement learning. A neural network inference engine is used as a dedicated actuator to receive the current state, including visual features and force feedback, as input to the interactive policy network and to calculate low-level motion commands, including joint velocity, gripper closing rate, and applied force increment. The low-level motion commands are transmitted to the motion or force controller for execution. During the execution of the low-level motion commands, the system status and external environment are continuously monitored, and fine-tuning or anomaly handling are performed. By continuously reading and processing real-time data from various sensors, the underlying controller can handle small disturbances and errors; at the same time, it monitors the execution error of the deviation between the expected state and the actual state, whether the force feedback exceeds the limit, and whether the visual target is lost. If a deviation less than the preset first allowable threshold is detected, the controller will compensate automatically; if a deviation exceeding the preset second allowable threshold or execution failure is detected, the execution layer will immediately stop the current action and send a feedback signal of execution failure and abnormal status to the planning layer, triggering the replanning process of the planning layer. The execution layer feeds back the results of the action execution and the latest information obtained during the process to the perception layer, reports the execution results of each sub-task or key step to the planning layer, updates environmental data, and transmits the latest sensor data obtained during the execution of the action to the perception layer. Protective measures are implemented at both the hardware and control software levels through the planning layer to prevent damage to cells, equipment, or the environment; the movement speed and acceleration of the manipulator are limited, and the range of motion of the tool is restricted, by setting maximum force / torque output limit thresholds in the controller.

9. The fully autonomous cellular operating system based on embodied intelligence as described in claim 8, characterized in that, The motion control strategy includes: driving the micromanipulator to track the collision-free path generated by the planning layer; and adopting a strategy that combines model-based feedforward control and PID feedback control. The feedforward control uses the dynamic model of the manipulator to predict the required driving force / torque to compensate for inertia and gravity effects and improve tracking accuracy. Based on real-time position / attitude errors from sensors, feedback adjustment is performed via a PID controller to suppress disturbances and model uncertainties; this combined control strategy aims to achieve fast and smooth trajectory tracking, ensuring that the tool moves precisely along the planned path; The force / interaction control strategy includes: precisely controlling the contact force or fine displacement of the tool tip when the tool makes direct contact and interacts with the cell, and executing the interaction strategy generated by the planning layer; the execution layer adopts a force / position hybrid control framework to allow control of position / velocity in some degrees of freedom and control of force in the contact direction degree of freedom; and providing real-time force feedback through a force sensor. The visual servoing control strategy includes: the execution layer directly adjusts the tool's pose using real-time visual information from the perception layer; and image-based visual servoing (IBVS) is used to directly define the error on the image plane, and the controller calculates the required tool motion to make the image error approach zero.

10. The fully autonomous cellular operating system based on embodied intelligence as described in claim 1, characterized in that, When the system starts, the system initialization and task definition include: Perform a comprehensive self-test: automatically detect and confirm the physical connection and basic communication with hardware interfaces including microscopes, micromanipulators, positioning platforms and force sensors; The system will load and initialize all core software modules, including an image processing algorithm library, a motion control engine, a planning decision unit, and a force feedback processing unit. To ensure operational accuracy, the system automatically calibrates the micromanipulation tools, specifying their tip position, opening and closing range, and zero-point force. The microscopic imaging system and micro-braking platform were activated, and a unified operating coordinate system was established to prepare for subsequent sensing and action. The system receives task instructions input by the user through an interactive interface; these can be high-level, near-natural language descriptions; after receiving the instructions, the system's built-in semantic understanding module will parse these natural language instructions and convert them into structured, machine-understandable task objectives and parameters, laying the foundation for subsequent autonomous planning and execution; When the system is shut down, the system reset includes: The system automatically confirms that the task has been completed as expected based on the task objectives; Clear the current subtask queue, reset the internal state machine, perform a tool state reset, and move the operator to the preset safe position; Save all log files and important results data from this run for subsequent analysis, tracing, or system performance evaluation.

Images