Control method of multi-mode sensing oral saliva suction system

Abstract

The invention relates to a control method of a multi-mode perception oral saliva suction system. The control method comprises the following steps: 1) data acquisition: continuously acquiring data by a camera and a depth camera; zero calibration is carried out on the force sensor; 2) environment perception and recognition: an environment perception module processes the data, recognizes the position of a saliva pool, and detects the positions of a tongue body and an odontoscope; 3) motion planning and execution: a motion planning module plans a motion track; the airflow regulating valve is turned to a medium negative pressure level; (4) force monitoring and dynamic adjustment are carried out, wherein a force sensor avoids the spraying pile; the visual system dynamically tracks saliva; 5) task completion and quit: the visual system judges that the saliva pool is basically cleared or directly issues a'stop 'instruction, and the system prepares to execute the next task. The control method of the multi-mode perception oral saliva suction system can accurately perceive a complex oral environment, intelligently plan a saliva suction path, and can be seamlessly integrated with existing medical equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to an oral diagnosis and treatment system, in particular to a control method of a multi-modal sensing oral saliva suction system, and belongs to the technical field of medical robots. BACKGROUND

[0002] In the process of oral diagnosis and treatment, it is necessary to continuously remove saliva, blood and flushing liquid in the patient's oral cavity to maintain a clear view of the surgical area. At present, this operation mainly relies on medical staff to complete it manually with a saliva suction tube, which has the following disadvantages: 1) increases the workload of medical staff, which is easy to cause fatigue; 2) the position precision and stability of manual operation are poor, which may block the doctor's view or cause discomfort to the patient; 3) increases the risk of cross-infection between doctors and patients.

[0003] In the prior art, although there are some integrated oral treatment robots or simple saliva suction devices, there are still obvious deficiencies. For example, although the Purevac HVE system of Dentsply Sirona company integrates the saliva suction function, it needs to bring its own saliva suction system, which leads to expensive equipment and poor flexibility, and does not have the ability of autonomous perception and control based on artificial intelligence. The modular saliva suction robot of FlexDental GmbH and other companies has low adaptability and only supports the original factory system. In addition, some autonomous saliva suction robot prototypes in the laboratory research stage (such as the system proposed by Su et al.), although they can remove about 95% of the liquid in a specific experiment, their real-time sensing accuracy, motion control flexibility and compatibility with traditional saliva suction equipment in dynamic, narrow and unstructured real oral environments still far from meet the requirements of clinical practice.

[0004] Therefore, in order to solve the above technical problems, it is necessary to provide an innovative control method of a multi-modal sensing oral saliva suction system to overcome the defects in the prior art. SUMMARY

[0005] To solve the above problems, the purpose of the present application is to provide a control method of a multi-modal sensing oral saliva suction system which can accurately, safely and efficiently automatically complete oral saliva suction operation.

[0006] To achieve the above object, the technical scheme adopted by the present application is as follows: a control method of a multi-modal perception oral saliva suction system, which adopts a multi-modal perception oral saliva suction system, comprising a robot mechanical arm unit, a multi-modal perception unit, a calculation and control unit and a modular saliva suction execution unit; wherein the multi-modal perception unit is integrated at the end of the mechanical arm and is driven by the robot mechanical arm, and comprises a sensing platform, a depth camera, a camera and a mechanical sensor; the modular saliva suction execution unit is installed on the multi-modal perception unit; the calculation and control unit is connected to and controls the robot mechanical arm unit, the multi-modal perception unit and the modular saliva suction execution unit. The control method comprises the following steps: 1) Data acquisition: the mechanical arm moves from the standby position to a predefined oral cavity observation starting point; the multi-modal perception unit starts to work, the camera and the depth camera continuously collect data; the force sensor performs zero point calibration; 2) Environment perception and identification: the environment perception module of the calculation and control unit processes the data in real time; within 0.1 seconds, the left lower molar saliva pool located beside the left lower molar in the oral cavity is identified, and the tongue in the oral cavity and the position of the mouth mirror are detected; 3) Motion planning and execution: based on the result of step 2), the motion planning module calculates a trajectory from the current position, bypassing the tongue, avoiding the mouth mirror and finally reaching the left lower saliva pool; the control instruction is sent to the mechanical arm, and the mechanical arm starts to move; at the same time, the calculation and control unit sends an instruction to the rotating motor of the air flow regulating valve to rotate the valve core to a moderate negative pressure level; 4) Force monitoring and dynamic adjustment: during the movement, the force sensor of the man-machine cooperation module detects that the saliva suction pipe collides with the tongue, and then the mechanical arm makes an avoidance action in the opposite direction of the force; at the same time, the visual system dynamically tracks the saliva to guide the saliva suction pipe to track the remaining saliva; when the saliva suction pipe reaches the target point and starts to suck saliva, if it is found that the liquid is viscous, the system can automatically increase the valve opening degree to increase the suction force; 5) Task completion and exit: the visual system judges that the saliva pool has been basically cleaned, or directly issues a "stop" instruction, moves to the standby position smoothly, the air flow regulating valve is completely closed, and the system is ready to perform the next task.

[0007] The control method of the multi-modal perception oral saliva suction system of the present application further comprises that the robot mechanical arm unit adopts a 6-DOF or 7-DOF mechanical arm.

[0008] The control method of the multi-modal perception oral saliva suction system of the present application further comprises that the calculation and control unit comprises an environment perception module, a motion planning module and a man-machine cooperation module; each module communicates through ROS.

[0009] The control method of the multi-modal perception oral saliva suction system of the application further comprises that the modular saliva suction execution unit comprises a connecting piece, an L-shaped support arm, a joint base, a saliva suction pipe and an airflow regulating valve; one end of the connecting piece is connected with a mechanical arm, and the other end is connected with the L-shaped support arm; the airflow regulating valve is connected to the L-shaped support arm; the joint base is installed on the airflow regulating valve and connected with the saliva suction pipe; and the airflow regulating valve can regulate the size of negative pressure airflow.

[0010] The control method of the multi-modal perception oral saliva suction system of the application further comprises that step 1) is specifically that four cameras are responsible for capturing high-resolution two-dimensional RGB color images inside the oral cavity to provide color and texture information for subsequent object recognition and classification; a depth camera synchronously acquires three-dimensional point cloud data of the scene to depict the geometric profile and spatial relationship of complex structures such as the tongue, teeth and soft tissue inside the oral cavity; and simultaneously, three six-dimensional force / torque sensors continuously monitor the contact force and torque between the robot and the oral tissue in X, Y and Z directions with millisecond-level accuracy to provide direct physical interaction information for the compliant control and safety monitoring of the system.

[0011] The control method of the multi-modal perception oral saliva suction system of the application further comprises that step 2) is specifically: 2-1), fine preprocessing of the original data obtained in step 1): distortion correction and color enhancement are performed on the RGB image, and noise reduction filtering and down-sampling processing are performed on the depth point cloud to improve the subsequent calculation efficiency; 2-2), through the pre-calibrated coordinate transformation parameters, the processed RGB image color information and three-dimensional point cloud geometric structure are accurately pixel-level registered to generate a fusion point cloud containing color and spatial information, and a complete oral three-dimensional environment model is constructed; 2-3), the calculation and control unit starts the deep learning model for parallel inference, adopts a DeepLabV3+ semantic segmentation network pre-trained and optimized on an oral data set to realize pixel-level classification of the RGB image, accurately outlines the profiles and boundaries of different categories such as "saliva / water area", "tongue", "soft tissue", "teeth" and "doctor's instrument"; and then a YOLOv8 target detection model is used to quickly locate key targets, i.e. saliva accumulation area and standardized saliva suction pipe, and generate a two-dimensional boundary box thereof; for the detected saliva suction pipe, further combined with its known three-dimensional model and image two-dimensional key points, the six-degree-of-freedom pose thereof in the world coordinate system is solved through the perspective n-point algorithm, including three-dimensional position and three-dimensional attitude; 2-4), finally outputting a real-time updated three-dimensional semantic map, which contains the geometric and color information of the environment, and gives each key target a semantic label, a three-dimensional boundary box and an accurate pose.

[0012] The control method of the multi-modal perception oral saliva suction system of the application further comprises that in the step 2-1), the image data preprocessing method is: 2-1), image distortion correction: for a camera, a Zhang Zhengyou calibration method is used to obtain a camera intrinsic matrix K and a distortion coefficient For each frame of the collected original image, pixel remapping is performed by using a radial distortion and tangential distortion model: wherein (x, y) is a normalized image coordinate, so as to eliminate the barrel distortion caused by the wide-angle lens; 2-2), point cloud denoising: for the original point cloud data generated by the depth camera, a statistical outlier removal algorithm is used to calculate the average distance d of each point to k nearest neighbors, calculate the global average distance mu and the standard deviation sigma; all noise points satisfying d > mu + alpha * sigma are removed, wherein alpha is a threshold coefficient, so as to eliminate the flying point noise caused by the water mist or high reflection in the oral cavity.

[0013] The control method of the multi-modal perception oral saliva suction system of the application further comprises that the step 3) is specifically: 3-1), task understanding based on the three-dimensional semantic map generated in the step 2), the recognized "saliva / water area" is defined as a task target point, and "tongue", "soft tissue" and "doctor's instrument" are marked as obstacles that need to be avoided in real time; 3-2), a path planner adopts a sampling-based RRT* algorithm or an optimization-based method, takes the current state of the mechanical arm as the starting point, takes the dynamically updated target saliva suction point as the ending point, and in consideration of the three-dimensional spatial distribution of all static and dynamic obstacles, calculates an optimal trajectory with no collision, shortest path and smooth motion in real time; 3-3), a deep fusion visual servo control strategy is adopted, real-time visual feedback is continuously used to correct the trajectory during the movement of the mechanical arm, when the tongue displacement is detected, the planner will immediately trigger local re-planning to ensure that the dynamic obstacles are always avoided; at the same time, the stiffness parameter of the mechanical arm end is dynamically adjusted according to the real-time feedback of the six-dimensional force sensor: when the contact force is perceived to increase, the stiffness of the mechanical arm is actively reduced, and the contact is buffered by generating a small compliant displacement.

[0014] The control method of the multi-modal perception oral saliva suction system of the application further comprises that the step 4) is specifically: 4-1), the six-dimensional force sensor continuously monitors the multi-dimensional contact information between the robotic arm end effector and oral tissue at an ultra-high sampling rate of over 1000Hz, including forces Fx, Fy, and Fz in three directions and torques Mx, My, and Mz on three axes; when the system detects that the contact force in any direction exceeds the preset safety threshold, it immediately triggers a compliant retraction based on the impedance control principle, that is, the robotic arm will produce a smooth displacement in the opposite direction of the contact force, while dynamically reducing the end effector stiffness; 4-2) The deep learning model performs real-time analysis of the continuous video stream. By comparing the changes in semantic segmentation results between adjacent frames, it can perceive the movement or shrinkage of the saliva pool, or the unexpected movement of the tongue or cheek tissue. This dynamic perception enables the system to recalculate the three-dimensional coordinates of the target saliva suction point in real time and guide the robotic arm to perform adaptive tracking and suction. 4-3) Based on the visual characteristics of the saliva area, namely the reflective properties exhibited by the area size and liquid viscosity, the system achieves precise matching of saliva suction intensity by adjusting the opening of the airflow regulating valve in real time: increasing the negative pressure for a large amount of thin saliva, and appropriately reducing the suction for a small amount of viscous liquid.

[0015] The control method of the multimodal sensing oral suction system of the present invention may further be: step 5) specifically includes: 5-1) When the system determines through real-time visual analysis that the saliva removal work has reached the preset standard, that is, the target area is less than a certain threshold, or the liquid reflective characteristics disappear, or the system receives an external stop command from the doctor and triggers the touch interface, the task termination and safe exit process is activated. 5-2), the system calculates an optimal collision-free trajectory from the current position of the suction tube to the preset standby position based on the real-time three-dimensional environment map of the oral cavity. The trajectory generation process takes into account the safety clearance between each link of the robotic arm and the oral cavity structure. 5-3), the robotic arm then executes the trajectory at a pre-set smooth speed curve; when the robotic arm completely exits the oral cavity area and reaches the standby position, the system automatically performs follow-up operations: closes the airflow regulating valve to stop negative pressure suction, resets the status flags of each software module, and prepares to respond to the next task command.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The control method of the multimodal sensing oral suction system of the present invention integrates multimodal sensing and adaptive robot control, realizing precise, safe and autonomous suction in a dynamic and narrow oral environment.

[0017] 2. The multimodal sensing oral suction system of the present invention can be directly connected to existing medical equipment. While breaking through the efficiency bottleneck of traditional manual operation and reducing the risk of cross-infection, it significantly lowers the application threshold, solves the core pain points of high cost and difficulty in implementation of similar intelligent prototype machines, and achieves a key leap from "laboratory prototype" to "clinical application". Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall structure of the multimodal sensing oral suction system of the present invention.

[0019] Figure 2 yes Figure 1 A schematic diagram of the robotic arm unit, multimodal sensing unit, and saliva suction execution unit in the diagram.

[0020] Figure 3 yes Figure 1 A schematic diagram of the multimodal sensing unit and the saliva suction execution unit in the diagram.

[0021] Figure 4 yes Figure 3 A schematic diagram of the airflow regulating valve in the diagram.

[0022] Figure 5 This is a flowchart of the control method for the multimodal sensing oral suction system of the present invention.

[0023] Figure 6 This is a software architecture diagram of the computing and control unit of the present invention. Detailed Implementation

[0024] Please refer to the instruction manual appendix. Figure 1 To be continued Figure 4 As shown, the present invention is a multimodal sensing oral suction system, which consists of several parts, including a robotic arm unit 1, a multimodal sensing unit 2, a computing and control unit 3, and a modular suction execution unit 4.

[0025] The robotic arm unit 1 employs a 6-DOF or 7-DOF robotic arm 1, possessing flexible movement and force control capabilities. In this embodiment, the robotic arm 1 includes a robot base 11, a rotary table 12, an upper arm 13, a connector 14, a sleeve 15, a lower arm 16, and a flange 17. The end of the robotic arm 1 is equipped with a quick-connect interface for connecting a standard saliva suction tube 44, ensuring rapid installation and replacement. The robotic arm 1 is designed with lightweight construction and safety in mind, avoiding harm to the patient while ensuring movement accuracy in confined oral environments.

[0026] The multimodal sensing unit 2 is integrated into the end of the robotic arm 1 and driven by the robotic arm 1. It consists of several parts, including a sensing platform 21, a depth camera 22, a camera 23, and a force sensor 24 located at the end of the suction tube 44. The camera 23 acquires two-dimensional color images of the oral cavity, while the depth camera 22 acquires three-dimensional point cloud data to reconstruct the three-dimensional structure of the oral cavity and identify areas of saliva accumulation, the tongue, soft tissue, and medical instruments. The force sensor 24 is a six-dimensional force / torque sensor that monitors the contact force between the robot and oral tissues in real time, ensuring smooth and safe operation. Multimodal data fusion provides comprehensive environmental information, laying the foundation for intelligent decision-making.

[0027] The modular suction actuation unit 4 is mounted on the multimodal sensing unit and connects to the existing negative pressure suction device 5 on the dental chair via a quick-connect interface. The modular suction actuation unit 4 includes a connector 41, an L-shaped support arm 42, a pagoda-shaped connector base 43, a standardized suction tube 44, and an airflow regulating valve 45. One end of the connector 41 is connected to the robotic arm 1, and the other end is connected to the L-shaped support arm 42; the airflow regulating valve 45 is connected to the L-shaped support arm 42. The connector base 43 is mounted on the airflow regulating valve 45 and connects to the standardized suction tube 44. The standardized suction tube 44 features a quick-replacement design, is compatible with existing dental equipment, and reduces operating costs. The airflow regulating valve 45 adjusts the magnitude of the negative pressure airflow and includes a valve body 451, a valve core 452, an inlet / outlet port 453, an adjusting port 454, and a rotary motor 455. By controlling the rotary motor, the airflow magnitude is adjusted, achieving adaptive control of the suction intensity. This modular design allows the system to directly utilize existing negative pressure systems without additional investment, improving practicality and cost-effectiveness.

[0028] The computing and control unit 3 connects to and controls the robot arm unit 1, the multimodal sensing unit 2, and the modular suction execution unit 4. The computing and control unit 3 includes an environmental perception module, a motion planning module, and a human-robot collaboration module; each module communicates via ROS to ensure efficient synchronization of data flow and control commands. Its architecture is shown in the appendix to the manual. Figure 6 As shown in the diagram, the environmental perception module employs lightweight deep learning models (YOLOv8, DeepLab, etc.) to process visual data in real time, enabling semantic segmentation, target detection, and pose estimation of saliva accumulation areas, the tongue, soft tissue, and medical instruments. The motion planning module, based on visual servoing and force feedback, generates motion trajectories that combine obstacle avoidance, precise positioning, and compliant contact. This module includes multiple adaptive saliva suction modes for different oral cavity regions. The human-robot collaboration module enables shared control between the doctor, robot, and patient. When abnormal forces are detected or medical instruments are identified as entering critical areas, the system can automatically pause or adjust its movement.

[0029] Please refer to the continuing instructions. Figure 5 As shown, the control method of the above-mentioned multimodal sensing oral suction system includes the following steps: 1) Data Acquisition: The robotic arm 1 moves from the standby position to a predefined starting point for oral cavity observation. The multimodal sensing unit 2 starts working, and the camera 23 and depth camera 22 continuously acquire data. The force sensor 24 performs zero-point calibration.

[0030] Specifically, four cameras 23 are responsible for capturing high-resolution two-dimensional RGB color images of the oral cavity to provide rich color and texture information for subsequent object recognition and classification. A depth camera 22 simultaneously acquires three-dimensional point cloud data of the scene, accurately depicting the geometric contours and spatial relationships of complex structures such as the tongue, teeth, and soft tissues inside the oral cavity. Meanwhile, three six-dimensional force / torque sensors integrated into the saliva suction tube interface continuously monitor the contact forces and torques between the robot and oral tissues in the X, Y, and Z directions with millisecond-level accuracy, providing direct physical interaction information for the system's compliant control and safety monitoring.

[0031] Furthermore, to ensure the system can effectively respond to rapid changes within the oral cavity (such as tongue movement and saliva flow), the data acquisition frame rate of all sensors is set to above 30Hz. The acquired data is transmitted to the computing and control unit 3 via the sensing platform 21 to achieve true real-time perception. In addition, the system ensures strict synchronization of visual and force data in the time domain through hardware triggering or precise software timestamp mechanisms. Furthermore, the coordinate framework of all data is unified beforehand through hand-eye calibration and sensor zero-position calibration, thus building an accurate and reliable data foundation for multimodal data fusion and intelligent decision-making in subsequent steps.

[0032] 2) Environmental Perception and Recognition: The environmental perception module of the computing and control unit 3 processes the data in real time. Within 0.1 seconds, it identifies the saliva pool located next to the lower left molar in the oral cavity, and simultaneously detects the tongue and the position of the mouth mirror.

[0033] Specifically, step 2-1) involves fine preprocessing of the raw data obtained in step 1). Distortion correction and color enhancement are performed on the RGB image to optimize quality. Simultaneously, noise reduction filtering and downsampling are applied to the depth point cloud to improve subsequent computational efficiency.

[0034] In this sub-step, the image data preprocessing method is as follows: 2-1) Image distortion correction: For camera 23, the Zhang Zhengyou calibration method is used to obtain the camera intrinsic parameter matrix K and distortion coefficients. For each frame of the original image acquired, pixel remapping is performed using radial and tangential distortion models: Where (x, y) are normalized image coordinates. This eliminates barrel distortion caused by wide-angle lenses.

[0035] 2-2) Point cloud noise reduction: For the raw point cloud data generated by depth camera 22, the statistical outlier removal algorithm is adopted. For each point, the average distance d to its k nearest neighbors is calculated, and the global average distance μ and standard deviation σ are calculated. All noise points that satisfy d>μ+α⋅σ are removed, where α is the threshold coefficient, in order to eliminate flying point noise caused by water mist or high reflectivity in the oral cavity.

[0036] 2-2) By using pre-calibrated coordinate transformation parameters, the processed RGB image color information is precisely registered with the three-dimensional point cloud geometry to generate a fused point cloud containing color and spatial information, thus constructing a complete three-dimensional oral cavity environment model.

[0037] (2-3) The computation and control unit 3 initiates a deep learning model for parallel inference. It employs a DeepLabV3+ semantic segmentation network pre-trained and optimized on an oral dataset to achieve pixel-level classification of RGB images, accurately outlining the contours and boundaries of different categories such as "saliva / water region," "tongue," "soft tissue," "teeth," and "doctor's instruments." Then, the YOLOv8 object detection model is used to quickly locate key targets, namely the saliva accumulation area and the standardized saliva suction tube, and generate their two-dimensional bounding boxes. For the detected saliva suction tube 44, its known three-dimensional model and two-dimensional key points in the image are further combined, and its six-degree-of-freedom pose in the world coordinate system, including three-dimensional position and three-dimensional orientation, is solved using a perspective n-point algorithm.

[0038] (2-4) Finally, a real-time updated 3D semantic map is output. This map not only contains the geometric and color information of the environment, but more importantly, it assigns semantic labels, 3D bounding boxes and their precise poses to each key target, providing a structured and understandable environmental perception foundation for subsequent motion planning and control.

[0039] 3) Motion Planning and Execution: Based on the results of step 2), the motion planning module calculates a trajectory from the current position, around the tongue, avoiding the mouth mirror, and finally reaching the lower left saliva pool. Control commands are sent to robotic arm 1, and robotic arm 1 begins to move; at the same time, the calculation and control unit 3 sends a command to the rotary motor 455 of the airflow regulating valve 45 to rotate the valve core 452 to a medium negative pressure level.

[0040] Specifically, 3-1), based on the three-dimensional semantic map generated in step 2), task understanding is performed, the identified "saliva / water area" is defined as the task target point, and dynamic entities such as "tongue", "soft tissue" and "doctor's instruments" are marked as obstacles that need to be avoided in real time.

[0041] 3-2) The path planner uses a sampling-based RRT* algorithm or an optimization-based method. Starting from the current state of the robotic arm and ending at the dynamically updated target saliva suction point, it calculates an optimal trajectory in real time that combines collision-free, shortest path and smooth motion, taking into account the three-dimensional spatial distribution of all static and dynamic obstacles.

[0042] 3-3) A deep fusion visual servo control strategy is adopted, continuously using real-time visual feedback to correct the trajectory online during the movement of the robotic arm. When displacement of the tongue is detected, the planner immediately triggers local replanning to ensure that dynamic obstacles are always avoided. At the same time, the stiffness parameters of the robotic arm end effector are dynamically adjusted based on real-time feedback from the six-dimensional force sensor: when an increase in contact force is sensed, the robotic arm actively reduces stiffness, buffering the contact by generating a small compliant displacement to avoid mechanical damage to the oral tissue.

[0043] Furthermore, the system integrates multiple adaptive suction modes to handle different clinical scenarios: a "buccal mode" with a gentle angle and faster speed is used in the open buccal region. In the sensitive sublingual region, it automatically switches to a "sublingual mode" with a more cautious path, slower movement speed, and lower force control threshold. When the vision system detects that the doctor's instruments have entered the critical operating area, it immediately activates the "doctor intervention mode," controlling the suction tube to either avoid or follow, ensuring that it does not interfere with the doctor's diagnostic and treatment operations. These strategies together constitute a motion control system that can adapt to complex oral environments and combines precision and safety.

[0044] 4) Force Monitoring and Dynamic Adjustment: During movement, the force sensor in the human-machine collaboration module detects a collision between the suction tube and the tongue, and controls the robotic arm to perform an avoidance action in the opposite direction of the force. Simultaneously, the vision system dynamically tracks the saliva to guide the suction tube to follow the remaining saliva. When the suction tube reaches the target point and begins suctioning, if the liquid is found to be viscous, the system can automatically increase the valve opening to enhance suction.

[0045] This step is specifically as follows: 4-1), the six-dimensional force sensor 24 continuously monitors the multi-dimensional contact information between the end effector of the robotic arm 1 and the oral tissue at an ultra-high sampling rate exceeding 1000Hz, including forces Fx, Fy, and Fz in three directions and torques Mx, My, and Mz on three axes. When the system detects that the contact force in any direction exceeds the preset safety threshold (such as a force of 1N or a torque of 0.1Nm), it immediately triggers a compliant retraction based on the principle of impedance control. That is, the robotic arm will produce a smooth displacement in the opposite direction of the contact force, while dynamically reducing the end effector stiffness, withdrawing sensitively like a "finger" with tactile sense, thereby eliminating overload contact within milliseconds and fundamentally eliminating the risk of mechanical tissue damage.

[0046] 4-2) The deep learning model performs real-time analysis of the continuous video stream. By comparing the changes in semantic segmentation results between adjacent frames, it can perceive the movement or shrinkage of the saliva pool, or the unexpected movement of the tongue or cheek tissue. This dynamic perception enables the system to recalculate the three-dimensional coordinates of the target saliva suction point in real time and guide the robotic arm to perform adaptive tracking and suction, ensuring that the best saliva suction effect can be maintained even when the patient moves involuntarily.

[0047] 4-3) Based on the visual characteristics of the saliva area, namely the reflective properties exhibited by the area size and liquid viscosity, the system adjusts the opening of the airflow regulating valve 45 in real time to achieve precise matching of saliva suction intensity: increasing the negative pressure for a large amount of thin saliva and appropriately reducing the suction for a small amount of viscous liquid, demonstrating the intelligent characteristics of situational adaptation.

[0048] 5) Task completion and exit: The vision system determines that the saliva pool has been basically cleared, or directly issues a "stop" command, smoothly moves to the standby position, the airflow regulating valve is completely closed, and the system is ready to execute the next task.

[0049] Specifically, 5-1), when the system determines through real-time visual analysis that the saliva removal work has reached the preset standard, that is, the target area is less than a certain threshold, or the liquid reflective characteristics disappear, or the system receives an external stop command from the doctor and triggers the touch interface, the task termination and safe exit process is activated.

[0050] 5-2) Based on a real-time 3D environment map of the oral cavity, the system calculates an optimal collision-free trajectory from the current position of the suction tube to a preset standby position. The trajectory generation process considers the safety clearances between the robotic arm's links and the oral cavity structure. This trajectory generation process fully considers the safety clearances between the robotic arm's links and the oral cavity structure, ensuring that there will be no scraping or collision with teeth, tongue, or soft tissues during the withdrawal process.

[0051] (5-3) The robotic arm then executes the trajectory at a pre-set smooth speed curve, with smooth and natural movements to avoid patient discomfort caused by sudden movements. Once the robotic arm has completely withdrawn from the oral cavity and reached the standby position, the system automatically performs follow-up operations: closing the airflow regulating valve to stop negative pressure suction, resetting the status flags of each software module, and preparing to respond to the next task command. This complete exit process ensures a perfect conclusion to a single saliva suction procedure, providing seamless technical support for continuous, multi-round oral treatments.

[0052] In summary, this invention, through the deep integration of multimodal perception, adaptive robot control, and modular integrated design, successfully achieves precise, safe, efficient, and autonomous saliva suction in complex oral environments. Its unique design breaks through the bottlenecks of cost and practicality, propelling this technology from "laboratory prototype" to "clinical application," and providing a practical solution for the intelligent upgrading of oral diagnosis and treatment.

[0053] The above-described specific embodiments are merely preferred embodiments of this invention and are not intended to limit this invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.

Claims

1. A method of controlling a multi-modal, perception-based oral suction system, the method comprising: It adopts a multi-modal perception oral saliva suction system, including a robot arm unit, a multi-modal perception unit, a calculation and control unit and a modular saliva suction execution unit; wherein, the multi-modal perception unit is integrated at the end of the robot arm and driven by the robot arm, which includes a sensing platform, a depth camera, a camera and a force sensor; the modular saliva suction execution unit is installed on the multi-modal perception unit; the calculation and control unit is connected and controls the robot arm unit, the multi-modal perception unit and the modular saliva suction execution unit; The control method comprises the following steps: 1) data acquisition: the robot arm moves from the standby position to a predefined oral cavity observation starting point; the multi-modal perception unit starts to work, the camera and the depth camera continuously collect data; the force sensor performs zero point calibration; 2) environment perception and identification: the environment perception module of the calculation and control unit processes the data in real time; within 0.1 seconds, the left lower molar saliva pool located beside the left lower molar in the oral cavity is identified, and the tongue in the oral cavity and the position of the mouth mirror are detected; 3) motion planning and execution: based on the result of step 2), the motion planning module calculates a trajectory from the current position, bypassing the tongue, avoiding the mouth mirror, and finally reaching the left lower saliva pool; the control instruction is sent to the robot arm, and the robot arm starts to move; at the same time, the calculation and control unit sends an instruction to the rotating motor of the air flow regulating valve to rotate the valve core to a moderate negative pressure level; 4) force monitoring and dynamic adjustment: during the movement, the force sensor of the man-machine cooperation module detects that the saliva suction pipe collides with the tongue, and then the robot arm makes an avoidance action in the opposite direction of the force; at the same time, the visual system dynamically tracks the saliva to guide the saliva suction pipe to track the remaining saliva; when the saliva suction pipe reaches the target point and starts to suck saliva, if it is found that the liquid is viscous, the system can automatically increase the valve opening to increase the suction force; 5) task completion and exit: the visual system judges that the saliva pool has been basically cleaned, or directly issues a "stop" instruction, smoothly moves to the standby position, the air flow regulating valve is completely closed, and the system is ready to perform the next task.

2. The method of controlling a multi-modal awareness oral suction system of claim 1, wherein: The robot arm unit adopts a 6-DOF or 7-DOF robot arm.

3. The method of controlling a multi-modal awareness oral suction system of claim 1, wherein: The calculation and control unit includes an environment perception module, a motion planning module and a man-machine cooperation module; Each module communicates through ROS.

4. The method of controlling a multi-modal awareness oral suction system of claim 1, wherein: The modular saliva suction execution unit includes a connecting piece, an L-shaped support arm, a joint base, a saliva suction pipe and an air flow regulating valve; one end of the connecting piece is connected to the robot arm, and the other end is connected to the L-shaped support arm; the air flow regulating valve is connected to the L-shaped support arm; the joint base is installed on the air flow regulating valve and connected with the saliva suction pipe; the air flow regulating valve can adjust the size of the negative pressure air flow.

5. The method of controlling a multi-modal awareness oral suction system of claim 1, wherein: The step 1) is specifically: 4 cameras are responsible for capturing high-resolution two-dimensional RGB color images inside the oral cavity to provide color and texture information for subsequent object recognition and classification; a depth camera synchronously acquires three-dimensional point cloud data of the scene to depict the geometric profile and spatial relationship of complex structures such as the tongue, teeth, and soft tissue inside the oral cavity; at the same time, 3 six-dimensional force / torque sensors continuously monitor the contact force and torque between the robot and the oral tissue in X, Y, and Z directions with millisecond-level accuracy, providing direct physical interaction information for the system's compliant control and safety monitoring.

6. The method of controlling a multi-modal awareness oral suction system of claim 1, wherein: The step 2) is specifically: 2-1), fine preprocessing of the original data obtained in step 1): distortion correction and color enhancement of the RGB image, and noise reduction filtering and down-sampling processing of the depth point cloud to improve subsequent computing efficiency; 2-2), through the pre-calibrated coordinate transformation parameters, accurately pixel-level registration of the processed RGB image color information and three-dimensional point cloud geometry, generating a fusion point cloud containing color and spatial information, and constructing a complete oral three-dimensional environment model; 2-3), the calculation and control unit starts the deep learning model for parallel inference, uses the DeepLabV3+ semantic segmentation network pre-trained and optimized on the oral data set to realize pixel-level classification of the RGB image, and accurately outlines the contours and boundaries of different categories such as "saliva / water area", "tongue", "soft tissue", "teeth", and "doctor's instruments"; then uses the YOLOv8 target detection model to quickly locate the key targets, i.e. the saliva accumulation area and the standardized saliva suction tube, and generates its two-dimensional bounding box; for the detected saliva suction tube, further combine its known three-dimensional model and image two-dimensional key points, and solve its six-degree-of-freedom pose in the world coordinate system, including three-dimensional position and three-dimensional attitude, through the perspective n-point algorithm; 2-4), finally output a real-time updated three-dimensional semantic map, which contains the geometric and color information of the environment, and gives each key target a semantic label, a three-dimensional bounding box, and an accurate pose.

7. The method of controlling a multi-modal awareness oral suction system of claim 6, wherein: In step 2-1), the image data preprocessing method is: 2-1), image distortion correction: for the camera, Zhang Zhengyou calibration method is used to obtain camera intrinsic parameter matrix K and distortion coefficient For each frame of the collected original image, pixel remapping is performed by using a radial distortion and tangential distortion model: where (x, y) are normalized image coordinates, so as to eliminate the barrel distortion brought by the wide-angle lens; 2-2), point cloud noise reduction: for the original point cloud data generated by the depth camera, a statistical outlier removal algorithm is used to calculate the average distance d of each point to its k nearest neighbors, calculate the global average distance μ and standard deviation σ; remove all noise points that satisfy d>μ+α⋅σ, where α is a threshold coefficient, to eliminate flying point noise caused by water mist or high reflectivity in the oral cavity.

8. The method of controlling a multi-modal awareness oral suction system of claim 1, wherein: The step 3) is specifically: 3-1), based on the three-dimensional semantic map generated in step 2), understand the task, define the recognized "saliva / water area" as the task target point, and mark "tongue", "soft tissue", and "doctor's instruments" as dynamic entities that need to be avoided in real time; 3-2), the path planner adopts a sampling-based RRT* algorithm or an optimization-based method, taking the current state of the robotic arm as the starting point and the dynamically updated target saliva suction point as the ending point, to calculate an optimal trajectory in real time in consideration of the three-dimensional spatial distribution of all static and dynamic obstacles, with the optimal trajectory having no collision, shortest path, and smooth motion; 3-3), a deep fusion visual servo control strategy is adopted, and real-time visual feedback is continuously used to correct the trajectory during the motion of the robotic arm, and when displacement of the tongue is detected, the planner will immediately trigger local re-planning to ensure that the dynamic obstacles are always avoided; at the same time, the stiffness parameter of the end of the robotic arm is dynamically adjusted according to the real-time feedback of the six-axis force sensor: when the contact force is perceived to increase, the robotic arm actively reduces the stiffness to buffer the contact by generating a small compliant displacement.

9. The method of controlling a multi-modal awareness oral suction system of claim 1, wherein: The step 4) is specifically: 4-1), the six-axis force sensor continuously monitors the multi-dimensional contact information between the end of the robotic arm and the oral tissue at an ultra-high sampling rate of more than 1000 Hz, including forces Fx, Fy, Fz in three directions and moments Mx, My, Mz in three axes; when the system detects contact force in any direction exceeding the preset safety threshold, it immediately triggers a compliant fallback based on the impedance control principle, that is, the robotic arm will generate a smooth displacement in the opposite direction of the contact force, while dynamically reducing the end stiffness; 4-2), a deep learning model analyzes the continuous video stream in real time, and perceives the position movement, range reduction of the saliva pool, or accidental movement of the tongue and cheek tissue by comparing the changes in semantic segmentation results between adjacent frames. This dynamic perception enables the system to immediately recalculate the three-dimensional coordinates of the target saliva suction point and guide the robotic arm to perform adaptive tracking and suction; 4-3), the system adjusts the opening degree of the airflow regulating valve in real time according to the visual features of the saliva area, i.e. the area size and the reflective characteristics of the liquid viscosity, to achieve precise matching of the suction strength: for a large amount of thin saliva, increase the negative pressure, and for a small amount of viscous liquid, appropriately reduce the suction force.

10. The method of controlling a multi-modal awareness oral suction system of claim 1, wherein: The step 5) is specifically: 5-1), when the system determines through real-time visual analysis that the saliva removal work has reached the preset standard, i.e. the target area is smaller than a certain threshold, or the liquid reflective characteristics disappear, or an external stop instruction from the doctor is received, the task terminates and the safety exit process is activated; 5-2), the system calculates an optimal collision-free trajectory from the current position of the saliva suction tube to the preset standby position based on the real-time three-dimensional environment map of the current oral cavity, and considers the safety clearance of each link of the robotic arm and the oral structure during the trajectory generation process; 5-3), the robotic arm then executes the trajectory at a pre-set smooth speed curve; after the robotic arm completely exits the oral cavity area and reaches the standby position, the system automatically performs the post-operations: closes the airflow regulating valve to stop the negative pressure suction, resets the state flags of each software module, and prepares to respond to the next task instruction.

Images