Hysteroscope surgery robot control method and system based on multi-modal perception

Abstract

The application belongs to the technical field of digital surgery, and particularly relates to a hysteroscope surgery robot control method and system based on multi-modal perception. The method comprises collecting visual, auditory, tactile, force sense and pose multi-modal input signals, uniformly performing time synchronization and perception trajectory construction, combining semantic behavior intention inference and tactile timing judgment, and realizing multi-dimensional consistency verification of operation instructions. After determining the effective result, control execution parameters are extracted and a mechanical arm is driven to perform accurate operation, and the execution process is recorded and control thresholds are updated to optimize feedback performance. The system comprises an input signal acquisition module, a six-dimensional force sensor module, a semantic inference module, a modal fusion module and the like, and a closed-loop structure from perception to control is constructed, which significantly improves the operation accuracy, interactive stability and safety response capability of the hysteroscope robot in surgery.

Description

Technical Field

[0001] This invention belongs to the technical field of digital surgery, specifically relating to a control method and system for a hysteroscopic surgical robot based on multimodal perception. Background Technology

[0002] Hysteroscopic surgery is a minimally invasive diagnostic and treatment technique widely used in gynecological clinics, with advantages such as minimal trauma, rapid recovery, and fewer postoperative complications. With the development of technologies such as image recognition, artificial intelligence, and robotic control, hysteroscopic surgery is gradually evolving towards intelligence and automation, showing great potential, especially in surgical path planning, precise instrument control, and intraoperative feedback.

[0003] Current hysteroscopic surgical robot control systems primarily rely on single-modal information (such as visual images) for path guidance and surgical manipulation. Most systems reconstruct the internal structure of the uterine cavity from 2D or 3D images and then execute instrument manipulations along preset paths. However, the uterine cavity environment is complex and subject to dramatic changes. Relying solely on visual signals often makes it difficult to accurately identify key lesion areas and avoid sensitive tissues, easily leading to operational errors or tissue damage. Furthermore, some systems attempt to incorporate force sensing to assist in manipulation, but these suffer from large feedback delays, high signal-to-noise ratios, and insufficient support for multi-target tasks, making it difficult to achieve dynamic adaptive control in complex surgical scenarios. Existing systems generally suffer from problems such as limited perception dimensions, poor environmental adaptability, and insufficient control precision, restricting their widespread application in precise and personalized hysteroscopic surgery.

[0004] To achieve a higher level of intelligent hysteroscopic surgery, it is urgent to introduce multimodal perception mechanisms and fusion control strategies to improve the robot system's dynamic understanding and collaborative response capabilities in surgical scenarios. Summary of the Invention

[0005] To address the above problems, the present invention aims to propose a control method for a hysteroscopic surgical robot based on multimodal perception, comprising the following steps:

[0006] S1. Acquire multimodal input signals: Acquire at least one frame of image data of the target area through an integrated vision device, acquire continuous audio data through a multi-channel auditory sensor, and obtain tactile pressure or stress change parameters as tactile data through a tactile feedback array; at the same time, the six-dimensional force sensor integrated at the end of the operating arm acquires the uterine wall contact state parameters in real time as force data, and the three-dimensional pose tracking module acquires the spatial pose information of the instruments in the uterine cavity during the operation as pose data.

[0007] S2. Synchronous calibration of multimodal signals: The image data, audio data, tactile data, force and pose data are marked with a unified timestamp based on the signal start time and the system reference clock to construct a time-synchronized multimodal perception trajectory;

[0008] S3. Semantic Behavioral Intent Inference: Collect voice commands and finger movement positions issued by the operator from the control terminal, and identify the gestures corresponding to the commands by comparing the voice content in the audio signal with the hand trajectory in the image, and establish a subset of directional intents;

[0009] S4. Operation timing judgment based on tactile pressure: Extract the tactile pressure change rate from the tactile sensor output, continuously monitor the pressure change trend after the operator contacts the operation terminal, calculate the average change rate, and make a judgment based on the preset stability threshold.

[0010] S5. Execute the synchronous discrimination strategy: Combine the semantic judgment and timing judgment conclusions to determine whether they are consistent; if they are consistent, proceed to the next operation; if they are inconsistent, output a warning message and suspend execution.

[0011] S6. Determine control execution parameters: Extract feature points of the target area from the image data, obtain the operator's hand description vector, and calculate the relative displacement. Control the movement trajectory of the robotic arm or the manipulation direction of the surgical tool through the relative displacement. The control parameters for controlling the movement trajectory of the robotic arm or the manipulation direction of the surgical tool are input to the control module after joint modeling and analysis by the modal fusion algorithm.

[0012] S7. Record and update operation data: During the execution of the robotic arm, the control input, motion trajectory and feedback signals are recorded in real time, and the input parameter thresholds are updated and the parameters are adjusted when the corresponding operation ends.

[0013] As a preferred technical solution, in step S2, the timestamp is applied to the frame capture time of image data, the start time of audio data acquisition, the start time of tactile pressure change, and the reading time of force / pose. It is uniformly processed by the time calibration module, and the perception trajectory is stored and retrieved based on a time series data structure.

[0014] As a preferred technical solution, in step S3, the hand trajectory vector extracted from the image data is compared with the pointing direction vector collected by the control terminal using a midpoint determination method, including:

[0015] Get the set of all points that the finger has moved forward through, and use that set as the finger's pointer;

[0016] Calculate the mean of all points in the set that the finger points to, and use that as the midpoint of the finger;

[0017] Obtain the set of operator pointers, calculate the mean of the operator pointer set, and use it as the intermediate point of the intent;

[0018] Compare the midpoint of the finger with the midpoint of the intention;

[0019] If the coordinates of the midpoint of the finger and the midpoint of the intention are within the preset error range, they are judged to be consistent, and it is considered that the finger is indeed pointing to the operator's intention.

[0020] Otherwise, if the difference between the two intermediate points exceeds the allowable error range, the operation is deemed invalid, the robot system will indicate that the operation is invalid, and the surgery will not be performed.

[0021] As a preferred technical solution, the calculation method for the pressure change rate in step S4 includes:

[0022] Record the total monitoring time, obtain the pressure change value per unit time, and calculate the average pressure change;

[0023] The average pressure change is compared with a preset amplitude threshold. If it exceeds the corresponding threshold, it is determined to be an unstable state; otherwise, it is determined to be a stable state.

[0024] As a preferred technical solution, in step S5:

[0025] When the semantic judgment conclusion is consistent with the timing judgment conclusion, the corresponding action is executed according to the displacement parameter;

[0026] When the conclusions are inconsistent, the instruction status is marked as non-executable, and an operation pause prompt is output.

[0027] As a preferred technical solution, in step S6, the control execution parameters include:

[0028] When performing motion control of the robotic arm, the target feature points in the image data are extracted synchronously with the position of the operating terminal and used as the starting reference point;

[0029] Obtain the operator's input instructions and form a hand model pointing vector based on the input instructions;

[0030] Extract real-time finger description vectors during execution;

[0031] Calculate the Euclidean distance between the hand model pointing vector and the real-time finger description vector;

[0032] Based on the system's preset pre-displacement amount, calculate the current control execution displacement amount;

[0033] The displacement amount, after being processed by the modal fusion algorithm, is input to the robot control module for real-time updates of the surgical tool's movement trajectory or angle adjustment.

[0034] As a preferred technical solution, step S7 includes:

[0035] A log data structure storage method based on periodic windows is established; the periodic windows are used to update input data in real time and to perform condition judgment and parameter correction on the input data;

[0036] Determine if there is a discrepancy between the input value and the output result.

[0037] If a deviation exists, the system calculates a correction value based on the deviation, adjusts the parameters, and updates the set of operating parameters.

[0038] The present invention also provides a hysteroscopic surgical robot control system based on multimodal perception for implementing the method, the control system comprising:

[0039] The input signal acquisition module is used to acquire image data, audio data, and tactile data, and synchronize them to a unified timeline;

[0040] A six-dimensional force sensor module is integrated into the end of the operating arm to monitor the contact status of the uterine wall in real time.

[0041] The three-dimensional spatial pose tracking module acquires the spatial pose information of the instruments in the uterine cavity in real time based on the visual positioning system.

[0042] The semantic inference module is used to extract operator voice and gesture behavior information and establish semantic-action mapping relationships;

[0043] The timing judgment module is used to analyze the rate of change of operator hand pressure and compare it with a stable threshold.

[0044] The synchronization judgment module is used to integrate the continuous change rate of operator hand pressure with the stable threshold, determine whether they are consistent, and output the corresponding synchronization flag bit;

[0045] The action execution module is used to extract control execution parameters and complete surgical actions under the condition of synchronous judgment and consistency.

[0046] The data recording module is used to record all input data, control commands and feedback signals as a surgical procedure log after the action is performed, and to update and calibrate parameters after the operation.

[0047] The modality fusion module is used to perform joint modeling and analysis of the fused multimodal features based on Bayesian estimation or graph neural networks to generate control reference parameters.

[0048] The status prediction module is used to combine historical operation trajectories with real-time input to predict future operation trends and issue avoidance signals in advance in high-risk areas.

[0049] As a preferred technical solution, the image data is three-dimensional image data acquired based on the hysteroscopy terminal, the audio data is the operator's voice command data, the tactile data is tactile pressure value and rate of change data, the force data is the triaxial force and triaxial torque signals output by the six-dimensional force sensor, and the pose data includes the position and attitude vector of the instrument end in three-dimensional space. Beneficial effects

[0050] This invention introduces a multimodal signal fusion mechanism to achieve temporal unification and semantic coordination of visual, auditory, tactile, force, and spatial pose information, avoiding the operational deviations caused by relying solely on single image recognition or voice control. In the complex environment of intrauterine surgery, the operator's true intention often needs to be reflected through multiple channels of signals. This invention, through timestamp synchronization and modal fusion modeling, constructs a more stable, continuous, and directional path for expressing operational intentions, thereby significantly improving the accuracy and response precision of control commands.

[0051] In terms of tactile control optimization, this invention determines the timing of operations based on the rate of change in tactile pressure and statistical characteristics, avoiding the one-sided reliance on instantaneous tactile pressure values ​​in traditional solutions. By establishing a joint judgment mechanism based on the average value and frequency of pressure changes, the system can effectively filter out interference signals generated by jitter, non-target contact, etc., triggering operation only when the tactile sensation is stable and consistent with the intended action. This strategy significantly reduces the risk of false triggering, and is particularly suitable for applications such as the uterine cavity where tissues are soft, space is narrow, and the precision of operation is extremely high.

[0052] This invention introduces a multi-dimensional verification and feedback correction mechanism in the synchronous discrimination and control parameter extraction stages. It not only establishes a dual consistency judgment of semantics and timing before execution but also records the control input and feedback trajectory in real time after operation. Threshold updates and parameter adjustments are achieved through the analysis of deviation values. A closed-loop adaptive control flow is constructed, enabling the system to continuously learn and dynamically correct, effectively addressing differences in operational responses between different operators and individual patients, and improving the versatility and personalized adaptability of the control system. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0054] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0055] To enhance understanding of the present invention, the present invention will be further described in detail below with reference to embodiments. These embodiments are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention. Example

[0056] according to Figure 1 As shown in the figure, this embodiment provides a detailed implementation of a hysteroscopic surgical robot control method based on multimodal perception, combining specific processes and signal processing methods. This method relies on an integrated sensor network and intelligent control system to achieve multimodal input recognition, semantic intent reasoning, operation timing judgment, and closed-loop control of command execution for the hysteroscopic surgical robot. The overall process includes seven steps.

[0057] S1. Acquire multimodal input signals:

[0058] Before the surgery begins, the control system initializes the entire operating environment, including setting surgical area parameters, activating various sensor modules, and establishing communication connections between the control host and each sensing terminal. It then enters the multimodal sensing phase, where the system collects raw data using multiple heterogeneous sensor sources.

[0059] Visual information is acquired by the camera module in the hysteroscopy terminal. This camera module has automatic focusing and high-definition imaging capabilities, and can stably output images under complex lighting and tissue backgrounds. During the acquisition process, a continuous video frame transmission method is used, and each frame image is marked to extract key targets in the uterine cavity, such as endometrial structures, lesion areas, bleeding points, etc., and to track the movement of instruments.

[0060] Audio information is collected via a directional microphone array deployed at the front of the control panel, which suppresses background noise and extracts only the operator's voice commands. Command words include standard terms such as "enter," "left," and "stop." To prevent accidental triggering by non-command voice commands, the system has a built-in semantic filtering module that recognizes only structurally complete voice segments that are clearly related to the task status.

[0061] The tactile information originates from the control panel of the operating terminal, which is covered with a high-density distributed pressure-sensitive film array. This array records in real time the intensity and path of pressure changes caused by the operator's hand touching the panel. This type of information is crucial for determining the operator's behavioral state, the duration of their actions, and the location of pauses.

[0062] Meanwhile, a six-dimensional force sensor is integrated at the end of the surgical robot's robotic arm to collect the force and torque values ​​in three directions on the contact surface. This is used to determine the contact dynamics between the instrument and the uterine cavity tissue and to identify whether there is slippage, rebound, or jamming.

[0063] In addition, the three-dimensional spatial pose information is acquired by the electromagnetic tracking system. This system works in conjunction with the implanted micro-positioning chip and fixed reference point to accurately track the spatial position and posture angle of the hysteroscopic lens and surgical instruments, ensuring the accuracy of subsequent execution path matching.

[0064] After all the aforementioned sensing information is collected, it is uploaded to the control center through a unified interface protocol, laying the foundation for subsequent data processing and fusion.

[0065] S2. Synchronous calibration of multimodal signals:

[0066] Multimodal signals have complex sources, and the sampling rates and response delays of each channel vary. Direct parallel processing can easily lead to timing misalignment and information drift. Therefore, this invention designs a unified clock synchronization mechanism.

[0067] The system first calls the high-precision hardware clock of the main control processor as a reference benchmark for various sensor data. All sensor data is automatically timestamped during acquisition, including the image frame capture time, the voice data start time, the tactile signal trigger time, and continuous sampling points for force and pose.

[0068] The synchronization module performs time alignment on various types of data and constructs a perceptual data structure according to a unified timeline. Interpolation alignment or sliding window methods are used to fill time differences, ensuring that different types of data correspond to the same behavioral cycle. Ultimately, a synchronized perceptual trajectory is formed with "time frames" as the basic unit, providing a unified input benchmark for semantic recognition and behavior judgment.

[0069] S3. Inference based on semantic behavior intent:

[0070] After receiving the synchronized sensing trajectory, the control system first enters the semantic analysis process. The goal of this process is to understand the operator's behavioral intentions conveyed through a combination of voice and gestures, thereby assisting in issuing control commands.

[0071] The speech analysis module converts the speech stream into structured instruction content and matches it with the current task stage based on the keyword list. For example, if the current stage is the advancement stage, the recognition of "forward" or "approach" has priority.

[0072] Simultaneously, the image processing module identifies the operator's hand model, extracts continuous motion trajectories from the image, and generates spatial direction vectors and motion trends. The control system determines whether the voice content is consistent with the image direction vector. For example, if the voice command is "adjust to the right," and the hand in the image is also pointing to the right of the target area, then the semantics and behavior are considered consistent.

[0073] To enhance robustness, the system calculates the midpoint of the hand trajectory and checks its overlap with the target area generated by the command direction vector. If the midpoint falls within the tolerance range of the target area, it is determined to be a valid intent.

[0074] If there is a contradiction between the voice and the image, for example, the voice command is "backward" but the hand movement is forward, the system will automatically interrupt the command and prompt the operator to confirm the operation to avoid incorrect execution.

[0075] S4. Determine the timing of operation based on tactile pressure:

[0076] After confirming the operator's intent, it is still necessary to determine whether they are in a stable state and whether they have completed the operation and entered a ready-to-execute phase. This determination is based on changes in the operator's hand pressure on the control panel.

[0077] The control system continuously monitors the changes in pressure amplitude and frequency per unit time. A reference window length (e.g., 1 second) is set, and the average value and number of changes in pressure fluctuation within this time period are statistically analyzed.

[0078] If the average change value within the time period exceeds the set amplitude threshold, and the number of changes exceeds the number threshold, the system determines that the current state is unstable and blocks the execution channel.

[0079] Conversely, if the average change value is below the amplitude threshold and the number of changes is below the number threshold, the system determines that the operator is in a stable pressing state, and the operation can be considered complete, proceeding to the next step.

[0080] This method effectively eliminates false triggers caused by brief hesitation, position adjustment, or unexpected shaking, ensuring the safety and precision of the surgical procedure.

[0081] S5. Execute the synchronization discrimination strategy:

[0082] To further enhance the accuracy of instructions, this invention establishes a dual synchronous discrimination mechanism for semantic intent and operation timing.

[0083] In steps S3 and S4, the system outputs a semantic judgment flag and a tactile timing judgment flag, respectively. These two flags are then sent to the synchronization discrimination module as inputs. This module determines whether both flags are valid simultaneously.

[0084] If the two match, the system generates an execution flag and initiates the subsequent process of extracting parameters and issuing motion paths for control commands.

[0085] If the two are inconsistent, the system will immediately lock the control channel and output a warning message "Command and action are not synchronized" on the operation terminal interface, prompting the operator to re-enter the command.

[0086] This synchronization strategy constructs a logical closed loop between multimodal inputs, preventing the system from exhibiting unexpected behavior due to misjudgment of a certain channel, and improving the robustness and fault tolerance of surgical control.

[0087] S6. Determine the control execution parameters:

[0088] After acquiring the execution flag, the system begins to extract key control parameters to generate the robot's motion path.

[0089] The image analysis module first extracts the boundary contour and positioning points of the target area from the current image frame, establishes the target spatial coordinate system, and determines the spatial relationship of the current instrument or lens.

[0090] Meanwhile, the system extracts the pointing vector from the gesture trajectory and combines it with directional or scale-related words in the speech (such as "five millimeters to the left") to perform parameter calibration.

[0091] Then, the fusion module is invoked to jointly model information such as image position, command vector, tactile feedback, pose change and force state, and a decision fusion algorithm (such as a neural network or Bayesian network) is used to generate the final control vector.

[0092] The control vector is decoded and converted into robot execution commands, including parameters such as propulsion speed, deflection angle, and tool gripping angle, which are directly transmitted to the robotic arm control board to achieve closed-loop drive of the operation path.

[0093] S7. Record and update operation data:

[0094] During the control process, the system records all input, processing and output data in real time to form a complete surgical log and to provide a basis for future adaptive parameter correction.

[0095] The system establishes a data log table based on a ring buffer structure, recording control input, pose trajectory, feedback force value, tactile state, and semantic judgment information in chronological order.

[0096] After an operation is completed, the system compares the control input with the actual feedback trajectory to analyze whether there is any response lag, displacement deviation or error drift.

[0097] If the deviation exceeds the system's preset threshold, the control parameters for the next operation will be automatically corrected based on historical deviation patterns to optimize the matching degree between target positioning and action response.

[0098] In addition, the system supports postoperative analysis mode, which can export log data for manual review or use it for retraining artificial intelligence models, thereby accumulating postoperative experience and controlling model evolution.

[0099] In summary, the multimodal control method of the present invention achieves highly stable and precise control execution in a minimally invasive surgical environment through the process of "intent confirmation - timing judgment - synchronous execution - closed-loop correction", thereby improving the practicality and safety of the robot in complex medical scenarios. Example

[0100] The following describes in detail a specific embodiment of a hysteroscopic surgical robot control system based on multimodal perception, in conjunction with the overall system architecture and functional modules of this invention. This system addresses the needs for acquiring, processing, fusing, and executing control of multi-dimensional sensory information during hysteroscopic surgery. Through modular design, it achieves high-precision, low-latency, and intelligent auxiliary operation control of the surgical process.

[0101] like Figure 2 As shown, the system of the present invention consists of ten core modules, namely: input signal acquisition module, six-dimensional force sensor module, three-dimensional spatial pose tracking module, semantic inference module, timing judgment module, synchronization judgment module, action execution module, data recording module, modality fusion module and state prediction module. The modules work together to form a complete closed loop from perception input, intent parsing, decision judgment to control feedback.

[0102] The input signal acquisition module serves as the entry point to the system's perception layer, deployed on the control terminal and execution arm infrastructure of the surgical robot. This module includes a high-definition image acquisition device, a directional microphone array, and a distributed tactile sensing panel. The image acquisition device, integrated into the hysteroscopy lens, captures high-definition 3D image data of the uterine cavity, supporting depth map overlay and image distortion correction, providing rich spatial information for target tissue recognition and instrument path planning. The audio channel acquires the operator's voice control commands in real time and extracts keywords and their temporal structure through noise suppression and semantic segmentation algorithms. The tactile sensors primarily record the pressure intensity change curve during the interaction between the operator's hand and the terminal, sampling the frequency of pressure fluctuations per unit time to provide a basis for judging the timing and state of the operation.

[0103] The six-dimensional force sensor module is integrated into the end effector of the surgical robot's manipulator arm, close to the interface between the instrument and the tissue. It collects linear contact force in three axes and torque feedback signals around these three axes in real time. This module employs a high-sensitivity piezoelectric force sensor combined with a high-speed A / D converter to ensure dynamic contact force information is output within milliseconds. By continuously monitoring this data, the system can determine whether the instrument is slipping, compressing, or pulling against the uterine wall, and dynamically adjust the execution parameters accordingly to prevent damage to delicate tissues.

[0104] The 3D spatial pose tracking module is crucial for achieving precise spatial manipulation in this system. This module employs a hybrid positioning system centered on visual positioning, utilizing an internal calibration grid of the hysteroscope lens in conjunction with external reference points to accurately track the spatial position and posture (including translation and rotation) of surgical instruments within the uterine cavity. The system generates continuous pose trajectories through feature point recognition and inter-frame image registration, comparing them in real-time with the ideal operational path established in the control model. Any trajectory deviation or posture abnormality triggers an automatic adjustment mechanism.

[0105] The semantic inference module, based on a collaborative mechanism of AI speech recognition and image recognition, is used to analyze the correspondence between voice commands and gestures issued by the operator. Internally, the module includes a voice command template library and a gesture direction library. The system first compares the collected voice data with the semantic templates and matches semantic labels according to the current task stage. Then, it extracts hand trajectory vectors from the image and determines whether their direction and amplitude match the target area. Finally, it outputs a semantic judgment flag indicating whether the "command-action" pair is consistent. The core of this module is to avoid ambiguity that may arise from judging solely based on voice or action, thus improving the robustness of behavior recognition under multimodal conditions.

[0106] The timing judgment module is primarily used to analyze whether the operator's tactile behavior after issuing an operation command meets the execution conditions. By processing data collected through the tactile channel in real time, the system calculates the average pressure change and the number of changes per unit time. If both indicators are within the system's set stable threshold range, a "timing valid" signal is output; if either indicator exceeds the limit, it is determined that the operator is still in an adjustment or hesitation state, and execution is not carried out. This module is used to block the triggering timing of erroneous operation commands, preventing execution errors caused by the operator not fully confirming their operational intentions.

[0107] The synchronization judgment module, located after the semantic judgment and timing judgment modules, primarily functions to fuse the results of both types of judgments and make a final decision on whether to execute. If both the semantic inference module and the timing judgment module output a valid status, a "synchronization confirmation" flag is generated, and the operation request is submitted to the execution module. If there is a discrepancy between the two, the system refuses to execute and outputs a prompt message on the control interface, requiring the user to re-enter the command. This module acts as the logical central hub for multimodal fusion verification in the system architecture, ensuring that all operation commands are implemented only when both the intent and the status are simultaneously satisfied.

[0108] After receiving the "synchronization confirmation" signal from the upstream module, the motion execution module begins the control parameter extraction and path generation process. This module first extracts the positioning information of the current target area from the image data, calculates the spatial relative displacement by combining it with the hand trajectory, and then jointly calibrates this displacement with the operational scale mentioned in the operator's voice (such as "advance five millimeters"). Subsequently, through the comprehensive control vector extracted by the fusion module, corresponding control execution parameters are generated, including the advancement speed, offset angle, and tool clamping angle. These parameters are then converted via an intermediate protocol and transmitted to the robotic arm drive system to complete the specific surgical actions.

[0109] The data logging module is used to build a complete database of behaviors, parameters, and feedback during surgery. This module synchronously records the following information according to fixed time windows and key event nodes: input image frames, audio stream, tactile pressure change curves, force feedback, instrument pose trajectory, system judgment flags, control vectors, and motion feedback. This data is not only used for postoperative behavior retrospection but also serves as training samples for subsequent parameter optimization, assisting the system in building an adaptive control model. After anonymization and classification, the recorded data supports local log querying, cloud storage, and physician behavior analysis.

[0110] The modality fusion module is the intelligent core of the system for complex decision modeling. After extracting features from various types of sensor data, this module uses a Bayesian inference model or a graph neural network for joint modeling, generating a multimodal fusion control parameter reference vector. The system constructs the prior probability structure in the fusion model using historical behavioral data, ensuring that the control result depends not only on the current input but also on past operating styles and environmental factors, thus improving the personalization and fault tolerance of the control. This module also outputs weighting factors for each modality, enabling the system to dynamically adjust its decision-making strategy based on the quality of the sensor signals.

[0111] The state prediction module is used for feedforward control judgment during execution. By analyzing the historical trajectory of the operation and the current input trend, it predicts the direction of the operation's evolution over a certain period of time. If the prediction results indicate that the target operation direction will enter a high-risk area (such as approaching the uterine wall, entering a blind zone, or undergoing rapid rotation), the system will automatically output a warning signal and trigger path correction suggestions. This module achieves time series prediction capabilities through a recurrent neural network and can work in conjunction with the modality fusion module to implement an active avoidance mechanism.

[0112] The modules are interconnected via a low-latency real-time bus protocol. The system is deployed on a high-performance embedded platform and employs a multi-threaded parallel processing structure. The image data consists of three-dimensional depth images acquired by the hysteroscopic terminal, the audio data is a high-precision voice command stream, the tactile data includes multi-point pressure sensing values ​​and rate of change curves, the force data is three-dimensional force and three-dimensional torque information, and the pose data is provided by the integrated navigation system, which provides spatial position and attitude vectors to ensure the integrity and accuracy of the perceived data.

[0113] In summary, the hysteroscopic surgical robot control system of the present invention, through a closed-loop structure of perception-decision-execution-feedback, combined with multimodal information fusion, semantic understanding and behavior prediction technologies, significantly improves the intelligence and control precision of surgical operations, providing solid technical support for complex minimally invasive intracavitary procedures.

[0114] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A hysteroscopic surgical robot control system based on multimodal perception, characterized in that, The control system includes: The input signal acquisition module is used to acquire image data, audio data, and tactile data, and synchronize them to a unified timeline; A six-dimensional force sensor module is integrated into the end of the operating arm to monitor the contact status of the uterine wall in real time. The three-dimensional spatial pose tracking module acquires the spatial pose information of the instruments in the uterine cavity in real time based on the visual positioning system. The semantic inference module is used to extract operator voice and gesture behavior information and establish semantic-action mapping relationships; The timing judgment module is used to analyze the rate of change of operator hand pressure and compare it with a stable threshold. The synchronization judgment module is used to integrate the continuous change rate of operator hand pressure with the stable threshold, determine whether they are consistent, and output the corresponding synchronization flag bit; The action execution module is used to extract control execution parameters and complete surgical actions under the condition of synchronous judgment and consistency. The data recording module is used to record all input data, control commands and feedback signals as a surgical procedure log after the action is performed, and to update and calibrate parameters after the operation. The modality fusion module is used to perform joint modeling and analysis of the fused multimodal features based on Bayesian estimation or graph neural networks to generate control reference parameters. The status prediction module is used to combine historical operation trajectories with real-time input to predict future operation trends and issue avoidance signals in advance in high-risk areas. The control method of the hysteroscopic surgical robot control system includes the following steps: S1. Acquire multimodal input signals: Acquire at least one frame of image data of the target area through an integrated vision device, acquire continuous audio data through a multi-channel auditory sensor, and obtain tactile pressure or stress change parameters as tactile data through a tactile feedback array; at the same time, the six-dimensional force sensor integrated at the end of the operating arm acquires the uterine wall contact state parameters in real time as force data, and the three-dimensional pose tracking module acquires the spatial pose information of the instruments in the uterine cavity during the operation as pose data. S2. Synchronous calibration of multimodal signals: The image data, audio data, tactile data, force and pose data are marked with a unified timestamp based on the signal start time and the system reference clock to construct a time-synchronized multimodal perception trajectory; S3. Semantic Behavioral Intent Inference: Collect voice commands and finger movement positions issued by the operator from the control terminal, and identify the gestures corresponding to the commands by comparing the voice content in the audio signal with the hand trajectory in the image, and establish a subset of directional intents; S4. Operation timing judgment based on tactile pressure: Extract the tactile pressure change rate from the tactile sensor output, continuously monitor the pressure change trend after the operator contacts the operation terminal, calculate the average change rate, and make a judgment based on the preset stability threshold. S5. Execute the synchronous discrimination strategy: Combine the semantic judgment and timing judgment conclusions, and determine whether the two are consistent; if they are consistent, proceed to the next operation; if they are inconsistent, output a warning message and suspend execution. S6. Determine control execution parameters: Extract feature points of the target area from the image data, obtain the operator's hand description vector, and calculate the relative displacement. Control the movement trajectory of the robotic arm or the manipulation direction of the surgical tool through the relative displacement. The control parameters for controlling the movement trajectory of the robotic arm or the manipulation direction of the surgical tool are input to the control module after joint modeling and analysis by the modal fusion algorithm. S7. Record and update operation data: During the execution of the robotic arm, the control input, motion trajectory and feedback signals are recorded in real time, and the input parameter thresholds are updated and the parameters are adjusted when the corresponding operation ends.

2. The hysteroscopic surgical robot control system based on multimodal perception according to claim 1, characterized in that, In step S2, the timestamp is applied to the frame capture time of image data, the start time of audio data acquisition, the start time of tactile pressure change, and the reading time of force / pose. It is uniformly processed by the time calibration module, and the perception trajectory is stored and retrieved based on a time series data structure.

3. The hysteroscopic surgical robot control system based on multimodal perception according to claim 1, characterized in that, In step S3, the hand trajectory vector extracted from the image data is compared with the pointing direction vector collected by the control terminal using a midpoint determination method, including: Get the set of all points that the finger has moved forward through, and use that set as the finger's pointer; Calculate the mean of all points in the set that the finger points to, and use that as the midpoint of the finger; Obtain the set of operator pointers, calculate the mean of the operator pointer set, and use it as the intermediate point of the intent; Compare the midpoint of the finger with the midpoint of the intention; If the coordinates of the midpoint of the finger and the midpoint of the intention are within the preset error range, they are judged to be consistent, and it is considered that the finger is indeed pointing to the operator's intention. Otherwise, if the difference between the two intermediate points exceeds the allowable error range, the operation is deemed invalid, the robot system will indicate that the operation is invalid, and the surgery will not be performed.

4. The hysteroscopic surgical robot control system based on multimodal perception according to claim 1, characterized in that, In step S4, the method for calculating the rate of change of pressure includes: Record the total monitoring time, obtain the pressure change value per unit time, and calculate the average pressure change; The average pressure change is compared with a preset amplitude threshold. If it exceeds the corresponding threshold, it is determined to be an unstable state; otherwise, it is determined to be a stable state.

5. The hysteroscopic surgical robot control system based on multimodal perception according to claim 1, characterized in that, In step S5: When the semantic judgment conclusion is consistent with the timing judgment conclusion, the corresponding action is executed according to the displacement parameter; When the conclusions are inconsistent, the instruction status is marked as non-executable, and an operation pause prompt is output.

6. The hysteroscopic surgical robot control system based on multimodal perception according to claim 1, characterized in that, In step S6, the control execution parameters include: When performing motion control of the robotic arm, the target feature points in the image data are extracted synchronously with the position of the operating terminal and used as the starting reference point; Obtain the operator's input instructions and form a hand model pointing vector based on the input instructions; Extract real-time finger description vectors during execution; Calculate the Euclidean distance between the hand model pointing vector and the real-time finger description vector; Based on the system's preset pre-displacement amount, calculate the current control execution displacement amount; The displacement amount, after being processed by the modal fusion algorithm, is input to the robot control module for real-time updates of the surgical tool's movement trajectory or angle adjustment.

7. The hysteroscopic surgical robot control system based on multimodal perception according to claim 1, characterized in that, Step S7 includes: A log data structure storage method based on periodic windows is established; the periodic windows are used to update input data in real time and to perform condition judgment and parameter correction on the input data; Determine if there is a discrepancy between the input value and the output result. If a deviation exists, the system calculates a correction value based on the deviation, adjusts the parameters, and updates the set of operating parameters.

8. The hysteroscopic surgical robot control system based on multimodal perception according to claim 7, characterized in that, The image data is three-dimensional image data acquired based on the hysteroscopy terminal, the audio data is the operator's voice command data, the tactile data is tactile pressure value and rate of change data, the force data is the triaxial force and triaxial torque signals output by the six-dimensional force sensor, and the pose data includes the position and orientation vector of the instrument end in three-dimensional space.

Images