Intelligent surgical robot control system based on multi-modal fusion control

Abstract

The invention discloses an intelligent surgical robot control system based on multi-modal fusion control, and relates to the technical field of robot control, the intelligent surgical robot control system comprises a remote control end, a local execution end, a consultation monitoring end, a picture sharing end, a service cloud platform and a communication base station; wherein the remote control end is used for inputting and outputting according to multi-modal data, sending a control instruction by using a control interface, and converting the control instruction into a robot control signal; the local execution end is used for receiving the control signal sent by the remote control end and driving the robot to execute corresponding operation; and the consultation monitoring end is used for acquiring the operation image transmitted by the local execution end, performing real-time consultation through video stream and data sharing, and providing collaborative decision support. Through cooperation of the remote control end and the local execution end, robot motion control, operation image real-time viewing and force feedback interaction are achieved, and meanwhile, in combination with the consultation monitoring end and the image sharing end, multi-party collaborative decision and real-time image sharing are supported.

Description

Technical Field

[0001] The present invention relates to the technical field of robot control, and more particularly to an intelligent surgical robot control system based on multi-modal fusion control. Background Art

[0002] With the development of medical technology, endoscopic surgery and surgical robot technology have gradually been introduced into the field of intracerebral hemorrhage surgery. The introduction of these technologies has made minimally invasive surgery possible, reduced trauma during the operation, shortened the patient's recovery time, and increased the success rate of the operation. In particular, endoscopic technology can guide doctors to observe deep brain structures through extremely small incisions, providing a clear view, enabling doctors to perform operations in a narrow space. However, endoscopic technology itself has limitations, mainly manifested as the lack of tactile feedback. Doctors can only rely on visual judgment to perform operations, which may not provide sufficient support for fine operations and complex surgeries, especially in emergency situations such as bleeding.

[0003] The introduction of surgical robot technology aims to solve these problems, especially by providing higher-precision control, stronger operation stability, and more intuitive force feedback, greatly enhancing doctors' operation capabilities in complex surgeries. Through precise control of the robotic arm, surgical robots can perform highly precise minimally invasive operations, overcoming the problems of hand tremors and unstable operations in traditional surgeries. The robot can perform real-time precise operations based on endoscopic images, making intracerebral hemorrhage surgery safer and more efficient. In addition, the robot system is usually equipped with a variety of sensors, including force feedback sensors and tactile perception systems, which can real-time feedback the contact force between the surgical tool and the tissue, enabling doctors to sense the force changes during the operation through the remote control system, and then precisely adjust the operation force to avoid damaging the brain tissue.

[0004] However, the current intracerebral hemorrhage surgical robot system still faces some technical bottlenecks. First, although the emergence of 5G network provides low-latency and high-bandwidth communication guarantee for remote surgery, in practical applications, the network stability and bandwidth requirements are still a technical challenge that cannot be ignored. Even though 5G technology can provide extremely low latency in theory, in some specific environments, especially when the hospital's network infrastructure is relatively complex, communication latency or network jitter may still affect the precision of the operation. Excessive latency or signal loss may cause the robot to respond slowly, affecting the normal progress of the operation.

[0005] In addition, the accuracy and real-time response of traditional force feedback systems pose another major challenge for intracerebral hemorrhage surgical robots. In traditional surgeries, surgeons judge the contact force between surgical instruments and tissues through touch. Especially during the operation, it is necessary to precisely perceive the toughness and elasticity of tissues to avoid excessive cutting or compression. Although existing robotic systems can provide a certain degree of force feedback, their accuracy and real-time performance still need to be improved. Force feedback signals may exhibit lag or errors, leading to inaccurate judgment of the operating force of surgical tools by doctors, thereby increasing the risk of surgical failure.

[0006] Therefore, although the introduction of surgical robots in intracerebral hemorrhage surgeries has significant advantages, there are still some problems in the existing technologies, such as communication delays, insufficient force feedback accuracy, limitations in image processing capabilities, and complex operation interfaces. These technical problems not only affect the actual application effect of surgical robots but also may bring unpredictable risks during the surgical process. Summary of the Invention

[0007] Based on this, it is necessary to provide an intelligent surgical robot control system based on multi-modal fusion control for the above-mentioned technical problems.

[0008] The present invention provides an intelligent surgical robot control system based on multi-modal fusion control. The intelligent surgical robot control system includes: a remote control terminal, a local execution terminal, a consultation and monitoring terminal, a video sharing terminal, a service cloud platform, and a communication base station;

[0009] Among them, the remote control terminal is used to send control commands through the control interface according to multi-modal data input and output, convert them into robot control signals, so as to complete robot motion control, real-time viewing of surgical images, reception and transmission of force feedback data, and perform precise robot operations;

[0010] The local execution terminal is used to receive the control signals sent by the remote control terminal, drive the robot to perform corresponding operations, and collect the surgical images, force feedback data, and robot motion state information during the surgical process in real time, and synchronously transmit them to the remote control terminal and the consultation and monitoring terminal;

[0011] The consultation and monitoring terminal is used to obtain the surgical images transmitted by the local execution terminal, conduct real-time consultations through video streams and data sharing, provide collaborative decision-making support, and establish a communication channel with the service cloud platform through the communication base station to build a data sharing network;

[0012] The video sharing terminal is used to establish a connection with the service cloud platform through the communication base station, act as a node of the data sharing network, and access the consultation and surgical video images in real time;

[0013] A service cloud platform is used to build a data sharing network among the local execution end, the consultation monitoring end, and the screen sharing end, and provide data backup, storage, and processing services;

[0014] A communication base station is used for 5G network communication, providing data transmission, low-latency communication, and high-bandwidth transmission, and using network slicing technology to process various data streams during the operation.

[0015] Furthermore, the remote control end includes: a visual image processing module, a force feedback display and perception module, a two-handed joystick operation control module, a decision support and assistance module, and a monitoring and security guarantee module;

[0016] Among them, the visual image processing module is used to build a three-dimensional stereoscopic image of the brain surgery area according to the endoscopic image in the received surgical image, combined with three-dimensional reconstruction technology, and optimize the image quality through real-time image processing to enhance the visibility of the lesion area and key structures;

[0017] The force feedback display and perception module is used to integrate tactile sensors, convert the force feedback data synchronized during the movement of the robot into tactile signals, and simulate the force perception during the operation;

[0018] The two-handed joystick operation control module is used for the operator to control the multi-degree-of-freedom control device for remote surgical operation and generate control instructions according to the surgical image, force feedback data, and robot movement state information fed back by the local execution end;

[0019] The decision support and assistance module is used to mark the lesion area, blood vessels, and tissues in the three-dimensional stereoscopic image in real time, identify the operation habits of the operator based on artificial intelligence, combine communication delay analysis, adaptively optimize the control instructions, and detect the accuracy of the operator during the operation by combining the dynamic kinematic model and real-time calculation, and output the optimized control signal;

[0020] The monitoring and security guarantee module is used to monitor the robot state, operation progress, and patient's physiological information during the operation, and perform real-time analysis on the operation behavior of the control device.

[0021] Furthermore, the decision support and assistance module includes: an image marking unit, an operator identification unit, an anti-delay compensation unit, a risk warning unit, and a decision output unit;

[0022] Among them, the image marking unit is used to obtain the real-time received endoscopic image, perform semantic segmentation and object detection on the endoscopic image using deep learning algorithms, extract the information of lesions, blood vessels, and tissues in the operation area, and mark them in the constructed three-dimensional stereoscopic image;

[0023] The operator recognition unit is used to obtain the operation habit data of the operator, identify and classify different operators by using a fusion neural network model, identify the operation characteristics of the operator, match the corresponding control mode in the entered information database, and update the matched control parameters;

[0024] The anti-delay compensation unit is used to analyze the network delay in the communication process, perform quantitative analysis through data modeling, automatically predict and correct the operation result when the delay occurs, superimpose the delay compensation value when outputting the control signal, and dynamically adjust the operation behavior of the robot;

[0025] The risk warning unit is used to detect the position of the robot end and the lesion area in the three-dimensional stereoscopic image in real time, calculate the distance between the robot end and blood vessels, tissues and the lesion area, and automatically trigger a risk warning reminder when the distance is within the risk range;

[0026] The decision output unit is used to monitor the operation process of the operator performing the two-handed operation in real time, combine the dynamic kinematic model with real-time calculation, detect and optimize the accuracy of the operator in the operation process in real time, and output an adaptively optimized control signal to the local execution end.

[0027] Further, analyzing the network delay in the communication process, performing quantitative analysis through data modeling, automatically predicting and correcting the operation result when the delay occurs, and superimposing the delay compensation value when outputting the control signal, and dynamically adjusting the operation behavior of the robot includes:

[0028] Real-time monitor the delay between the remote control end and the local execution end through a network protocol tool, obtain the real-time delay value, calculate the mean and variance of the delay by using a sliding window, and determine whether the delay is abnormal;

[0029] Take the delay as a dynamic variable, use an autoregressive model to predict the change of the delay at a future moment, and trigger an event-driven mechanism when the predicted delay value exceeds the set threshold range;

[0030] Based on the robot kinematics and dynamics models, combined with historical control signals and feedback data, predict the motion trajectory of the robot end effector, obtain the motion state at a future moment, and calculate the delay compensation value caused by the delay according to the predicted motion state.

[0031] Further, monitoring the operation process of the operator performing the two-handed operation in real time, combining the dynamic kinematic model with real-time calculation, detecting and optimizing the accuracy of the operator in the operation process in real time, and outputting an adaptively optimized control signal to the local execution end includes:

[0032] Obtain the control instructions of the operator when controlling the multi-degree-of-freedom control device, use the control instructions to construct a real-time behavior model, and record the operation trajectory, action amplitude and speed change of the control device;

[0033] Based on the motion state information fed back by the local execution end, a forward kinematics model and an inverse kinematics model are constructed. The forward kinematics model is used to convert the manipulation instruction into the desired position and attitude of the robot end, and the inverse kinematics model is used to calculate the control signal of the robot end effector.

[0034] Obtain the actual motion state of the robot fed back by the local execution end, subtract the actual motion state from the desired position of the robot end effector to calculate the operation error, and perform real-time evaluation on the operation error to determine whether the error accuracy standard is met.

[0035] Based on the operation error calculated in real time, provide perceptual feedback to the operator through a force feedback device or a tactile display device, and adjust the manipulation instruction input by the operator according to the real-time feedback of the operation error, convert it into an optimized control signal, and finally superimpose the time delay compensation value onto the control signal and send it to the local execution end.

[0036] Furthermore, the compensation formula for the optimized control signal is:

[0037] u comp = u k + g(Δx predict ) = u k + g(y k+1 - x k )

[0038] y k+1 = f(x k , u k , τ k )

[0039] In the formula, u comp represents the optimized control signal; u k represents the originally input control signal; x k represents the current position of the robot end effector; y k+1 represents the desired position of the robot end effector; τ k represents the current time delay compensation value between the remote control end and the local execution end; Δx predict represents the compensation signal; g(·) represents the controller gain.

[0040] Furthermore, the local execution end includes: a robot motion control module, a force perception feedback module, a surgical image acquisition module, a motion state feedback module, a safety and fault tolerance module, and a data fusion and calculation module;

[0041] Among them, the robot motion control module is used to receive the control signal issued by the remote control end, control the motion trajectories of multiple groups of robotic arms in the robot, and perform surgical operations.

[0042] A force perception feedback module, which is used to install multi-dimensional tactile sensors at the end of the robot and on the robotic arm, capture the contact force between the robot and the tissue in real time, obtain force vector information, convert it into force feedback data, and synchronously transmit it to the remote control terminal;

[0043] A surgical image acquisition module, which is used to use an endoscope to acquire endoscopic images inside the brain in real time, and use an external camera to acquire external images of the robot during the surgical process, merge them into surgical images, and transmit them to the remote control terminal and the service cloud platform through a data channel;

[0044] A motion state feedback module, which is used to use motion sensors and encoders to monitor the motion states of the joints and tools of the robot in real time, and transmit the motion state information to the remote control terminal;

[0045] A safety and fault tolerance module, which is used to monitor the motion state, force feedback data and environmental conditions of the robot in real time. When there are abnormal phenomena, it immediately triggers the safety explosion protection program;

[0046] A data fusion and calculation module, which is used to integrate the monitoring data of multiple types of sensors, perform real-time fusion and calculation, display the comprehensive surgical process, and analyze and evaluate the surgical progress and completion degree.

[0047] Furthermore, the robot motion control module includes: a transmission conversion unit, a dual-loop control unit, a dynamic adjustment and optimization unit, a multi-modal fusion unit, and an end effector unit;

[0048] Among them, the transmission conversion unit is used to receive the control signal sent by the remote control terminal, and through protocol conversion processing, generate a data signal that conforms to the robot reading protocol;

[0049] The dual-loop control unit is used to connect a cascade proportional-integral-derivative controller to the robot to form a dual-loop control, input the control signal after protocol conversion, use the outer-loop controller as the main loop, and the inner-loop controller as the secondary loop to control the motion of each robotic arm of the robot;

[0050] The dynamic adjustment and optimization unit is used to monitor the position information during the motion of the robot, and perform real-time dynamic adjustment on the trajectory and operation behavior during the motion of the robot;

[0051] The multi-modal fusion unit is used to fuse the visual signal and tactile signal when the robot is moving, combine the operation error and the time delay compensation value, and maintain the dynamic coordination between the end effector of the robot and the environment;

[0052] The end effector unit is used to drive the end effector of the robot to perform brain surgery operations.

[0053] Furthermore, the control expression of the outer-loop controller is:

[0054]

[0055] In the formula, θ i represents the output of the i-th robotic arm joint angle of the robot, where i = 1, 2, …, N; N represents the number of joints of the robotic arm of the robot; θ ki represents the expected output of each joint angle of the robotic arm; K θp represents the proportionality coefficient in the angle control; K θi represents the integral time coefficient in the angle control; K θd represents the differential time coefficient in the angle control;

[0056] The control expression of the inner loop controller is:

[0057]

[0058] In the formula, ω i represents the angular velocity of each joint of the i-th robotic arm of the robot, where i = 1, 2, …, N; N represents the number of joints of the robotic arm of the robot; ω ki represents the expected operating angular velocity of each joint angle of the robotic arm; K ωp represents the proportionality coefficient in the angular velocity control; K ωi represents the integral time coefficient in the angular velocity control; K ωd represents the differential time coefficient in the angular velocity control.

[0059] Furthermore, monitoring the position information during the movement of the robot and performing real-time dynamic adjustment on the trajectory and operation behavior during the movement of the robot includes:

[0060] Real-time detecting the actual motion state of the end effector of the robot through the tactile sensor and the motion sensor, subtracting the actual motion state from the expected position of the end effector of the robot to obtain the operation error, and using model predictive control to plan the control signals for several time steps, introducing real-time constraints on the trajectory planning. When the operation error exceeds the set error threshold, dynamically adjusting the control gain.

[0061] The beneficial effects of the present invention are:

[0062] 1. Through the precise coordination between the remote control terminal and the local execution terminal, robot motion control, real-time viewing of surgical images, and force feedback interaction are realized. Meanwhile, combined with the consultation monitoring terminal and the screen sharing terminal, it supports multi-party collaborative decision-making and real-time screen sharing. With the help of 5G communication base stations and service cloud platforms, the system can achieve efficient and low-latency data transmission and storage, ensuring the security and real-time nature of surgical data. Compared with traditional technologies, the present invention has significantly improved in terms of remote control accuracy, multi-modal data fusion ability, collaborative consultation support, and communication stability, and is particularly suitable for complex and high-risk remote intelligent surgical scenarios, which helps improve surgical efficiency and safety.

[0063] 2. The remote control terminal can construct a three-dimensional stereoscopic image of the lesion area in real time, optimize the image quality and enhance the visibility of key structures, effectively simulate the force perception during the surgical process, and provide intuitive tactile feedback. It can precisely control multi-degree-of-freedom manipulation devices, generate precise manipulation instructions, and combine artificial intelligence and dynamic kinematic models to optimize the operator's operation habits and reduce the impact of communication latency, thereby improving the reliability of control signals and the safety of surgery, and ensuring real-time safety protection during the surgical process.

[0064] 3. Through a highly integrated design, the local execution terminal comprehensively realizes the efficiency, stability, and safety of the surgical execution process. It can control multiple groups of robotic arms to precisely execute the operation instructions issued by the remote control terminal, meeting the requirements of complex surgeries. By obtaining force vector information in real time, it provides intuitive force feedback support for the remote end. By integrating multi-sensor information, it can display the surgical process and completion degree in real time, providing strong support for surgical decision-making. The local execution terminal effectively realizes the precise execution, real-time feedback, and safety protection of robot operations, improving the execution efficiency and reliability of remote surgeries. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings described herein are used to provide a further understanding of the present invention and form a part of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0066] Figure 1 is a system principle block diagram of an intelligent surgical robot control system based on multi-modal fusion control according to an embodiment of the present invention.

[0067] Reference numerals in the drawings: 1. Remote control terminal; 2. Local execution terminal; 3. Consultation monitoring terminal; 4. Screen sharing terminal; 5. Service cloud platform; 6. Communication base station. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0068] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0069] Please refer to Figure 1 , there is provided an intelligent surgical robot control system based on multi-modal fusion control, characterized in that the intelligent surgical robot control system includes: a remote control terminal 1, a local execution terminal 2, a consultation and monitoring terminal 3, a screen sharing terminal 4, a service cloud platform 5 and a communication base station 6.

[0070] The remote control terminal 1 is used to input and output according to multi-modal data, send control instructions through the control interface, convert them into robot control signals, so as to complete robot motion control, real-time viewing of surgical images, reception and transmission of force feedback data, and perform precise robot operations.

[0071] In the description of the present invention, the remote control terminal 1 includes: a visual image processing module (not shown in the figure), a force feedback display and perception module (not shown in the figure), a two-handed joystick operation control module (not shown in the figure), a decision support and assistance module (not shown in the figure) and a monitoring and security guarantee module (not shown in the figure).

[0072] The visual image processing module is used to build a three-dimensional stereoscopic image of the brain surgery area according to the endoscopic image in the received surgical image, combined with three-dimensional reconstruction technology, and optimize the image quality through real-time image processing to enhance the visibility of the lesion area and key structures.

[0073] Specifically, the visual image processing module receives the endoscopic image transmitted in real time by the local execution terminal 2, and combines three-dimensional reconstruction technology to intuitively present the brain surgery area in the form of a three-dimensional stereoscopic image, facilitating the operator to comprehensively understand the spatial structure and lesion distribution of the surgery area.

[0074] This module uses real-time image processing technology to automatically enhance the key areas in the endoscopic image, optimize the image quality, and improve the visibility of the lesion area and key tissues (such as blood vessels and nerves). In addition, this module combines denoising and enhancement algorithms to maintain the clarity and stability of the image in complex environments such as low light or blurred images, thereby providing accurate visual information support for the remote control terminal.

[0075] The force feedback display and perception module is used to integrate tactile sensors, convert the force feedback data synchronized during the robot movement into tactile signals, and simulate the force perception during the surgery process.

[0076] Specifically, the force feedback display and perception module collects real-time force feedback data when the end of the robot contacts human tissues during operation through a multi-dimensional tactile sensor integrated and installed at the local execution end, converts it into tactile signals, and transmits them to the remote control end.

[0077] This module uses force perception data to accurately simulate the operator's tactile perception of tissues during surgery, enabling the remote operator to real-time perceive the force condition of the surgical tool and tissue characteristics, and avoiding operation errors caused by the lack of force sense information. The force feedback display and perception module combines fine force vector calculation and tactile signal encoding technology to provide the operator with a realistic force sense feedback experience, improving the accuracy and safety of surgical operations.

[0078] The two-handed operation control module is used to generate control commands for the operator to perform remote surgical operations by controlling a multi-degree-of-freedom control device according to the surgical images, force feedback data, and robot motion state information fed back by the local execution end 2.

[0079] Specifically, the two-handed operation control module combines a multi-degree-of-freedom mechanical control device to provide the operator with a flexible and intuitive operation method.

[0080] By receiving the surgical images, force feedback data, and robot motion state information fed back by the local execution end, the operator inputs operation commands by controlling the hand-crank device. The module instantly converts the commands into motion control signals for the end of the robot, driving the robot to perform corresponding actions. The module is embedded with a dynamic kinematic model to calculate the operation path of the operator and the accuracy of the robot motion in real time, and dynamically adjusts the commands through an intelligent optimization algorithm to ensure that the robot has high-precision motion performance in complex operation scenarios, thus meeting the operation requirements of high-risk brain surgery.

[0081] The decision support and assistance module is used to mark the lesion areas, blood vessels, and tissues in the three-dimensional stereoscopic image in real time, identify the operator's operation habits based on artificial intelligence, combine communication delay analysis to adaptively optimize the control commands, and combine the dynamic kinematic model and real-time calculation to detect the accuracy of the operator during the operation, and output optimized control signals.

[0082] In the description of the present invention, the decision support and assistance module includes: an image marking unit (not shown in the figure), an operator identification unit (not shown in the figure), an anti-delay compensation unit (not shown in the figure), a risk warning unit (not shown in the figure), and a decision output unit (not shown in the figure).

[0083] Among them, the image marking unit is used to obtain the endoscope images received in real time, perform semantic segmentation and target detection on the endoscope images using deep learning algorithms, extract the information of lesions, blood vessels, and tissues in the surgical area, and mark them in the constructed three-dimensional stereoscopic image.

[0084] An operator identification unit, which is used to obtain the action habit data of the operator, identify and classify different operators by using a fusion neural network model, identify the action characteristics of the operator, match the corresponding control mode in the already entered information database, and update the matching control parameters.

[0085] An anti-delay compensation unit, which is used to analyze the network delay in the communication process, perform quantitative analysis through data modeling, automatically predict and correct the operation result when the delay occurs, superimpose the delay compensation value when outputting the control signal, and dynamically adjust the operation behavior of the robot.

[0086] In the description of the present invention, analyzing the network delay in the communication process, performing quantitative analysis through data modeling, automatically predicting and correcting the operation result when the delay occurs, and superimposing the delay compensation value when outputting the control signal and dynamically adjusting the operation behavior of the robot includes:

[0087] Step S101: Real-time monitor the delay between the remote control end 1 and the local execution end 2 through a network protocol tool, obtain the real-time delay value, calculate the mean and variance of the delay by using a sliding window, and determine whether the delay is abnormal.

[0088] Specifically, through a network protocol tool (such as Ping, UDP / TCP delay measurement tool or customized delay monitoring module), real-time monitor the communication delay between the remote control end and the local execution end, and obtain the current real-time delay value. By using the sliding window technology, calculate the mean and variance of the delay in the recent period of time, obtain the dynamic change trend of the delay, and avoid misjudgment caused by instantaneous jitter.

[0089] Finally, according to the set normal delay threshold range (such as set based on historical statistical data or system requirements), determine whether the delay value is abnormal. When the mean or variance of the delay exceeds the preset range, it is considered that there is a delay abnormality, providing a basis for subsequent compensation and adjustment.

[0090] Step S102: Take the delay as a dynamic variable, use an autoregressive model to predict the change of the delay at a future moment, and trigger an event-driven mechanism when the predicted delay value exceeds the set threshold range.

[0091] Specifically, input the real-time collected delay value into an autoregressive prediction model (such as AR, ARIMA or LSTM), and predict the change trend of the delay in the future period of time according to the historical sequence data of the delay. The autoregressive model can accurately predict the delay value at the next moment or in a short period of time by capturing the time dependence of the delay change.

[0092] When the predicted delay value exceeds the preset tolerance range (for example, when the delay exceeds a certain fixed value or the network jitter is too large), the event-driven mechanism is triggered, and the delay compensation and dynamic adjustment process is entered to ensure that the trigger mechanism can respond quickly in case of sudden anomalies and avoid affecting the operation accuracy.

[0093] Step S103: Based on the robot kinematics and dynamics models, combined with historical control signals and feedback data, predict the motion trajectory of the robot end effector, obtain the motion state at a future moment, and calculate the delay compensation value caused by the delay according to the predicted motion state.

[0094] Specifically, combined with the robot kinematics and dynamics models of the robot end effector, input the historical control signals and feedback data to predict the motion trajectory and state (such as position, velocity, and acceleration) of the robot end effector at a future moment. Use the kinematics model to describe the relationship between each joint of the robot and the end position to ensure a high degree of matching between the motion prediction and the actual state. Combined with the dynamics model, consider the influence of force and torque on the robot motion to improve the accuracy of trajectory prediction.

[0095] The risk warning unit is used to detect the position of the robot end and the lesion area in the three-dimensional stereoscopic image in real time, calculate the distances between the robot end and blood vessels, tissues, and the lesion area, and automatically trigger a risk warning reminder when the distance is within the risk range.

[0096] Specifically, the risk warning unit is based on the three-dimensional stereoscopic image and real-time analysis algorithm to continuously detect and calculate the distances of the position of the robot end and the key structures (such as lesions, blood vessels, nerves, etc.) in the surgical area during the operation. By monitoring the spatial relationship between the robot end and the surrounding tissues, when the robot end approaches the predetermined risk range, the risk warning unit can quickly issue an acoustic, optical, or image prompt to remind the operator that the operation is about to enter a dangerous area.

[0097] In addition, the module is embedded with an artificial intelligence analysis algorithm to dynamically adjust the threshold of the risk range in combination with real-time data to ensure the accuracy and real-time nature of risk assessment in different surgical scenarios, and further improve the safety and fault tolerance of the operation.

[0098] The decision output unit is used to monitor the operation process of the operator performing the double-handed cranking operation in real time, combine the dynamic kinematics model and real-time calculation to detect and optimize the accuracy of the operator in the operation process in real time, and output an adaptively optimized control signal to the local execution end 2.

[0099] In the description of the present invention, monitoring the operation process of the operator performing the double-handed cranking operation in real time, combining the dynamic kinematics model and real-time calculation to detect and optimize the accuracy of the operator in the operation process in real time, and outputting an adaptively optimized control signal to the local execution end 2 includes:

[0100] Step S111: Obtain the manipulation instructions of the operator when controlling the multi-degree-of-freedom manipulation device, construct a real-time behavior model using the manipulation instructions, and record the operation trajectory, movement amplitude, and speed change of the manipulation device.

[0101] Specifically, obtain the manipulation instruction data of the operator when controlling the multi-degree-of-freedom manipulation device (such as a two-handed joystick or a force feedback device), including the operation trajectory, movement amplitude, and speed change. Using these data, construct a real-time behavior model of the operator, and dynamically record the change rules and motion characteristics of the manipulation device to reflect the operation behavior characteristics of the operator.

[0102] In a remote surgery scenario, the operator controls the surgical robot through a two-handed joystick. The left joystick is used for the forward and backward movement of the robotic arm at the end, and the right joystick is used for the rotation and height adjustment of the robotic arm. The system records the movement amplitude and speed change of the joystick in real time, generates manipulation trajectory data (such as a sequence of trajectory points: (x 1 , y 1 , z 1 ), (x 2 , y 2 , z 2 ) and establishes a behavior model for tracking the operation dynamics.

[0103] Step S112: Based on the motion state information fed back by the local execution end 2, construct a forward kinematics model and an inverse kinematics model. Use the forward kinematics model to convert the manipulation instructions into the desired position and orientation of the robot end, and use the inverse kinematics model to calculate the control signal of the robot end effector.

[0104] Specifically, according to the manipulation instructions and the motion state information fed back by the local execution end, respectively construct a forward kinematics model and an inverse kinematics model of the robot end effector.

[0105] The forward kinematics model converts the manipulation instructions into the desired position and orientation of the robot end (such as coordinate position and angle), while the inverse kinematics model calculates the control signals of each joint of the robot according to the desired position and orientation to achieve precise end operation.

[0106] Step S113: Obtain the actual motion state of the robot fed back by the local execution end 2, subtract the actual motion state from the desired position of the robot end effector, calculate the operation error, and perform a real-time evaluation of the operation error to determine whether it meets the error accuracy standard.

[0107] Step S114: Based on the operation error calculated in real time, provide perceptual feedback to the operator through a force feedback device or a tactile display device, and adjust the manipulation instructions input by the operator according to the real-time feedback of the operation error, convert them into optimized control signals, and finally superimpose the time delay compensation value on the control signals and send them to the local execution end 2.

[0108] In the description of the present invention, the compensation formula for the optimized control signal is as follows:

[0109] u comp = u k + g(Δx predict ) = u k + g(y k+1 - x k )

[0110] y k+1 = f(x k , u k , τ k )

[0111] In the formula, u comp represents the optimized control signal; u k represents the control signal of the original input; x k represents the current position of the end effector of the robot; y k+1 represents the desired position of the end effector of the robot; τ k represents the current time delay compensation value between the remote control end 1 and the local execution end 2; Δx predict represents the compensation signal; g(·) represents the controller gain.

[0112] The monitoring and safety guarantee module is used to monitor the robot state, the surgical progress, and the physiological information of the patient during the operation, and perform real-time analysis on the operation behavior of the control device.

[0113] Specifically, the main function of the monitoring and safety guarantee module is to comprehensively monitor multiple key dimensions involved in the operation process, including the robot state, the surgical progress, and the physiological information of the patient. At the same time, it performs real-time analysis on the operation behavior of the control device to ensure the safety and efficiency of the operation process. Once an abnormality or potential risk is detected, the module will trigger an early warning mechanism and pause or adjust the operation as needed to avoid uncontrollable risks during the operation.

[0114] For the robot state monitoring, the motion state data of each part of the robot (such as the robotic arm, the end effector) are collected in real time, including parameters such as position, speed, joint angle, and motor torque, and it is analyzed whether there are any abnormalities, such as joint jamming, overload, and exceeding the motion range. If an abnormal state is detected, the module will issue an alarm and prompt the operator to check the device, and trigger the safety stop mechanism if necessary.

[0115] For surgical progress tracking, according to the surgical plan and the operation path formulated before the operation, the real-time progress of the operation is monitored. By comparing the current operation step with the expected step, it is judged whether the operation is carried out as planned, whether there are missing steps or operation delays. If abnormal progress is found, the system will automatically prompt the operator or adjust the operation plan.

[0116] For patient physiological information monitoring, it is necessary to connect with the patient monitoring device to collect the patient's key physiological parameters (such as heart rate, blood pressure, blood oxygen saturation, etc.) in real time. Once there are abnormal fluctuations in the patient's physiological indicators (such as too high blood pressure or too low heart rate), the module will issue a warning to the operator and recommend pausing or adjusting the surgical operation.

[0117] For the analysis of the behavior of the control device, it is necessary to perform real-time analysis on the operation instructions input by the operator through the remote control terminal, including the input signal strength, stability of the control device, and the matching degree between the instruction and the actual execution result. If abnormal operations are detected (such as the control instruction deviating from the normal trajectory, the force feedback increasing abnormally, etc.), the module will provide feedback reminders to the operator's behavior and optimize the control signal through real-time calculation to reduce the risk of misoperation.

[0118] The local execution end 2 is used to receive the control signal sent by the remote control terminal, drive the robot to perform corresponding operations, and collect the surgical images, force feedback data and robot motion state information during the operation in real time, and synchronously transmit them to the remote control terminal and the consultation monitoring end.

[0119] In the description of the present invention, the local execution end 2 includes: a robot motion control module (not shown in the figure), a force perception feedback module (not shown in the figure), a surgical image acquisition module (not shown in the figure), a motion state feedback module (not shown in the figure), a safety fault tolerance module (not shown in the figure), and a data fusion calculation module (not shown in the figure).

[0120] The robot motion control module is used to receive the control signal issued by the remote control terminal 1, control the motion trajectory of multiple groups of robotic arms in the robot, and perform surgical operations.

[0121] In the description of the present invention, the robot motion control module includes: a transmission conversion unit (not shown in the figure), a dual-loop control unit (not shown in the figure), a dynamic adjustment and optimization unit (not shown in the figure), a multi-modal fusion unit (not shown in the figure), and an end effector unit (not shown in the figure).

[0122] The transmission conversion unit is used to receive the control signal sent by the remote control terminal 1 and generate a data signal that conforms to the robot reading protocol through protocol conversion processing.

[0123] Specifically, it receives manipulation instruction signals (such as position signals, attitude signals or force control signals) remotely and in real time. It converts the control signals from the standard protocols (such as TCP / IP, UDP, etc.) of the remote control end into the underlying protocols (such as CAN bus, serial port protocol, etc.) required by the robot hardware, and processes data formats (such as encoding, decoding, segmentation, etc.). It ensures the synchronization of various manipulation signals (such as position, speed, force feedback) in time and space, uses a verification mechanism to detect and repair possible signal errors, and finally generates data packets that conform to the robot's reading protocol, including position signals, motion trajectory instructions and control parameters, etc., and finally sends them to the multi-modal fusion unit.

[0124] For example, in a remote brain surgery, the operator inputs a robotic arm movement instruction (for example, move to the target coordinate position: X = 10.5 mm, Y = 12.3 mm, Z = 15.7 mm). The transmission and conversion unit analyzes and converts the instruction data transmitted from the control end through the TCP / IP protocol into the CAN bus protocol format inside the robot, and at the same time verifies the integrity and timeliness of the data packet, and then sends the standardized coordinate signal to the multi-modal fusion unit.

[0125] The dual-loop control unit is used to connect a cascade proportional-integral-derivative controller to the robot to form a dual-loop control. It inputs the control signal after protocol conversion, uses the outer-loop controller as the main loop and the inner-loop controller as the secondary loop to control the movement of each robotic arm of the robot.

[0126] In the description of the present invention, the control expression of the outer-loop controller is:

[0127]

[0128] In the formula, θ i represents the output of the i-th robotic arm joint angle of the robot, i = 1, 2,..., N; N represents the number of joints of the robotic arm of the robot; θ ki represents the expected output of each joint angle of the robotic arm; K θp represents the proportional coefficient in angle control; K θi represents the integral time coefficient in angle control; K θd represents the differential time coefficient in angle control;

[0129] The control expression of the inner-loop controller is:

[0130]

[0131] In the formula, ω i represents the angular velocity of each joint of the i-th robotic arm of the robot, i = 1, 2,..., N; N represents the number of joints of the robotic arm of the robot; ω ki represents the expected operating angular velocity of each joint angle of the robotic arm; Kωp represents the proportional coefficient in angular velocity control; K ωi represents the integral time coefficient in angular velocity control; K ωd represents the derivative time coefficient in angular velocity control.

[0132] A dynamic adjustment and optimization unit for monitoring the position information during the movement of the robot, and performing real-time dynamic adjustment on the trajectory and operation behavior during the movement of the robot.

[0133] In the description of the present invention, monitoring the position information during the movement of the robot and performing real-time dynamic adjustment on the trajectory and operation behavior during the movement of the robot includes:

[0134] The actual motion state of the robot end effector is detected in real time through a tactile sensor and a motion sensor. The desired position of the robot end effector is subtracted from the actual motion state to obtain an operation error, and a control signal for several time steps is planned using model predictive control (MPC) to introduce real-time constraints into the trajectory planning. When the operation error exceeds the set error threshold, the control gain is dynamically adjusted.

[0135] Specifically, the force feedback data when the end effector contacts the environment is obtained in real time through a tactile sensor, and at the same time, the actual motion state (including position, attitude, and speed) of the end effector is detected through a motion sensor (such as an accelerometer, a gyroscope, an encoder, etc.) to ensure obtaining the interaction information between the end effector and the current environment and providing real-time data support for trajectory adjustment and error compensation.

[0136] By establishing a kinematic and dynamic model of the robot, the motion trajectory for several future time steps is predicted to generate an optimized control signal sequence to ensure the continuity and accuracy of the motion trajectory of the end effector. Real-time constraint conditions, such as the physical limitations (speed, acceleration) of the actuator and the safe distance from the environment, are introduced into the trajectory planning to ensure that the trajectory planning result meets the actual operation requirements. The trajectory and operation behavior of the robot end can be planned in advance to optimize the control signal and avoid trajectory deviation caused by error accumulation.

[0137] For example, during a brain surgery by the robot, the tactile sensor detects the force value when the end effector contacts the tissue, and the motion sensor records a slight deviation of the actuator. These data are transmitted to the control system in real time. When it is predicted that the actuator is about to approach the fragile blood vessel wall, the MPC planning will adjust the motion trajectory to decelerate the end effector and deviate from the sensitive area.

[0138] A multi-modal fusion unit for fusing the visual signal and tactile signal when the robot moves, and combining the operation error and the delay compensation value to maintain the dynamic coordination between the robot end effector and the environment.

[0139] Specifically, the multi-modal fusion unit fuses multi-modal signals (such as visual signals and tactile signals) during the movement of the robot, combines the operation error and the time-delay compensation value, realizes the dynamic coordination between the robot end effector and the environment, and ensures the stability and accuracy of the surgical operation.

[0140] For example, during the operation, when the end sensor of the robot detects that the contact force with the brain tissue is too large (such as exceeding the set threshold of 0.8 N), the multi-modal fusion unit timely adjusts the movement trajectory through the tactile signal, reduces the movement speed of the end, and at the same time analyzes the movement limit range of the key area according to the visual signal to ensure the safety of the actions of the end effector. In addition, when the network time delay is 30 ms, the unit will optimize the movement trajectory command in advance according to the time-delay compensation mechanism to avoid errors caused by operation delay.

[0141] The end effector unit is used to drive the robot end effector to perform brain surgery operations.

[0142] The force perception feedback module is used to install multi-dimensional tactile sensors on the robot end and the robotic arm, capture the contact force between the robot and the tissue in real time, obtain the force vector information, and convert it into force feedback data, which is synchronously transmitted to the remote control terminal 1.

[0143] The surgical image acquisition module is used to use an endoscope to collect the endoscope image inside the brain in real time, and use an external camera to collect the external image of the robot surgical process, combine them to form a surgical image, and transmit it to the remote control terminal 1 and the service cloud platform 5 through the data channel.

[0144] The motion state feedback module is used to use motion sensors and encoders to monitor the motion states of the robot joints and tools in real time, and transmit the motion state information to the remote control terminal 1.

[0145] The safety and fault tolerance module is used to monitor the robot motion state, force feedback data and environmental conditions in real time. When there are abnormal phenomena, it immediately triggers the safety explosion protection program.

[0146] The data fusion and calculation module is used to integrate the monitoring data of multiple types of sensors, perform real-time fusion and calculation, display the comprehensive surgical process, and analyze and evaluate the surgical progress and completion degree.

[0147] The consultation monitoring terminal 3 is used to obtain the surgical images transmitted by the local execution terminal, conduct real-time consultation through video stream and data sharing, provide collaborative decision-making support, and establish a communication channel with the service cloud platform through the communication base station to build a data sharing network.

[0148] Specifically, the consultation monitoring terminal 3 is mainly used to receive real-time surgical images transmitted by the local execution terminal. Through video streaming and data sharing technologies, it supports remote experts to conduct real-time consultations and provide collaborative decision-making suggestions. At the same time, it establishes a communication channel with the service cloud platform through the communication base station and participates in the construction of the surgical data sharing network.

[0149] The consultation monitoring terminal 3 receives high-resolution endoscopic surgical images transmitted by the local execution terminal in real time and displays them to remote consultation experts through video streaming technology to achieve real-time observation. Based on the surgical images and data, it supports consultation experts to discuss surgical plans and make collaborative decisions, and feedback suggestions to the remote control terminal through the network to optimize the surgical process. It establishes a two-way communication channel with the communication base station and the service cloud platform, participates in the surgical data sharing network, and realizes the efficient flow, storage, and backup of data.

[0150] The screen sharing terminal 4 is used to establish a connection with the service cloud platform through the communication base station, serve as a node of the data sharing network, and access the consultation and surgical video screens in real time.

[0151] Specifically, as a node in the data sharing network, the screen sharing terminal 4 is mainly used to establish a connection with the service cloud platform 5 through the communication base station 6, access the consultation screen and surgical video in real time, and provide the surgical screen sharing function for other terminal devices or authorized users.

[0152] For example, in surgical teaching, the screen sharing terminal connects to the service cloud platform, shares the surgical video stream to multiple student devices. Students can watch the surgical screen in real time and learn the key operation processes. At the same time, through the permission settings of the screen sharing terminal, it ensures that surgical data is only transmitted within the authorized scope.

[0153] The service cloud platform 5 is used to build a data sharing network among the local execution terminal, the consultation monitoring terminal, and the screen sharing terminal, and provide data backup, storage, and processing services.

[0154] Specifically, the service cloud platform 5 is the data processing and storage core of the surgical system. Its main task is to build a data sharing network among the local execution terminal, the consultation monitoring terminal, and the screen sharing terminal, and at the same time provide data backup, storage, processing, and analysis functions.

[0155] Through a centralized cloud architecture, it connects the local execution terminal, the consultation monitoring terminal, and the screen sharing terminal to realize real-time data sharing and collaboration among multiple devices, perform real-time backup of key data (such as video streams, operation records, etc.) during the surgical process, provide long-term storage services, and support access at any time. It uses high-performance computing resources to process the massive data generated during the surgical process (such as surgical images, force feedback data, latency compensation values, etc.) to provide support for subsequent optimization and research.

[0156] Communication base station 6, which is used for 5G network communication, provides data transmission, low-latency communication and high-bandwidth transmission, and uses network slicing technology to process various data streams during the operation process.

[0157] Specifically, the communication base station 6 uses 5G network technology to provide high-bandwidth and low-latency data transmission services for the surgical system, ensuring smooth data flow between the remote control end, the consultation and monitoring end, the local execution end and the service cloud platform. At the same time, different types of data streams (such as video streams and control signals) are distinguished and optimized through network slicing technology.

[0158] In summary, by means of the above technical solutions of the present invention, through the precise cooperation between the remote control end and the local execution end, robot motion control, real-time viewing of surgical images and force feedback interaction are realized. At the same time, combined with the consultation and monitoring end and the screen sharing end, multi-party collaborative decision-making and real-time screen sharing are supported; with the help of 5G communication base stations and service cloud platforms, the system can achieve efficient and low-latency data transmission and storage, ensuring the security and real-time nature of surgical data; compared with traditional technologies, the present invention has significant improvements in remote control accuracy, multi-modal data fusion ability, collaborative consultation support and communication stability, and is particularly suitable for complex and high-risk remote intelligent surgical scenarios, which helps to improve surgical efficiency and safety. The remote control end can construct a three-dimensional stereoscopic image of the lesion area in real time, optimize the image quality and enhance the visibility of key structures, effectively simulate the force perception during the operation process, and provide intuitive tactile feedback; it can accurately control multi-degree-of-freedom manipulation devices, generate precise manipulation instructions, and combine artificial intelligence and dynamic kinematic models to optimize the operator's operation habits and reduce the impact of communication delay, thereby improving the reliability of control signals and the safety of the operation, ensuring real-time safety guarantee during the operation process. The local execution end, through a highly integrated design, comprehensively realizes the efficiency, stability and safety of the surgical execution process, can control multiple groups of robotic arms to accurately execute the operation instructions issued by the remote control end, meeting the requirements of complex surgeries; by obtaining force vector information in real time, it provides intuitive force feedback support for the remote end, and through integrating multi-sensor information, it real-time displays the surgical process and completion degree, providing strong support for surgical decision-making. The local execution end effectively realizes the precise execution, real-time feedback and safety guarantee of robot operations, improving the execution efficiency and reliability of remote surgeries.

[0159] It should be understood that although the steps in the flowchart of the accompanying drawings are shown sequentially in the direction of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise clearly stated in this document, there is no strict order restriction for the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

Claims

1. An intelligent surgical robot control system based on multimodal fusion control, characterized in that: The intelligent surgical robot control system includes: a remote control terminal, a local execution terminal, a consultation monitoring terminal, a screen sharing terminal, a service cloud platform and a communication base station; The remote control terminal is used to send control instructions based on the multimodal data input and output, and convert them into robot control signals to complete robot motion control, real-time viewing of surgical images, reception and transmission of force feedback data, and perform precise robot operations; The local execution end is used to receive the control signal sent by the remote control end, drive the robot to perform the corresponding operation, and collect the surgical images, force feedback data and robot motion state information during the operation in real time, and synchronously transmit them to the remote control end and the consultation monitoring end; The consultation monitoring terminal is used to obtain the surgical images transmitted by the local execution terminal, conduct real-time consultation through video streaming and data sharing, provide collaborative decision support, and establish a communication channel with the service cloud platform through the communication base station to build a data sharing network; The screen sharing terminal is used to establish a connection with the service cloud platform through the communication base station, serve as a node of the data sharing network, and access the consultation and surgery video screens in real time; The service cloud platform is used to build a data sharing network between the local execution terminal, the consultation monitoring terminal and the screen sharing terminal, and provide data backup, storage and processing services; The communication base station is used to provide data transmission, low-latency communication and high-bandwidth transmission based on 5G network communication, and uses network slicing technology to process various data streams during the operation.

2. According to claim 1, the intelligent surgical robot control system based on multimodal fusion control is characterized in that: The remote control terminal includes: a visual image processing module, a force feedback display perception module, a double-handed cranking operation control module, a decision support auxiliary module and a monitoring safety assurance module; The visualization image processing module is used to construct a three-dimensional stereoscopic image of the brain surgical area based on the endoscopic image in the received surgical image in combination with the three-dimensional reconstruction technology, and optimize the image quality through real-time image processing to enhance the visibility of the lesion area and key structures; The force feedback display perception module is used to integrate a tactile sensor to convert the synchronized force feedback data during the robot movement into a tactile signal to simulate the force perception during the surgery; The two-handed cranking control module is used to enable the operator to control the multi-degree-of-freedom control device to perform remote surgery and generate control instructions based on the surgical image, force feedback data and robot motion state information fed back by the local execution end; The decision support auxiliary module is used to mark the lesion area, blood vessels and tissues in the three-dimensional image in real time, recognize the operator's operating habits based on artificial intelligence, combine communication delay analysis, adaptively optimize the control instructions, and combine dynamic kinematics model with real-time calculation to detect the operator's accuracy during the operation and output the optimized control signal; The monitoring safety assurance module is used to monitor the robot status, surgical progress and patient's physiological information during the operation, and to perform real-time analysis on the operation behavior of the control device.

3. The intelligent surgical robot control system based on multimodal fusion control according to claim 2 is characterized in that: The decision support auxiliary module includes: an image marking unit, an operator identification unit, an anti-delay compensation unit, a risk warning unit and a decision output unit; The image marking unit is used to obtain the endoscopic image received in real time, perform semantic segmentation and target detection on the endoscopic image using a deep learning algorithm, extract information on lesions, blood vessels and tissues in the surgical area, and mark them in the constructed three-dimensional image; The operator identification unit is used to obtain the operator's action habit data, identify and classify different operators using a fusion neural network model, identify the operator's action characteristics, match the corresponding control mode in the entered information library, and update the matching control parameters; The anti-delay compensation unit is used to analyze the network delay in the communication process, perform quantitative analysis through data modeling, automatically predict and correct the operation results when the delay occurs, superimpose the delay compensation value when outputting the control signal, and dynamically adjust the robot operation behavior; The risk warning unit is used to detect the position of the robot end and the lesion area in the three-dimensional image in real time, calculate the distance between the robot end and the blood vessel, tissue and lesion area, and automatically trigger a risk warning reminder when the distance is within the risk range; The decision output unit is used to monitor the operation process of the operator performing the two-hand shaking operation in real time, combine the dynamic kinematics model with real-time calculation, detect and optimize the operator's accuracy during the operation in real time, and output the adaptively optimized control signal to the local execution end.

4. The intelligent surgical robot control system based on multimodal fusion control according to claim 3 is characterized in that: The network delay in the communication process is analyzed, quantitative analysis is performed through data modeling, the operation result is automatically predicted and corrected when the delay occurs, the delay compensation value is superimposed when the control signal is output, and the robot operation behavior is dynamically adjusted, including: Monitor the delay between the remote control end and the local execution end in real time through a network protocol tool, obtain the real-time delay value, calculate the mean and variance of the delay using a sliding window, and determine whether there is an abnormality in the delay; Taking delay as a dynamic variable, the autoregressive model is used to predict the delay changes at future moments. When the predicted delay value exceeds the set threshold range, the event-driven mechanism is triggered. Based on the robot kinematics and dynamics model, combined with historical control signals and feedback data, the motion trajectory of the robot's end effector is predicted to obtain the motion state at the future moment. According to the predicted motion state, the delay compensation value caused by the delay is calculated.

5. The intelligent surgical robot control system based on multimodal fusion control according to claim 4 is characterized in that: The real-time monitoring of the operation process of the operator performing the two-hand cranking operation, combining the dynamic kinematics model with the real-time calculation, real-time detection and optimization of the accuracy of the operator during the operation, and outputting the adaptively optimized control signal to the local execution end include: Obtain the operator's control instructions when controlling the multi-degree-of-freedom control device, use the control instructions to build a real-time behavior model, and record the operation trajectory, movement amplitude and speed change of the control device; Based on the motion state information fed back by the local execution end, a forward kinematics model and an inverse kinematics model are constructed, the control command is converted into the desired position and posture of the robot end by using the forward kinematics model, and the control signal of the robot end effector is calculated by using the inverse kinematics model; Obtaining the actual motion state of the robot fed back by the local execution end, subtracting the actual motion state from the expected position of the robot end effector, calculating the operation error, and performing real-time evaluation on the operation error to determine whether the error accuracy standard is met; Based on the real-time calculated operation error, perceptual feedback is provided to the operator through a force feedback device or a tactile display device, and according to the real-time feedback of the operation error, the control instructions input by the operator are adjusted and converted into optimized control signals, which are finally combined with the delay compensation value and superimposed on the control signal and sent to the local execution end.

6. The intelligent surgical robot control system based on multimodal fusion control according to claim 5, characterized in that: The compensation formula of the optimized control signal is: u comp =u k +g(Δx predict )=u k +g(y k+1 -x k ) y k+1 =f(x k ,u k ,t k ) In the formula, u comp represents the optimized control signal; u k A control signal representing the original input; x k Indicates the current position of the robot's end effector; y k+1 represents the desired position of the robot end effector; τ k Indicates the current delay compensation value between the remote control end and the local execution end; Δx predict represents the compensation signal; g(·) represents the controller gain.

7. The intelligent surgical robot control system based on multimodal fusion control according to claim 2, characterized in that: The local execution end includes: a robot motion control module, a force sensing feedback module, a surgical image acquisition module, a motion state feedback module, a safety fault tolerance module and a data fusion calculation module; The robot motion control module is used to receive the control signal from the remote control terminal, control the motion trajectory of multiple sets of mechanical arms in the robot, and perform surgical operations; The force sensing feedback module is used to install multi-dimensional tactile sensors on the robot end and the mechanical arm to capture the contact force between the robot and the tissue in real time, obtain force vector information, convert it into force feedback data, and synchronously transmit it to the remote control end; The surgical image acquisition module is used to use an endoscope to collect endoscopic images in the brain in real time, and use an external camera to collect external images of the robotic surgery process, merge them into surgical images, and transmit them to the remote control terminal and the service cloud platform through a data channel; The motion state feedback module is used to monitor the motion state of each joint and tool of the robot in real time using motion sensors and encoders, and transmit the motion state information to the remote control end; The safety fault-tolerant module is used to monitor the robot's motion state, force feedback data and environmental conditions in real time, and immediately initiate a safety explosion program when an abnormal phenomenon occurs; The data fusion calculation module is used to integrate the monitoring data of multiple types of sensors, perform real-time fusion and calculation, display the comprehensive surgical progress, and analyze and evaluate the progress and completion of the surgery.

8. The intelligent surgical robot control system based on multimodal fusion control according to claim 7 is characterized in that: The robot motion control module includes: a transmission conversion unit, a dual-loop control unit, a dynamic adjustment optimization unit, a multi-modal fusion unit and an end execution unit; The transmission conversion unit is used to receive the control signal sent by the remote control terminal and generate a data signal that complies with the robot reading protocol through protocol conversion processing; The dual-loop control unit is used to connect a cascade proportional-integral-differential controller to the robot to form a dual-loop control, input the control signal after protocol conversion, use the outer loop controller as the main loop and the inner loop controller as the secondary loop to control the movement of each mechanical arm of the robot; The dynamic adjustment and optimization unit is used to monitor the position information of the robot during movement and to dynamically adjust the trajectory and operation behavior of the robot during movement in real time; The multimodal fusion unit is used to fuse the visual signal and tactile signal of the robot during movement, and to maintain the dynamic coordination between the robot end effector and the environment by combining the operation error and the delay compensation value; The end effector unit is used to drive the robot end effector to perform brain surgery.

9. The intelligent surgical robot control system based on multimodal fusion control according to claim 8, characterized in that: The control expression of the outer loop controller is: In the formula, θ i Represents the joint angle output of the robot's i-th manipulator, i = 1, 2, ..., N; N represents the number of joints of the robot arm; θ ki Represents the expected output of each joint angle of the robot; K θp Indicates the proportional coefficient in angle control; K θi Indicates the integral time coefficient in angle control; K θd Indicates the differential time coefficient in angle control; The control expression of the inner loop controller is: In the formula, ω i represents the angular velocity of each joint of the i-th manipulator of the robot, i = 1, 2, ..., N; N represents the number of joints of the robot arm; ω ki Indicates the expected angular velocity of each joint angle of the robot; K ωp Represents the proportional coefficient in angular velocity control; K ωi Represents the integral time coefficient in angular velocity control; K ωd Indicates the derivative time coefficient in angular velocity control.

10. The intelligent surgical robot control system based on multimodal fusion control according to claim 9, characterized in that: The monitoring of the position information of the robot during the movement and the real-time dynamic adjustment of the trajectory and operation behavior of the robot during the movement include: The actual motion state of the robot's end effector is detected in real time through tactile sensors and motion sensors. The operation error is obtained by subtracting the actual motion state from the expected position of the robot's end effector. The control signal of several time steps is planned using model predictive control, and real-time constraints are introduced into trajectory planning. When the operation error exceeds the set error threshold, the control gain is dynamically adjusted.

Images