Control method, system and equipment of throat intelligent surgical robot and medium

Abstract

The invention discloses a control method, system and equipment of an intelligent throat surgical robot and a medium. According to the scheme, a three-dimensional tissue model is constructed according to visual image data, laser imaging data and tomography image data; determining lesion data according to the historical lesion data and the tomography image data, superposing the lesion data and the three-dimensional tissue model, and determining a target operation model; determining prediction trajectory data according to the tomography image data and the physiological motion data; planning a path according to the predicted trajectory data, the target operation model and the constraint condition, and controlling the operation robot to perform an operation according to the planned path; through multi-modal data fusion, a three-dimensional tissue model is established to improve the surgical accuracy; the data are analyzed to determine the lesion direction, the motion trail of the organ tissue is predicted, the three-dimensional tissue model is integrated for path planning, the interference of physiological activities of the organ tissue on the operation is avoided, and the safety of the operation is improved. The embodiment of the invention can be widely applied to the field of intelligent robots.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robot technology, and in particular to a control method, system, electronic equipment and storage medium for a throat intelligent surgical robot. Background Art

[0002] With the development of cutting-edge technologies such as robotics and automatic control, robots are increasingly being used in medical surgeries, such as preoperative planning and minimally invasive surgery. For throat tissue, traditional open surgery has problems such as severe trauma, heavy bleeding, and many complications. Existing throat surgery robots use optical navigation systems and built-in automation strategies to independently plan throat surgery paths and automatically perform minimally invasive surgery, which to a certain extent solves the problems of open surgery. However, for the complex environment of throat tissue, the surgical accuracy of existing throat surgery robots is not high and the surgical safety is low. Summary of the Invention

[0003] The main purpose of the embodiments of the present invention is to provide a control method, system, electronic equipment and storage medium for a throat intelligent surgical robot, which can improve surgical accuracy and safety.

[0004] To achieve the above objectives, an embodiment of the present invention provides a control method for a throat intelligent surgical robot, the method comprising:

[0005] Determine a three-dimensional tissue model based on analysis of acquired visual image data, laser imaging data, and tomographic image data;

[0006] Analyzing the historical lesion data and the tomographic image data to determine lesion direction data; and superimposing the lesion direction data and the three-dimensional tissue model to determine a target surgical model;

[0007] Predicting based on the collected tomographic image data and the acquired physiological motion data to determine predicted trajectory data;

[0008] Path planning is performed based on the predicted trajectory data, the target surgical model, and preset constraints, an execution path is determined, and the surgical robot is controlled based on the execution path.

[0009] In some embodiments, analyzing the acquired visual image data, laser imaging data, and tomographic image data to determine the three-dimensional tissue model specifically includes:

[0010] Modeling is performed based on the visual image data to determine a tissue geometry model; analyzing the laser imaging data to determine tissue elasticity distribution data, and superimposing the tissue elasticity distribution data with the tissue geometry model to determine a three-dimensional model;

[0011] Performing feature extraction on the visual image data to determine vascular feature data; performing feature extraction on the tomographic image data to determine basement membrane feature data; and performing cross-modal registration based on the vascular feature data and the basement membrane feature data to determine a standard feature data set; wherein the standard feature data set includes the registered vascular feature data and the registered basement membrane feature data;

[0012] Calculating based on the blood vessel data and a first preset formula to determine a spatial weight; fusing based on the spatial weight and the standard feature data set to determine blood vessel-basement membrane fusion atlas data;

[0013] The blood vessel-basement membrane fusion atlas data is superimposed on the three-dimensional model to determine a three-dimensional tissue model.

[0014] In some embodiments, analyzing the historical lesion data and the tomographic image data to determine the lesion direction data specifically includes:

[0015] Adjusting parameters of a preset model according to the historical lesion data to determine a lesion prediction model;

[0016] The features of the tomographic image are extracted according to the lesion prediction model to determine lesion feature data; and the lesion feature data are analyzed to determine the lesion direction data.

[0017] In some embodiments, the step of performing prediction based on the collected tomographic image data and the acquired physiological motion data to determine the predicted trajectory data specifically includes:

[0018] Extracting data from the physiological motion data to determine respiratory phase data and blood flow pulsation data;

[0019] Performing gated slicing processing based on the tomographic image data and the respiratory phase data to determine a respiratory image dataset; wherein the respiratory image dataset includes an expiratory image dataset and an inspiratory image dataset;

[0020] Performing registration processing on the respiratory image dataset and the blood flow pulsation data to determine a standard image dataset, and performing vector extraction on the standard image dataset to determine a vector dataset;

[0021] The vector data set is predicted and processed according to the trained prediction model to determine the predicted motion trajectory data.

[0022] In some embodiments, performing path planning based on the predicted trajectory data, the target surgical model, and preset constraints to determine the execution path specifically includes:

[0023] Discretizing the target surgical model to determine a spatiotemporal voxel map data set; analyzing the predicted motion trajectory data to determine an operation constraint area;

[0024] A multi-optimization model is constructed according to the preset constraint conditions, the spatiotemporal voxel map dataset, and the operation constraint area, and the multi-optimization model is solved to determine the execution path.

[0025] In some embodiments, the method further comprises:

[0026] Collecting current tomographic data, current visual image data, and current predicted motion trajectory data according to a preset period, and obtaining current position information of the surgical robot;

[0027] updating the target surgical model according to the current tomographic data to determine an updated target surgical model; and encoding the current visual image data and the current predicted motion trajectory data to determine dynamic obstacle information;

[0028] Local replanning is performed based on the current position information, the updated target surgical model, and the dynamic obstacle information to determine path adjustment information; and the execution path is adjusted based on the path adjustment information and the operation constraint area to determine an adjusted execution path.

[0029] In some embodiments, the method further comprises:

[0030] Get the current vocal cord vibration frequency and current force feedback information;

[0031] Comparing the current force feedback information with a preset tissue deformation model to determine current tissue information, and correcting the surgical robot according to the current tissue information;

[0032] The current vocal cord frequency is compared with a preset threshold range. If the current vocal cord frequency falls within the preset threshold range, the current state of the surgical robot is maintained; otherwise, the contact pressure of the surgical robot is adjusted.

[0033] To achieve the above objectives, another aspect of the present invention provides a control system for a throat intelligent surgical robot, the system comprising:

[0034] The first module is used to analyze the acquired visual image data, laser imaging data, and tomographic image data to determine a three-dimensional tissue model;

[0035] The second module is configured to analyze the historical lesion data and the tomographic image data to determine lesion direction data; and to determine a target surgical model by superimposing the lesion direction data and the three-dimensional tissue model;

[0036] The third module is used to make predictions based on the collected tomographic image data and the acquired physiological motion data to determine predicted trajectory data;

[0037] The fourth module is used to perform path planning based on the predicted trajectory data, the target surgical model and preset constraints, determine the execution path, and control the surgical robot according to the execution path.

[0038] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0039] To achieve the above objectives, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0040] The implementation of the embodiment of the present invention includes the following beneficial effects: the present embodiment provides a control method, system, electronic device and storage medium of a throat intelligent surgical robot, which constructs a three-dimensional tissue model by analyzing the acquired visual image data, laser imaging data and tomographic image data; then, the lesion direction data is determined based on the historical lesion data and tomographic image data, and the determined lesion direction data is superimposed on the constructed three-dimensional tissue model to determine the target surgical model; then, prediction is made based on the tomographic image data and physiological motion data to determine the predicted trajectory data; finally, the predicted trajectory data, the target surgical model and the preset constraints are used to determine the target surgical model. The system performs path planning based on the conditions, obtains the execution path, and controls the surgical robot to perform the surgical operation according to the execution path; constructs a three-dimensional tissue model by combining multiple types of data, realizes multimodal data fusion, improves the accuracy of the surgical environment, and thus improves the accuracy of the surgery; determines the direction of the lesion by analyzing the collected image data, and predicts the movement trajectory of the organ tissue by combining the analysis of image data and physiological motion data; comprehensively considers the three-dimensional tissue model fused by multimodal data, the predicted movement trajectory of the organ tissue and the determined lesion direction for path planning, considers the interference of the physiological activities of the organ tissue on the surgery during the operation, and performs corresponding path planning to avoid it, thereby improving the safety of the surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1This is a schematic flow chart of the steps of a control method of a throat intelligent surgical robot provided by an embodiment of the present invention;

[0042] Figure 2 This is a schematic flow chart of the steps of determining a three-dimensional tissue model in a control method of a throat intelligent surgical robot provided by an embodiment of the present invention;

[0043] Figure 3 This is a flowchart of the steps for determining lesion direction data in a control method for a throat intelligent surgical robot provided by an embodiment of the present invention;

[0044] Figure 4 This is a flowchart of the steps for determining predicted motion trajectory data in a control method for a throat intelligent surgical robot provided by an embodiment of the present invention;

[0045] Figure 5 This is a flowchart of the steps for determining an execution path in a control method for a throat intelligent surgical robot provided by an embodiment of the present invention;

[0046] Figure 6 This is a flowchart of the steps for adjusting the execution path in a control method for a throat intelligent surgical robot provided by an embodiment of the present invention;

[0047] Figure 7 This is a schematic flow chart of another step for adjusting a throat intelligent surgical robot according to a control method of the present invention;

[0048] Figure 8 This is a schematic flow chart of the steps for performing trajectory prediction output in a specific embodiment provided by an embodiment of the present invention;

[0049] FIG9( a ) is a schematic diagram of the steps of performing atlas fusion in a specific embodiment provided by an embodiment of the present invention;

[0050] FIG9( b ) is a schematic flow chart of the steps for determining a blood vessel-basement membrane fusion map in a specific embodiment provided by an embodiment of the present invention;

[0051] Figure 10 This is a schematic flow chart of steps for generating a trajectory of a surgical robot in a specific embodiment provided by an embodiment of the present invention;

[0052] Figure 11 This is a flowchart of steps for implementing a hierarchical strategy in a specific embodiment provided by an embodiment of the present invention;

[0053] Figure 12 This is a flowchart of emergency response steps in a specific embodiment provided by an embodiment of the present invention;

[0054] Figure 13This is a structural block diagram of a control system for a throat intelligent surgical robot provided by an embodiment of the present invention;

[0055] Figure 14 1 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention;

[0056] Among them, 1 is capturing expansion motion, 2 is analyzing contraction elasticity, 3 is extracting 415nm surface blood vessels, 4 is 5μm tomography results, 5 is NBI: Frangi filtering, 6 is OCT: U-Net model, 7 is error > 20μm, 8 is error ≤ 20μm, 9 is Bi-CLSTM model, 10 is multispectral fusion, 11 is 3D-ResNeXt prediction, 12 is 300ms look-ahead, 13 is force / pose / anatomical constraints, 14 is time optimal constraint, 15 is risk index ≤ 0.3, 16 is 0.3 < risk index ≤ 0.7, and 17 is risk index > 0.7. DETAILED DESCRIPTION

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0058] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0059] In the following description, the terms "first\second\third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0060] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention pertains. The terms used in the embodiments of the present invention are for the purpose of describing the embodiments of the present invention only and are not intended to limit the present invention.

[0061] Figure 1 This is an optional flow chart of a control method for a throat intelligent surgical robot provided by an embodiment of the present invention. Figure 1 The method may include but is not limited to steps S101 to S104.

[0062] Step S101, analyzing the acquired visual image data, laser imaging data, and tomographic image data to determine a three-dimensional tissue model;

[0063] Step S102: Analyze historical lesion data and tomographic image data to determine lesion direction data; and superimpose the lesion direction data and the three-dimensional tissue model to determine a target surgical model;

[0064] Step S103, performing prediction based on the collected tomographic image data and the acquired physiological motion data to determine predicted trajectory data;

[0065] Step S104 , performing path planning based on the predicted trajectory data, the target surgical model, and preset constraints, determining an execution path, and controlling the surgical robot based on the execution path.

[0066] In steps S101 to S104 shown in the embodiment of the present application, data of the patient's throat is collected through a binocular endoscope, laser speckle imaging technology, and optical coherence tomography technology to obtain visual image data, laser imaging data, and tomography image data, respectively; then, by analyzing the acquired image data, the surface blood vessel distribution characteristics, deep basement membrane distribution, and tissue elasticity distribution of the throat tissue are extracted from different types of image data, and the different types of data are aligned and unified. Then, image fusion is performed based on the aligned image data to reconstruct a three-dimensional tissue model of the narrow cavity of the throat. The three-dimensional tissue model includes the tissue elastic properties of the throat, as well as the surface blood vessel distribution and deep basement membrane distribution of the throat, forming a digital twin of the surgical environment with both mechanical properties and spatial constraints; then, analysis is performed based on previous throat lesion data For example, the pathological section specimen data of early throat lesions is used to establish a lesion infiltration analysis model by analyzing the lesion data, and then the collected tomographic image data of the throat is input into the established lesion infiltration analysis model for prediction to obtain the micro-invasion results or lesion evolution results of the throat lesions, and the predicted lesion micro-invasion results are superimposed on the established three-dimensional tissue model to obtain a high-precision reconstruction model of the patient's throat; then, based on the collected tomographic data of the throat and the monitored physiological activity data of the patient's throat, such as breathing, swallowing, etc.; by analyzing the tomographic data and physiological activity data, a vocal cord motion trajectory prediction model is established, and then the vocal cord motion trajectory within a period of time is predicted; then, path planning is performed based on the established high-precision reconstruction model and the predicted vocal cord motion trajectory, and according to the planning results, the surgical robot is controlled to avoid the vocal cord vibration and throat movement caused by physiological activities, and surgery is performed accurately on the lesion location.

[0067] In some embodiments, in step S101, a micro-snake-shaped robotic arm can be used to perform the surgery. The robotic arm has a small diameter and can fit into narrow cavities. A tactile sensor can also be provided at the end of the robotic arm to detect the contact pressure between the robotic arm and the tissue to avoid damage to tissues and organs, without limitation.

[0068] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S204:

[0069] Step S201: Modeling is performed based on visual image data to determine a tissue geometry model; analyzing laser imaging data to determine tissue elasticity distribution data, and superimposing the tissue elasticity distribution data with the tissue geometry model to determine a three-dimensional model;

[0070] Step S202: Feature extraction is performed on the visual image data to determine vascular feature data; feature extraction is performed on the tomographic image data to determine basement membrane feature data; and cross-modality registration is performed based on the vascular feature data and the basement membrane feature data to determine a standard feature data set; wherein the standard feature data set includes the registered vascular feature data and the registered basement membrane feature data;

[0071] Step S203, performing calculation based on the blood vessel data and the first preset formula to determine the spatial weight; fusing the spatial weight with the standard feature data set to determine the blood vessel-basement membrane fusion atlas data;

[0072] Step S204: superimpose the blood vessel-basement membrane fusion atlas data with the three-dimensional model to determine a three-dimensional tissue model.

[0073] In step S201 of some embodiments, the geometric configuration of the narrow cavity is obtained by analyzing the collected visual image data and combining it with the corresponding high-precision positioning data for modeling; then, the collected laser imaging data is analyzed, and the tissue elasticity of different tissues is determined by the reflection of the laser by different tissues in the throat; the tissue elasticity of different tissues obtained by analysis is superimposed on the constructed geometric configuration to obtain a digital twin of the surgical environment with mechanical properties and spatial constraints, so as to facilitate subsequent path planning and surgical monitoring and avoid trauma to normal tissues.

[0074] In step S202 of some embodiments, the system analyzes and extracts features from the collected visual image data to obtain surface vascular distribution feature data for the region, which serves as one of the boundary conditions for subsequent path planning, thereby avoiding uncontrollable bleeding during surgery. Simultaneously, the system performs feature analysis on the tomographic data of the throat to determine the distribution features of the basement membrane deep in the throat, which also serves as one of the boundary conditions for subsequent path planning, thereby avoiding damage to the basement membrane during surgery, thereby affecting the normal function of the throat. The system then registers the obtained vascular distribution features with the basement membrane distribution features to achieve data alignment and improve data accuracy. In this embodiment, the registration is performed using the following formula:

[0075]

[0076] Among them, T(x) is the deformation field to be determined, f NBI (x i ) is the centerline feature of the blood vessels in the NBI image, f OCT (x i ) is the boundary feature of the OCT basement membrane; in the above formula, NBI image is visual image data, OCT is tomographic image data, the first half of the formula is the feature point matching term between different image data, and the second half is the smoothing term of image deformation. By solving the above formula, the deformation field T with the minimum ε(T) is obtained. The image data is processed according to the deformation field to achieve alignment between different data.

[0077] In step S203 of some embodiments, the system needs to fuse the registered vascular distribution features and basement membrane distribution features to obtain a fused image that provides a dynamic anatomical navigation reference for the subsequent surgical robot, avoiding high-risk blood vessels and basement membranes in the throat area, and improving surgical accuracy and safety. In this embodiment, the system determines a spatial weight based on the blood vessel density in the vascular feature data. Because in the actual environment, the vascular distribution and basement membrane distribution are both three-dimensional, it is necessary to determine a spatial weight. The formula for determining the spatial weight is as follows:

[0078]

[0079] Among them, ω(x) is the spatial weight, D vessel (x) is the percentage of blood vessel pixels in the 3*3 area centered on x, k and D0 are preset parameters. In this embodiment, k=5 and D0=0.3. Then, the system performs image fusion according to the following formula:

[0080] I fused (x)=ω(x)*I NBI (x)+(1-ω(x))*I OCT (x)

[0081] Among them, I fused (x) is the fused blood vessel-basement membrane image, ω(x) is the spatial weight, I NBI (x) is the NBI image, I OCT (x) is the OCT image.

[0082] In step S204 of some embodiments, the fused blood vessel-basement membrane image is superimposed on the previously constructed three-dimensional model to further improve the accuracy of the constructed model, making it closer to the throat tissue in the real environment, thereby improving the surgical accuracy and safety of the surgical robot.

[0083] See also Figure 3 In some embodiments, step S102 may include but is not limited to steps S301 to S302:

[0084] Step S301, adjusting parameters of a preset model based on historical lesion data to determine a lesion prediction model;

[0085] Step S302: extract features from the tomographic image according to the lesion prediction model to determine lesion feature data; analyze the lesion feature data to determine lesion direction data.

[0086] In step S301 of some embodiments, the system uses a machine learning model to analyze the collected historical pathological section specimens, learns the development trend and direction characteristics of the lesions, and obtains a lesion prediction model; in this embodiment, the system uses the 3D-ResNeXt-101 model for feature learning to construct a lesion prediction model, specifically including learning the infiltration direction of the lesion along the muscle fiber, that is, the anisotropic expansion model; and learning the preferential path of perivascular invasion, that is, the area where the perivascular space is greater than fifty microns; and learning the early signs of cartilage destruction, that is, the three-dimensional connectivity analysis of the reflectivity drop area; by constructing a lesion prediction model, the system learns the invasion pattern of the lesion in three-dimensional space, breaking through the limitations of traditional two-dimensional pathology analysis, and providing a quantitative navigation basis for directional expanded resection in robotic surgery to improve surgical accuracy.

[0087] In step S302 of some embodiments, after obtaining the lesion prediction model, the collected tomographic image data is input into the lesion prediction model for analysis to identify the distribution area of ​​the current patient's throat lesions and possible development direction, such as spread along nerve bundles, perivascular spaces, or fascial spaces. In this embodiment, the system inputs the collected tomographic image data into the lesion prediction model for processing. The lesion prediction model maps the learned features and input data and outputs the micro-invasion direction data of the current patient's lesions. This is specifically achieved through the following logic:

[0088] enter:

[0089] Network: 3D-ResNeXt-101 (grouped convolution kernel size is 3×3×3)

[0090] Output: (Main direction angle of micro-invasion)

[0091] in, is the input tensor, i.e., the 3D medical image volume data; 512×512 is the spatial resolution of a single-layer medical image, i.e., the number of pixels on the x / y axis; 400 is the number of slices on the z axis, i.e., the depth of the anatomical structure covered; Characterize a three-dimensional medical imaging volume dataset; in the network, the group convolution parameters g = 32, k = 3 × 3 × 3; where the number of groups is g = 32, that is, the input channels are divided into 32 groups, each group performs a convolution operation independently, reducing the number of input data parameters to 1 / g × standard convolution; the convolution kernel size is 3 × 3 × 3, the convolution kernel is a three-dimensional convolution kernel, the size of each direction (x / y / z) is 3, and the total receptive field of each convolution kernel is 3 3 =27 voxels; The output scalar is the predicted value of the main direction angle of the micro-invasion, and its value range is [0,2π).

[0092] See also Figure 4 In some embodiments, step S103 may include but is not limited to steps S401 to S404:

[0093] Step S401, extracting physiological motion data to determine respiratory phase data and blood flow pulsation data;

[0094] Step S402, performing gated slicing processing based on the tomographic image data and the respiratory phase data to determine a respiratory image dataset; wherein the respiratory image dataset includes an expiratory image dataset and an inspiratory image dataset;

[0095] Step S403, performing registration processing on the respiratory image dataset and the blood flow pulsation data to determine a standard image dataset, and performing vector extraction on the standard image dataset to determine a vector dataset;

[0096] Step S404 : performing prediction processing on the vector data set according to the trained prediction model to determine predicted motion trajectory data.

[0097] In step S401 of some embodiments, the patient's physiological activity information, such as breathing and swallowing, is collected by setting up chest and abdominal motion sensors and electromyographic signal sensors, and the patient's blood flow pulsation information is collected through corresponding sensors; by analyzing the collected data of the sensors, the phase data of the patient's breathing, the relevant characteristic data of the swallowing action and the blood flow pulsation data are obtained, and then the impact of the patient's physiological activity on the vocal cord movement in the throat area is analyzed, so as to subsequently accurately predict the vocal cord movement, and compensate and control the operation of the surgical robot based on the prediction results to improve the surgical accuracy and safety.

[0098] In step S402 of some embodiments, a three-dimensional tomographic scan of the patient's throat is performed for gated slicing, from which the patient's chest movement data during the exhalation and inhalation phases are extracted. Based on the extracted exhalation phase data and inhalation phase data, the impact of respiratory movement on the organs and tissues of the throat is analyzed, and the movement trajectory of the throat is predicted. In this embodiment, a spiral CT scan with a doorway period of 0.5 seconds is used to collect 4D-CT time series data of the patient's respiratory movement.

[0099] In step S403 of some embodiments, the system performs gated slicing processing on the collected respiratory motion data, extracts image data of the exhalation phase and the inhalation phase, and then performs spatiotemporal registration of other physiological activity data such as blood flow pulsation data with the exhalation phase and the inhalation phase, aligns other physiological activities with the patient's respiratory movements in three-dimensional angles and time dimensions, and simulates the patient's actual physiological activities with high precision; the system performs vector extraction on the spatiotemporal registered data to determine the motion vector data of different tissues and organs in the throat under the influence of physiological activities for subsequent motion trajectory prediction; in this embodiment, after performing spatiotemporal registration on different data, the system can segment the registered data and extract the motion image of the vocal cords therein to reduce the amount of data analysis and improve analysis efficiency; in this embodiment, the gating processing is performed according to the following gating mechanism equation:

[0100]

[0101] Among them, the gate control mechanism equation is the input gate equation i t , forget gate equation f t , candidate cell state equation g t , cell state update equation c t , output gate equation o t And the hidden state update equation h t ;W xi For input X t The weight matrix is ​​used to map the current input to the hidden space, and its dimension is W hiis the previous hidden state h t-1 The weight matrix is ​​used to control the influence of historical information, and its dimension is b i is the bias vector of the input gate, which is used to adjust the activation threshold and has the dimension σ is the Sigmoid activation function, which is used to determine which information is retained, and its output range is [0,1]; W xf For input X t The weight matrix of , but used to independently learn the forgetting logic; W hf Previous hidden state h t-1 The weight matrix of is used to independently learn the forgetting logic; b f is the bias vector of the forget gate, which is used to control “how much old memory to forget”; W xg is the weight parameter of the candidate state equation, whose dimension is the same as W of the input gate xi Same; W hg is the previous hidden state h t-1 The weight matrix of the input gate has the same dimension as W hi Same; b g is the bias vector of the candidate state; tanh is the hyperbolic tangent activation function, which is used to generate new candidate memories, and its output range is [-1,1]; f t ⊙c t-1 The forget gate controls the retention ratio of the old cell state; i t ⊙g t Controls the ratio of adding new candidate memories for the input gate; W xo For input X t The weight matrix is ​​used to independently learn the output logic; W ho is the previous hidden state h t-1 The weight matrix is ​​used to independently learn the output logic; b o is the bias vector of the output gate, which is used to determine the exposure degree of the current cell state; tanh(c t ) is used to compress the cell state to the range [-1,1] to avoid gradient explosion; o t ⊙tanh(c t ) is the output gate controlling the hidden state that is finally exposed to the next time step; * is the 3D convolution operation, and ⊙ is the element-by-element multiplication, i.e., the Hadamard product, which is used to selectively retain or discard information.

[0102] In step S404 of some embodiments, the vector data extracted in step S403 is input into a constructed and trained prediction model for processing. The prediction model extracts the temporal features of the throat tissue movement over time at different stages through spatiotemporal convolution, fuses the extracted temporal features, establishes the temporal features of the complete respiratory cycle, and outputs the motion trajectory prediction result after regression processing of the temporal features of the complete respiratory cycle. In this embodiment, the system uses a Bi-CLSTM model to predict the motion trajectory of the motion vector data and a bidirectional convolution LSTM core to extract features of the motion vector. Among them, the forward CLSTM captures the expansion period motion features, and the reverse CLSTM analyzes the contraction period elastic features:

[0103] Forward propagation:

[0104] Backward propagation:

[0105] in, The input data at time step t represents the 3D-CT volume data of the tth frame; W conv is the 3*3*3 convolution kernel parameter; is the forward hidden state at time step t; is the forward hidden state of the next time step t+1; W conv The convolution kernel parameters are 3*3*3; is the backward hidden state at time step t; is the backward hidden state at the next time step t+1; the Bi-CLSTM model fuses the extracted feature data to obtain the spatiotemporal feature tensor; the Bi-CLSTM model performs fully connected regression processing on the spatiotemporal feature tensor to obtain the motion trajectory prediction result; wherein, the Bi-CLSTM model performs fully connected regression processing according to the following trajectory regression loss function:

[0106]

[0107] Among them, the first half of the formula is the position error of the motion trajectory prediction, and the second half is the motion smoothness constraint of the motion trajectory prediction; the time step number T is the total length of the time series data, and the predicted value is the predicted output of the model at time step t, the true value is the true observation value at time step t, The smaller the value of , the stronger the model fitting ability; It is the time series difference term, which represents the difference between the predicted value and the true value of adjacent time steps.

[0108] Specifically, and The mathematical definitions are as follows:

[0109]

[0110] See also Figure 5 In some embodiments, step S104 may include but is not limited to steps S501 to S502:

[0111] Step S501: discretize the target surgical model to determine a spatiotemporal voxel map data set; analyze the predicted motion trajectory data to determine the operation constraint area;

[0112] Step S502 : constructing a multi-optimization model based on preset constraints, the spatiotemporal voxel map dataset, and the operation constraint region, solving the multi-optimization model, and determining an execution path.

[0113] In step S501 of some embodiments, the system constructs a three-dimensional tissue model of the throat by analyzing different types of image data, and predicts the vocal cord movement trajectory of the throat based on the tomography data and physiological activity data. Then, based on preset constraints, the system plans the operation path of the surgical robot according to the predicted vocal cord movement trajectory and the constructed three-dimensional tissue model; the system discretizes the constructed three-dimensional tissue model into voxels of a certain size to obtain a spatiotemporal voxel map; in this embodiment, the voxel size is 0.1*0.1*0.1mm 3 , each voxel is represented by the following mathematical formula:

[0114]

[0115] Among them, ρ 组织 is tissue density, derived from OCT reflectivity; To predict the movement speed, it is derived from the prediction model; R 风险 is the risk weight, including the risk of vascular contact and the risk of lesion infiltration. The system can update the voxel attribute information at a certain frequency to compensate for the model topology changes caused by tissue movement and physiological movement. Then, the system analyzes the vocal cord motion trajectory prediction results output by the prediction model and generates a spatiotemporal tunnel constraint. This serves as the area where the surgical robot is allowed to operate, and the surgical robot's trajectory planned by the system cannot exceed this area. In this embodiment, the spatiotemporal tunnel constraint is as follows:

[0116]

[0117] The formula is expressed as the intersection of the "allowed operation area" at all times within the time or space index range t = k to k + N; k is the index value of the starting time step or spatial position; N is the number of consecutive operation steps or the length of the time window starting from the initial index k; is the position coordinate or key parameter value at index t; B 0.2mm It is a spherical (or cubic) area with a radius of 0.2 mm and the origin as the center; ∩ intersection refers to the common part of the extended area at all times; is the expansion operation of the set; N = 10, representing the prediction window. Specifically, B 0.2mm The expression is as follows:

[0118]

[0119] In step S502 of some embodiments, the system constructs a multi-objective optimization model based on the obtained spatiotemporal voxel map, the obtained allowable operation area, and the preset constraints. The preset constraints include force, posture, and anatomical safety constraints, including but not limited to the range of motion of the surgical robot's robotic arm not exceeding the throat space and the force applied by the robot to the throat tissue not being too large. At the same time, the system also sets an efficiency optimization goal to reduce the harm caused to the patient by excessive surgical time, while avoiding the impact of excessive surgical time on the surgical effect. In this embodiment, the system constructs the following multi-objective optimization model:

[0120]

[0121] Among them, α is the integral weight coefficient, α>0, the larger its value, the smoother the path; is the first-order derivative of q(t) with respect to time, indicating the instantaneous velocity; q(t) is the path function, indicating the state of the system at time t; t0 is the starting time of the path; t f is the termination time of the path; β is the risk weight coefficient; R risk (V i ) represents area V i The risk assessment function is usually a nonlinear mapping; the mathematical expression of the risk assessment function is as follows:

[0122] R risk (v i )=exp(-dist(p(t),V i ))

[0123] In this mathematical representation, the closer to the danger zone, the higher the risk value; δ i is a binary variable (0 or 1) indicating whether the path passes through region V i ;γ is the time weight coefficient, which is used to penalize the total time cost and balance efficiency with other goals; t total For the total time.

[0124] Constraints: Robot arm kinematics: Joint angular velocity limits Safety margin: ≥0.5mm from the high-risk area, vascular contact force ≤0.3N; real-time: planning cycle ≤5ms; among them, high-risk areas are divided according to surgical regulations, such as the recurrent laryngeal nerve pathway is divided into a red restricted area with a protection radius of 2mm and a confidence level of >99%, and the vocal cord muscle layer of the throat is divided into a yellow warning area with an allowed contact time of <5s and a pressure threshold of 0.2N; or, according to the constructed three-dimensional tissue model and relevant surgical regulations, corresponding surgical quantitative indicators are set: basement membrane rupture area: red highlight (threshold: rupture length >200μm); abnormal blood vessel clusters: yellow mark (density >5 / mm 2 and diameter variability >30%); safety margin: 0.5mm from the edge of the lesion (including a vascular avoidance buffer zone); the system obtains the shortest path that meets the constraints by solving the constructed multi-objective optimization model. For example, the RRT* algorithm is used for solving, and the global optimal path is searched in the constructed three-dimensional tissue model through random sampling exploration and progressive optimization mechanism. Specifically, the initial path connecting the starting point and the target is quickly generated in the high-dimensional configuration space, and inefficient nodes are continuously replaced by rewiring, gradually approaching the shortest, low-risk theoretical optimal solution.

[0125] See also Figure 6 In some embodiments, the control method of the throat intelligent surgical robot provided by the embodiment of the present invention may also include but is not limited to steps S601 to S603:

[0126] Step S601, collecting current tomographic data, current visual image data, and current predicted motion trajectory data according to a preset period, and obtaining current position information of the surgical robot;

[0127] Step S602: updating the target surgical model according to the current tomographic scan data to determine the updated target surgical model; and encoding the current visual image data and the current predicted motion trajectory data to determine dynamic obstacle information;

[0128] Step S603 , performing local replanning based on the current position information, the updated target surgical model, and the dynamic obstacle information to determine path adjustment information; and adjusting the execution path based on the path adjustment information and the operation constraint area to determine the adjusted execution path.

[0129] In step S601 of some embodiments, the system controls the surgical robot to perform surgery according to the planned motion path. The patient's physiological activities during the operation will drive the movement of tissues in the surgical site, such as breathing, blood pulsation, etc., so that normal organ tissues will hinder the original robot motion path; the system needs to monitor the multimodal data of the surgical site in real time, including visual images, laser imaging images, tomography images, physiological activity data, etc., update the three-dimensional model in real time, and predict the motion trajectory of tissues such as the vocal cords; according to the current position information of the surgical robot, the robot's motion path is adjusted in real time.

[0130] In step S602 of some embodiments, the system updates the constructed three-dimensional model in real time based on the real-time collected and sensed data, and updates the predicted results of the vocal cord movement trajectory in the throat in real time; in this embodiment, the system encodes the multimodal data such as the vocal cord movement prediction, blood vessel distribution and infiltration direction sensed and collected in real time into dynamic obstacles according to the incremental update strategy, and sets the expansion radius to 0.3mm to define a certain safety area; then the updated surgical model is evaluated to determine the area affected by physiological activities, such as the change volume within ±3mm 3 The area of ​​the cube; based on the updated three-dimensional model and the determined area, the motion path of the surgical robot is adjusted in real time.

[0131] In step S603 of some embodiments, the system dynamically adjusts the current motion path of the surgical robot locally based on the updated three-dimensional model, the predicted vocal cord trajectory, and the determined impact area. In this embodiment, the system uses a local path replanning strategy for dynamic path adjustment and combines risk bias sampling and spatiotemporal tunnel constraints for path adjustment. Risk bias sampling reduces the sampling probability of high-risk areas by 80%. At the same time, in order to smooth the locally replanned path, the replanned path is processed using an elastic band optimization strategy. The elastic band optimization strategy uses the following energy function:

[0132]

[0133] The goal of this energy function is to balance the smoothness, compactness, and security of the path by minimizing the total energy, which correspond to k·curvature in the energy function. term, λ·‖Δl i ‖ 2 Term and μ·risk field repulsion energy i Term; where E is the total energy, which represents the comprehensive cost function of the path; k is the curvature weight coefficient, the larger its value, the stronger the penalty for the curvature of the path; curvature is the penalty for amplifying high curvature areas; λ is the compactness weight coefficient, which is used to adjust the weight of path length change. The larger its value, the more restricted the local length change of the path; μ is the safety weight coefficient, which is used to adjust the priority of the safety item; risk field repulsion energy i is the distance-based repulsive potential function, which is mathematically expressed as follows:

[0134]

[0135] Among them, σ is the attenuation coefficient, which is used to control the range of the repulsive force, for example, σ = 0.5m. According to this formula, the path point p i The closer to the obstacle, the higher the repulsion energy, forcing the path away from the danger zone.

[0136] See also Figure 7 In some embodiments, a control method of a surgical robot provided by an embodiment of the present invention may further include but is not limited to steps S701 to S703:

[0137] Step S701, obtaining the current vocal cord vibration frequency and current force feedback information;

[0138] Step S702 , comparing the current force feedback information with a preset tissue deformation model to determine current tissue information, and modifying the surgical robot based on the current tissue information;

[0139] Step S703, compare the current vocal cord frequency with the preset threshold range. If the current vocal cord frequency falls within the preset threshold range, maintain the current state of the surgical robot; otherwise, adjust the contact pressure of the surgical robot.

[0140] In step S701 of some embodiments, when the surgical robot performs a surgical operation along a path, the system collects the vibration frequency of the patient's vocal cords through relevant instruments, and detects the contact pressure information of the surgical robot's end or clamping instrument on the tissue through sensors in real time; the system adjusts the robot in real time according to the detected vocal cord vibration frequency and contact pressure information to ensure that the surgical robot retains the patient's vocal function to the maximum extent when performing surgery to remove the lesion.

[0141] In step S702 of some embodiments, the system collects contact pressure data of the throat tissue using a force sensor and a pressure array. Based on the collected contact pressure data, the system quantifies the nonlinear relationship between the contact force and the compression of the throat tissue, and constructs a tissue deformation model. The system compares the real-time collected contact pressure information with the constructed tissue deformation model to determine whether the pressure exceeds the tissue's tolerance limit and whether the current contact tissue is normal. Specifically, the system performs piezoelectric impedance spectroscopy analysis on the contact pressure information to quantify the difference in elastic modulus between the mucosal layer (0.5-3 kPa) and the muscular layer (5-15 kPa) in real time. When a sudden change in the hardness threshold is detected, such as ΔE > 5 kPa at the mucosa-muscular layer junction, the area is identified as an abnormally hardened area, enabling dynamic correction of pathological grading and resection range during surgery to improve surgical accuracy.

[0142] In step S703 of some embodiments, the system collects the vocal cord vibration frequency, quantifies the mechanical characteristics of the vocal cord tissue, such as tension, elastic modulus, etc., and compares the real-time monitored vocal cord vibration frequency with the normal physiological frequency range. When it is detected that the vocal cord vibration frequency deviates from the normal physiological frequency range, such as when the frequency drops by more than 20% due to the contact of the surgical robot's clamped instrument with the vocal cord, the system dynamically adjusts the contact pressure of the surgical robot's clamping instrument through the set PID controller to avoid mucosal edema or vocal cord muscle fiber damage caused by excessive compression; at the same time, the system controls the robot's mechanical arm through vibration phase synchronization technology, so that the mechanical arm movement is decoupled from the vocal cord vibration rhythm, thereby minimizing damage to the vocal cord tissue; in this embodiment, a laser Doppler vibrometer (LDV) is used to capture the vocal cord surface vibration waveform at a sampling rate of 20kHz, and a vocal cord stiffness-vibration coupling model (goodness of fit R 2 >0.9); when the frequency offset >15% or the amplitude attenuation >30% is detected, the PID controller calculates the contact force data according to the predefined transfer function, such as The real-time viscosity adjustment of the magnetorheological fluid joint (response <5ms) limits the instrument pressure to a safe threshold (e.g., adjustable from 0.05 to 0.3N). At the same time, the phase-locked loop (PLL) synchronizes the instrument movement with the vibration peak / valley timing (error <1ms), allowing the robotic arm to perform precise operations during the vocal cord relaxation period (when the amplitude is maximum), avoiding interference with the vibration rhythm. Ultimately, sub-Newton force control accuracy and micron-level movement synchronization are achieved, ensuring that the postoperative vocal cord vibration characteristic retention rate is >98%.

[0143] The following describes the embodiments of the present invention in detail with reference to specific application examples:

[0144] The control method of the throat intelligent surgical robot of the embodiment of the present invention is applied to a system of a micro-snake-shaped robotic arm, on which a dual-mode laser-ultrasonic knife and a nano-hemostatic clip deployer are provided. Before performing the operation, the system collects multimodal data through various sensors and imaging technologies, generates a three-dimensional surgical restricted area map based on the tomographic data, and divides it into different warning areas, such as setting the recurrent laryngeal nerve area as a red restricted area, which the robotic arm needs to avoid. Then, refer to Figure 8 , the three-dimensional scanning data with time series information in the multimodal data is input into the Bi-CLSTM model for processing, the vocal cord motion trajectory is reconstructed, the vocal cord motion trajectory within a certain window is predicted, and it is updated in real time according to a certain frequency; Among them, the Bi-CLSTM model is determined by constructing a bidirectionally coupled dynamic equation based on the laryngeal tissue movement and instrument deformation. The specific equation is as follows:

[0145]

[0146] Where M is the mass matrix, which is used to describe the inertial characteristics of the tissue. It is a diagonal matrix or contains coupling terms. C is the damping matrix, which characterizes the viscous energy dissipation of the tissue. K is the stiffness matrix, which reflects the elastic characteristics of the tissue. x is the displacement vector, which represents the displacement of the key point on the tissue surface or inside. F ext is the external force vector, including the contact force of the surgical instrument and the periodic force generated by physiological movements (such as breathing and heartbeat); τ is the joint torque vector, which is the torque required to drive the surgical instrument (such as the robotic arm joint); F contact is the contact force vector, which represents the force / torque combination generated by the contact between the instrument tip and the tissue; is the joint angular velocity vector, which is used to represent the instantaneous rotational velocity of each joint of the instrument; D(θ) is the angle-dependent damping matrix, which is used to describe the asymmetric matrix of instrument joint friction and drive system damping, and is usually modeled as follows:

[0147] D(θ)=diag(d1(θ1),……,d n (θ n ))

[0148] Among them, d i Varies with the joint angle θ.

[0149] J is the Jacobian matrix, which is a geometric derivative matrix used to map the Cartesian space velocity to the joint space. It is expressed as a mapping relationship from the joint angle θ to the displacement x. The mathematical expression is as follows:

[0150] Dimensions:

[0151] The dimensions include n joints and a 6-DOF end.

[0152] Then, the system combines the visual image data and laser imaging images in the multimodal data to build a basic three-dimensional model; referring to Figures 9(a) and 9(b), the system extracts features from the NBI data and OCT data in the multimodal data, extracting the vascular centerline feature data and basement membrane distribution features; then, the system fuses the extracted feature data and performs cross-modal registration on the fused data to achieve image fusion and obtain a vascular-basement membrane fusion image; the system quantifies surgical indicators based on the vascular-basement membrane fusion image to guide the system to avoid high-risk areas such as abnormal vascular clusters and basement membrane rupture areas when planning the path, and dynamically adjust the robotic arm and instruments Safety boundaries of motion trajectory. For example, during laser ablation, the fusion atlas can simultaneously identify the proximity of the instrument tip to the blood vessel (threshold <0.3mm), triggering adaptive impedance control to reduce the contact force (dynamically adjusted from 0.5N to 0.2N), thereby reducing the risk of intraoperative bleeding by 62% and ensuring that the range of damage to the basement membrane integrity is controlled within ±50μm, significantly improving the operational accuracy and safety of narrow cavity surgery. Based on the constructed 3D model and fused images, the system analyzes and predicts the direction of micro-invasion of the lesion based on previous pathological section specimen data and current tomographic image data, providing a navigation basis for the path planning of the robotic arm. Figure 10 The system plans the path of the robotic arm based on the three-dimensional model, fused images, micro-invasion direction of the lesion, and predicted vocal cord motion trajectory. It searches for the optimal path to the lesion in the three-dimensional model through the RRT algorithm, and locally replans the optimal path based on multimodal data such as real-time vocal cord motion prediction, vascular distribution, and infiltration direction. This allows for real-time motion compensation of the robotic arm, adjusting the robotic arm's posture and path to avoid tissue movement caused by vocal cord movement and breathing. The system controls the robotic arm to perform laryngeal surgery along the planned path, and determines tissue properties by analyzing real-time force feedback data for tissue hardness testing. The specific analysis formula is as follows:

[0153]

[0154] Among them, V out is the output voltage, which changes with time t and is used to represent the total output signal of the system; α i is the channel weight coefficient, which represents the gain or weight coefficient of the i-th channel, and is used to adjust the contribution ratio of the channel to the total output. If α i >1, it means amplifying the channel signal; if 0<α i <1, indicating attenuation of the channel signal; P i (t) is the input signal function of the i-th channel, which changes with time t; e -βtis the global time attenuation factor, which is used to synchronously modulate all channel signals. β is the attenuation constant, which is determined by the system time constant. According to the tissue properties, the corresponding instrument is switched to perform tissue separation. For example, for the mucosal layer, a dual-mode laser is used for precise vaporization, while for the muscle layer, the robotic arm switches to mechanical vibration for cutting. At the same time, the system monitors the blood vessel density through laser speckle imaging data, determines the bleeding risk level in real time, and takes appropriate treatment measures. When the blood flow rate is determined to be less than 1mm / s through analysis of laser speckle imaging data, the robotic arm performs local electrocoagulation on the bleeding location. If the system detects jetting bleeding, the robotic arm deploys hemostatic clamps for hemostasis. At the same time, refer to Figure 11 The system is equipped with an autonomous grading strategy. It calculates the risk index based on real-time multimodal data to enable autonomous surgery and emergency response by the robotic arm. When the calculated risk index is ≤0.3, the system adopts an autonomous execution strategy, namely the level 2 strategy, which uses vision-force perception for servo control. The specific control mechanism is as follows:

[0155]

[0156] Among them, τ cmd J is the torque command sent to the robot joint driver to control the joint movement; T is the Jacobian matrix, used for differential motion mapping from joint space to Cartesian space; K p is the feedback gain coefficient of the visual error, which is used to amplify or attenuate the correction effect of the visual error; e vis is the visual tracking error, which is the difference between the target pose of the end effector and the actual pose, i.e., the OCT and CT registration result; K f It is the feedforward compensation gain of force change, used to predict external disturbances and generate torque in advance; is the damping torque of the speed section, which is used to suppress mechanical vibration and overshoot. Its mathematical expression is as follows:

[0157]

[0158] Where B is the damping coefficient matrix, is the joint angular velocity vector; ΔF is the contact force error, which represents the change in the interaction force between the end effector and the environment. It is measured by a six-dimensional force sensor and its mathematical expression is as follows:

[0159]

[0160] The Level 2 strategy sets a three-level alarm mechanism to supervise the robotic arm's autonomous surgery. Specifically, the following are the following: Level 1 alarm lights up yellow: When the contact force is greater than 0.25N, the robotic arm automatically slows down by 50%; Level 2 alarm lights up orange: When the visual deviation is greater than 0.3mm, the robotic arm initiates path replanning; Level 3 alarm lights up red: When the physiological signal is abnormal, the robotic arm is forced to switch to AI planning and confirm it with the doctor, which is the Level 1 strategy. When the risk index is 0.3 < ≤ 0.7, or when the instrument is detected entering the 2mm protection zone of the recurrent laryngeal nerve or the red restricted zone, the Level 1 strategy is used to plan several candidate paths. These are presented to the doctor through an augmented reality interface and the doctor selects them. For example, three paths with decreasing risk are generated: red, yellow, and green. The red path is the shortest, but the probability of nerve contact is greater than 5%; the yellow path detours a certain distance, but takes 20% longer; and the green path completely avoids the restricted zone, but takes 50% longer. In addition, a dual-channel optical encoder is set to verify the consistency of the displacement command as a safety mechanism. When the risk index is greater than 0.7, the Level 3 strategy is used for fully autonomous emergency response. Figure 12 ; Among them, the risk index is calculated by the following formula.

[0161]

[0162] The implementation of the embodiment of the present invention includes the following beneficial effects: the present embodiment provides a control method, system, electronic device and storage medium of a throat intelligent surgical robot, which constructs a three-dimensional tissue model by analyzing the acquired visual image data, laser imaging data and tomographic image data; then, the lesion direction data is determined based on the historical lesion data and tomographic image data, and the determined lesion direction data is superimposed on the constructed three-dimensional tissue model to determine the target surgical model; then, prediction is made based on the tomographic image data and physiological motion data to determine the predicted trajectory data; finally, the predicted trajectory data, the target surgical model and the preset constraints are used to determine the target surgical model. The system performs path planning based on the conditions to obtain the execution path, and controls the surgical robot to perform the surgical operation according to the execution path; constructs a three-dimensional tissue model by combining multiple different types of data, realizes multimodal data fusion, improves the accuracy of the surgical environment, and thus improves the accuracy of the surgery; determines the direction of the lesion by analyzing the collected image data, and predicts the movement trajectory of the organ tissue by combining the analysis of image data and physiological motion data; comprehensively considers the three-dimensional tissue model fused by multimodal data, the predicted movement trajectory of the organ tissue, and the determined lesion direction for path planning, considers the interference of the physiological activities of the organ tissue during the operation on the operation, and performs corresponding path planning to avoid it, thereby improving the safety of the surgery. After the robotic arm completes the autonomous surgery, it can be verified by femtosecond laser microdissection combined with Raman spectroscopy analysis. When the cutting edge is positive, the robotic arm automatically performs extended resection to improve the surgical effect.

[0163] like Figure 13 As shown, an embodiment of the present invention further provides a control system for a throat intelligent surgical robot, which can implement the above-mentioned control method for a throat intelligent surgical robot. The system includes:

[0164] The first module is used to analyze the acquired visual image data, laser imaging data, and tomographic image data to determine a three-dimensional tissue model;

[0165] The second module is configured to analyze the historical lesion data and the tomographic image data to determine lesion direction data; and to determine a target surgical model by superimposing the lesion direction data and the three-dimensional tissue model;

[0166] The third module is used to make predictions based on the collected tomographic image data and the acquired physiological motion data to determine predicted trajectory data;

[0167] The fourth module is used to perform path planning based on the predicted trajectory data, the target surgical model and preset constraints, determine the execution path, and control the surgical robot according to the execution path.

[0168] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0169] The present application also provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned control method for a throat intelligent surgical robot. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0170] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0171] See also Figure 14 , Figure 14 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0172] The processor 1401 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0173] The memory 1402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1402 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1402 and is called by the processor 1401 to execute the control method of a throat intelligent surgical robot in the embodiments of this application;

[0174] Input / output interface 1403, used to implement information input and output;

[0175] Communication interface 1404, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0176] Bus 1405 , which transmits information between various components of the device (e.g., processor 1401 , memory 1402 , input / output interface 1403 , and communication interface 1404 );

[0177] The processor 1401 , the memory 1402 , the input / output interface 1403 and the communication interface 1404 are connected to each other in communication within the device via a bus 1405 .

[0178] Among them, the memory is a non-transient computer-readable storage medium that can be used to store non-transient software programs and non-transient computer executable programs. The memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory optionally includes a remote memory remotely arranged relative to the processor, and these remote memories can be connected to the processor via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0179] In addition, the embodiments of the present application further disclose a computer program product or computer program, which is stored in a computer-readable storage medium. The processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above-mentioned method. Similarly, the contents of the above-mentioned method embodiment are all applicable to the present storage medium embodiment, and the functions specifically implemented by the present storage medium embodiment are the same as those of the above-mentioned method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned method embodiment.

[0180] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned control method of the throat intelligent surgical robot.

[0181] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0182] It is understood that all or some steps, systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those of ordinary skill in the art, the term computer storage medium is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data) and is volatile and non-volatile, removable and non-removable media. Computer storage media includes but is not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or can be used to store desired information and any other medium that can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0183] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A control method for a throat intelligent surgical robot, characterized in that: The method comprises: Determine a three-dimensional tissue model based on analysis of acquired visual image data, laser imaging data, and tomographic image data; Analyzing the historical lesion data and the tomographic image data to determine lesion direction data; and superimposing the lesion direction data and the three-dimensional tissue model to determine a target surgical model; Predicting based on the collected tomographic image data and the acquired physiological motion data to determine predicted trajectory data; Path planning is performed based on the predicted trajectory data, the target surgical model, and preset constraints, an execution path is determined, and the surgical robot is controlled based on the execution path.

2. The method according to claim 1, characterized in that The step of analyzing the acquired visual image data, laser imaging data, and tomographic image data to determine a three-dimensional tissue model specifically includes: Modeling is performed based on the visual image data to determine a tissue geometry model; analyzing the laser imaging data to determine tissue elasticity distribution data, and superimposing the tissue elasticity distribution data with the tissue geometry model to determine a three-dimensional model; Performing feature extraction on the visual image data to determine vascular feature data; performing feature extraction on the tomographic image data to determine basement membrane feature data; and performing cross-modal registration based on the vascular feature data and the basement membrane feature data to determine a standard feature data set; wherein the standard feature data set includes the registered vascular feature data and the registered basement membrane feature data; Calculating based on the blood vessel data and a first preset formula to determine a spatial weight; fusing based on the spatial weight and the standard feature data set to determine blood vessel-basement membrane fusion atlas data; The blood vessel-basement membrane fusion atlas data is superimposed on the three-dimensional model to determine a three-dimensional tissue model.

3. The method according to claim 1, characterized in that The analyzing the historical lesion data and the tomographic image data to determine the lesion direction data specifically includes: Adjusting parameters of a preset model according to the historical lesion data to determine a lesion prediction model; The features of the tomographic image are extracted according to the lesion prediction model to determine lesion feature data; and the lesion feature data are analyzed to determine the lesion direction data.

4. The method according to claim 1, wherein The step of performing prediction based on the collected tomographic image data and the acquired physiological motion data to determine the predicted trajectory data specifically includes: Extracting data from the physiological motion data to determine respiratory phase data and blood flow pulsation data; Performing gated slicing processing based on the tomographic image data and the respiratory phase data to determine a respiratory image dataset; wherein the respiratory image dataset includes an expiratory image dataset and an inspiratory image dataset; Performing registration processing on the respiratory image dataset and the blood flow pulsation data to determine a standard image dataset, and performing vector extraction on the standard image dataset to determine a vector dataset; The vector data set is predicted and processed according to the trained prediction model to determine the predicted motion trajectory data.

5. The method according to claim 1, wherein The performing of path planning based on the predicted trajectory data, the target surgical model, and preset constraints to determine the execution path specifically includes: Discretizing the target surgical model to determine a spatiotemporal voxel map data set; analyzing the predicted motion trajectory data to determine an operation constraint area; A multi-optimization model is constructed according to the preset constraint conditions, the spatiotemporal voxel map dataset, and the operation constraint area, and the multi-optimization model is solved to determine the execution path.

6. The method according to claim 1, characterized in that The method further comprises: Collecting current tomographic data, current visual image data, and current predicted motion trajectory data according to a preset period, and obtaining current position information of the surgical robot; updating the target surgical model according to the current tomographic data to determine an updated target surgical model; and encoding the current visual image data and the current predicted motion trajectory data to determine dynamic obstacle information; Local replanning is performed based on the current position information, the updated target surgical model, and the dynamic obstacle information to determine path adjustment information; and the execution path is adjusted based on the path adjustment information and the operation constraint area to determine an adjusted execution path.

7. The method according to claim 1, characterized in that The method further comprises: Get the current vocal cord vibration frequency and current force feedback information; Comparing the current force feedback information with a preset tissue deformation model to determine current tissue information, and correcting the surgical robot according to the current tissue information; The current vocal cord frequency is compared with a preset threshold range. If the current vocal cord frequency falls within the preset threshold range, the current state of the surgical robot is maintained; otherwise, the contact pressure of the surgical robot is adjusted.

8. A control system for a throat intelligent surgical robot, characterized in that: include: The first module is used to analyze the acquired visual image data, laser imaging data, and tomographic image data to determine a three-dimensional tissue model; The second module is used to analyze the historical lesion data and the tomographic image data to determine the lesion direction data; and determining a target surgical model based on superimposing the lesion direction data and the three-dimensional tissue model; The third module is used to make predictions based on the collected tomographic image data and the acquired physiological motion data to determine predicted trajectory data; The fourth module is used to perform path planning based on the predicted trajectory data, the target surgical model and preset constraints, determine the execution path, and control the surgical robot according to the execution path.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 7 when executed by the processor.

Images