Multi-sensor fusion operation process automatic management method and device

Abstract

The invention provides an automatic operation process management method and device based on multi-sensor fusion, and relates to the technical field of automatic processes, and the method comprises the steps: constructing an instrument pose flow and operation field geometric model under a unified coordinate system through unified timestamp alignment and spatial geometry solution of multi-source sensor data; semantic segmentation and three-dimensional reprojection are combined to generate a shielding probability graph, cross-channel compensation reconstruction is carried out, and the image integrity is improved. And forming a conflict risk set through voxel field discretization and short-time prediction, and performing combined solution with standardized operation process constraints to generate a scheduling instruction and a safety time window. And finally, mapping the image, the pose and the physiological signal to a knowledge graph, and outputting consistency judgment and deviation correction suggestions in combination with a time sequence attention recognition process node. According to the method, the problem that the operation process management is not accurate enough due to the problems of view overlapping and the like of multi-source sensor data can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automated processes, in particular to a multi-sensor fusion surgical process automatic management method and device. BACKGROUND

[0002] At present, by constructing a closed-loop architecture of "perception-analysis-decision-execution", intelligent management of surgical processes is realized, that is, multi-modal data is acquired by surgical table position sensors, surgical field image acquisition devices, physiological monitoring sensors and surgical instrument tracking sensors, and after cross-sensor synchronization is realized via unified timestamp and calibration algorithm, the data is input into a fusion processing model, process node discrimination and prediction are completed by combining time-space registration, feature extraction and multi-modal fusion, risks and deviations are identified by matching with standardized surgical paths, and prompt or correction schemes are generated, and finally, the state and decision support are presented in real time in the multi-modal interaction interface and auxiliary equipment, realizing the transformation from passive recording to active monitoring and intelligent scheduling.

[0003] In the scene of multi-instrument collaborative operation, different endoscopes, catheters and energy surgical instruments will enter the patient's body at the same time, and the sensor data carried by them may have visual overlap, signal interference and operation trajectory intersection, so the multi-sensor fusion method needs to further introduce a cross-instrument perspective compensation mechanism on the basis of time sequence alignment and feature fusion, eliminate the occlusion and deviation of the perception range of each instrument through geometric registration and space mapping algorithm, and at the same time, establish a path conflict detection model to predict and collide the real-time collected multi-instrument pose trajectory, and dynamically check whether the operation conforms to the established rules in combination with the safety constraint conditions of the standardized surgical process, so that the surgical process management can not only monitor the operation sequence, but also ensure the safety and consistency of multi-instrument collaboration.

[0004] Therefore, a method is needed to solve the problem that the surgical process management is not accurate enough due to the visual overlap of multi-source sensor data and other problems. SUMMARY

[0005] The present application provides a multi-sensor fusion surgical process automatic management method and device, which can solve the problem that the surgical process management is not accurate enough due to the visual overlap of multi-source sensor data and other problems.

[0006] In a first aspect of the present application, a multi-sensor fusion surgical process automatic management method is provided, which comprises: Acquiring an endoscope image sequence, a catheter pose stream, an energy surgical instrument pose stream, a surgical field environment depth, a patient physiological signal sequence and an intraoperative speech event, and completing cross-sensor time sequence alignment by using a unified timestamp mechanism to obtain a synchronous data stream; perform spatial geometric solving on the synchronous data stream based on a hand-eye calibration matrix, imaging internal and external parameters, an endoscope end cap distortion model, and a light source intensity response curve, generate an instrument pose stream and an operating field imaging geometric model in a unified coordinate system, and output a cross-instrument transformation set; jointly solve the endoscope image sequence and the operating field imaging geometric model, extract pixel masks of an instrument knife head, a forceps end, a catheter head end, and an anatomical structure through semantic segmentation, perform three-dimensional back projection combining depth estimation and the cross-instrument transformation set, construct an occlusion probability graph and a viewable domain graph, and output a view angle compensation candidate source set; perform space-time registration on the main channel endoscope image and the view angle compensation candidate source set in a unified coordinate, replace cross-channel content for an occluded area, and perform local reconstruction with luminosity consistency and edge consistency as constraints, and output a compensated operating field image; discretize the instrument pose stream and the cross-instrument transformation set to a space-time occupancy voxel field, introduce an energy surgical instrument thermal diffusion safety radius and a catheter advancement safety radius, and generate a conflict risk set based on short-time prediction; jointly solve the conflict risk set and a safety constraint set of a standardized surgical procedure node, generate a scheduling instruction set and an executable safety time window according to an instrument function role and an operation intention; map the operating field image, the instrument pose stream, and the patient physiological signal sequence to a surgical procedure knowledge graph and input a procedure recognition network, compare a standardized path template with time sequence attention, output a current procedure node and a node confidence, and generate an instrument cooperation consistency discrimination result and a correction suggestion combining the conflict risk set and the scheduling instruction set.

[0007] In a second aspect of the present application, a multi-sensor fusion surgical procedure automatic management device is provided, which is used to perform any one of the multi-sensor fusion surgical procedure automatic management methods described above, and the device comprises an acquisition module, a processing module, and an output module, wherein: The acquisition module is used to collect an endoscope image sequence, a catheter pose stream, an energy surgical instrument pose stream, an operating field environment depth, a patient physiological signal sequence, and an intraoperative speech event, complete cross-sensor time sequence alignment using a unified timestamp mechanism, and obtain a synchronous data stream; The processing module is used to perform spatial geometric solving on the synchronous data stream based on a hand-eye calibration matrix, imaging internal and external parameters, an endoscope end cap distortion model, and a light source intensity response curve, generate an instrument pose stream and an operating field imaging geometric model in a unified coordinate system, and output a cross-instrument transformation set; The processing module is configured to jointly solve the endoscope image sequence and the surgical field imaging geometric model, extract pixel masks of an instrument knife head, a forceps end, a catheter head end and an anatomical structure through semantic segmentation, perform three-dimensional back projection combined with depth estimation and the cross-instrument transformation set, construct an occlusion probability graph and a viewable domain graph, and output a view angle compensation candidate source set; The processing module is configured to perform spatiotemporal registration on the main channel endoscope image and the view angle compensation candidate source set in a unified coordinate, replace cross-channel content for an occluded area, and perform local reconstruction with luminosity consistency and edge consistency as constraints, and output a compensated surgical field image. The processing module is configured to discretize the instrument pose stream and the cross-instrument transformation set to a spatiotemporal occupancy voxel field, introduce an energy surgical instrument thermal diffusion safety radius and a catheter advancement safety radius, and generate a conflict risk set based on short-time prediction. The processing module is configured to jointly solve the conflict risk set and a safety constraint set of a standardized surgical procedure node, generate a scheduling instruction set and an executable safety time window according to an instrument function role and an operation intention. The output module is configured to map the surgical field image, the instrument pose stream and the patient physiological signal sequence to a surgical procedure knowledge graph and input the procedure recognition network, compare a standardized path template with time sequence attention, output a current procedure node and a node confidence, and generate an instrument cooperation consistency discrimination result and a correction suggestion in combination with the conflict risk set and the scheduling instruction set.

[0008] In a third aspect of the present application, an electronic device is provided, which includes a processor, a memory, a user interface and a network interface, the memory is configured to store instructions, the user interface and the network interface are configured to communicate with other devices, and the processor is configured to execute the instructions stored in the memory to enable the electronic device to perform the method of any one of the above aspects.

[0009] In a fourth aspect of the present application, a computer readable storage medium is provided, which stores instructions, and when the instructions are executed, the method of any one of the above aspects is performed.

[0010] In summary, the one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1.The application introduces a unified timestamp mechanism and spatial geometry solving after multi-source sensor data collection, enabling different modal data to be fused and expressed in the same space-time coordinates, combining the joint solving of endoscopic images and surgical field geometry models and cross-channel perspective compensation to effectively eliminate occlusion and information loss caused by overlapping fields of view; at the same time, the use of space-time occupancy voxel field and safety radius parameters realizes the prediction and avoidance of potential conflicts, and the surgical field image, instrument pose stream and patient physiological signal are unified and mapped to the surgical process knowledge graph, and the process node is accurately identified and consistent rectification is realized through the time sequence attention comparison standardized path template, thereby ensuring that accurate and reliable surgical process management results can still be output in the presence of overlapping fields of view and multi-sensor interference.

[0011] 2.Spatial geometry solving based on hand-eye calibration matrix, imaging internal and external parameters, endoscope end cap distortion model and light source intensity response curve can realize the spatial unification and photometric normalization of instrument pose stream and endoscopic images, so that subsequent multi-modal data can be processed in the same coordinate and photometric scale, thereby significantly improving the accuracy of instrument pose and image registration and ensuring the stability of the surgical field geometry model.

[0012] 3.Combining semantic segmentation and three-dimensional back projection to generate an occlusion probability map and a viewable area map, the joint solving of endoscopic image sequences and surgical field imaging geometry models can quantify visibility and filter compensable sources when multiple instruments are operating in the surgical field and there is local occlusion, thereby improving the information complementarity between multi-channel images and significantly reducing visual loss caused by occlusion.

[0013] 4.Through cross-channel space-time registration and local reconstruction of the main channel image based on the perspective compensation candidate source set, the image content of the occluded area can be effectively restored, and the brightness smoothness and boundary continuity after image compensation can be ensured through photometric consistency and edge consistency constraints, thereby enhancing the integrity and continuous visibility of the surgical field image.

[0014] 5.By mapping the instrument pose stream and cross-instrument transformation set to the space-time occupancy voxel field and introducing the safety radius parameters of energy instruments and catheters, combined with short-term prediction to generate a conflict risk set, potential instrument collision or energy diffusion risks in the future can be identified in advance, thereby improving the proactive risk prediction and active protection capabilities of the surgical process.

[0015] 6.Combining the conflict risk set with the safety constraint set of the standardized surgical process node to jointly solve the scheduling instruction set and the executable safety time window, the dynamic coordination and resource allocation of different instrument operation intentions can be realized, and the action execution can meet the safety boundary and process specification, thereby ensuring the safety and process consistency under multi-instrument cooperation. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1is a flowchart of a multi-sensor fusion surgical procedure automatic management method disclosed by an embodiment of the present application. Figure 2 is a module schematic diagram of a multi-sensor fusion surgical procedure automatic management device disclosed by an embodiment of the present application. Figure 3 is a structural schematic diagram of an electronic device disclosed by an embodiment of the present application.

[0017] Label explanation: 201, acquisition module; 202, processing module; 203, output module; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0018] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described below in conjunction with the drawings in the embodiments of the specification. Obviously, the described embodiments are only some of the embodiments of the present application, not all.

[0019] In the description of the embodiments of the present application, the words such as "for example" or "for instance" are used to represent an example, illustration or description. Any embodiment or design scheme described as "for example" or "for instance" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words such as "for example" or "for instance" are intended to present the relevant concept in a specific manner.

[0020] In the description of the embodiments of the present application, the term "a plurality of" means two or more. For example, a plurality of systems means two or more systems, and a plurality of screen terminals means two or more screen terminals. In addition, the terms "first" and "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. The terms "include", "contain", "have" and their variants mean "include but are not limited to", unless otherwise specifically emphasized.

[0021] The existing intelligent management of surgical procedures realizes the collection, synchronization, fusion and process node identification of multi-modal data by constructing a closed-loop architecture of "perception-analysis-decision-execution", and generates risk prompts and correction schemes combined with standardized surgical paths, thereby completing the transition from passive recording to active monitoring and intelligent scheduling. In the multi-instrument collaborative operation scene, due to the visual overlap, signal interference and trajectory intersection of endoscope, catheter and energy surgical instrument sensor data, a cross-instrument perspective compensation mechanism and a path conflict detection model need to be introduced to predict and collide the multi-instrument pose trajectory, and combined with the safety constraint conditions, the consistency and compliance of the operation are dynamically checked, so as to solve the problem of inaccurate surgical process management under multi-source sensor data fusion.

[0022] The embodiment discloses a kind of surgical procedure automatic management methods of multi-sensor fusion, refer to Figure 1 , including following steps S110-S170: S110, collection endoscope image sequence, catheter pose stream, energy surgical instrument pose stream, operating field environment depth, patient physiological signal sequence and interoperative speech event, unified time stamp mechanism is used to complete cross-sensor time sequence alignment, and synchronous data stream is obtained.

[0023] The surgical procedure automatic management method of multi-sensor fusion disclosed by the embodiment of the application is applied to a server, and the server includes but is not limited to electronic devices such as mobile phones, tablet computers, wearable devices, PC (Personal Computer, personal computer) and the like, and can also be a background server running a surgical procedure automatic management method of multi-sensor fusion. The server can be realized by an independent server or a server cluster composed of multiple servers.

[0024] The endoscope image sequence is collected, the continuous video frames of the operating field are captured in real time by the endoscope imaging device, and these video frames are input as time-sequenced image sequences. The meaning of endoscope image sequence here is a set of endoscope images arranged in chronological order, which usually includes visible information such as anatomical structure, instrument knife head and forceps end in the operating field. During the collection process, the resolution, frame rate and uniformity of the endoscope image need to be ensured, and the data is transmitted to the fusion processing module through the image acquisition interface to ensure that the image sequence can be compared and fused with other sensor data in the same time domain.

[0025] Collecting catheter pose stream, using optical tracking sensors or electromagnetic tracking sensors to obtain the three-dimensional position and three-dimensional attitude of the catheter at each moment during the operation, and arranging the three-dimensional position and three-dimensional attitude of the continuous moments in time sequence to form a time sequence data stream. The meaning of catheter pose stream here is the dynamic trajectory information of the catheter in the unified coordinate system, covering translation parameters and rotation parameters, which can represent the operation process of the catheter in the body, such as advancing, retracting and rotating. During the collection process, high sensitivity and low delay of the catheter tracking sensor need to be ensured, so as to combine with the endoscope image sequence for subsequent spatial registration.

[0026] Collecting energy surgical instrument pose stream, through the end effector sensor of surgical robot or external optical tracking system, real-time obtaining the three-dimensional position and three-dimensional attitude of the energy surgical instrument in the operation process, and continuously outputting it as a time sequence pose data stream. The meaning of energy surgical instrument pose stream here is the spatial trajectory data describing the dynamic motion of energy cutting instrument, energy coagulation instrument, etc. in the operation process, including the spatial position change and attitude change of the instrument end. During the collection process, the tracking frequency and the endoscope image acquisition frequency need to be kept consistent, so as to provide accurate input for subsequent path conflict detection and safety constraint checking.

[0027] Collecting operating field environment depth, using structured light depth camera, binocular stereo imaging device or laser scanning device to obtain the depth information of each pixel point in the operating field scene, and forming a depth map sequence. The meaning of operating field environment depth here is the collection of spatial distances from the imaging plane of the anatomical surface and the instrument surface in the operating field, which is a necessary input for three-dimensional reconstruction and occlusion detection. During the collection process, the stability of the depth sensor under the endoscope illumination condition needs to be ensured, and the errors caused by light reflection and tissue absorption are corrected through real-time calibration mechanism.

[0028] Collecting patient physiological signal sequence, using multi-parameter physiological monitor to obtain physiological parameters such as electrocardiogram, blood pressure, respiratory rate, blood oxygen saturation and brain electrical activity in real time, and arranging them in time sequence to form a physiological signal sequence. The meaning of patient physiological signal sequence here is a set of multi-dimensional physiological parameters arranged in a unified time domain, which is used to reflect the general state of the patient during the operation. During the collection process, the continuity and low delay of physiological signal monitoring need to be ensured, and the electrical noise caused by surgical operation is removed through anti-interference filtering algorithm, so as to provide reliable input for risk identification in process management.

[0029] The intraoperative speech event is collected by a speech collection device to collect the language communication between the surgeon and the assistant in real time during the operation, and the speech stream is converted into a semantic event sequence by a speech recognition algorithm. The intraoperative speech event here means that the voice content after recognition and time marking, such as instructions, prompts, responses and other key information, can directly reflect the operation intention of the operator. During the collection process, the robustness of speech recognition needs to be ensured under the noise of the operating room environment, and keyword recognition and context analysis are used to ensure that the recognition result can be used for operation intention judgment and process node labeling.

[0030] By embedding a unified time synchronization protocol in all sensor inputs, the laparoscope image sequence, catheter pose stream, energy surgical instrument pose stream, operating field environment depth, patient physiological signal sequence and intraoperative speech event are uniformly marked as multi-modal data streams with the same reference time. The unified timestamp mechanism here means that the same reference time is provided for different data sources to ensure that different types of data can maintain time consistency within milliseconds. The aligned data set is called a synchronized data stream, which represents a multi-modal input sequence arranged in a unified time coordinate, providing a unified data basis for subsequent spatial geometry solving, semantic segmentation, path conflict detection and process node recognition.

[0031] In S120, based on the hand-eye calibration matrix, the intra- and extra- imaging parameters, the laparoscope end cap distortion model and the light source intensity response curve, spatial geometry solving is performed on the synchronized data stream to generate an instrument pose stream and an operating field imaging geometry model in a unified coordinate system, and a cross-instrument transformation set is output.

[0032] In a possible implementation, based on the hand-eye calibration matrix, the intra- and extra- imaging parameters, the laparoscope end cap distortion model and the light source intensity response curve, spatial geometry solving is performed on the synchronized data stream to generate an instrument pose stream and an operating field imaging geometry model in a unified coordinate system, and a cross-instrument transformation set is output, specifically including: projecting the instrument pose stream to the laparoscope imaging coordinate system and mapping it to the unified reference coordinate system by using the hand-eye calibration matrix; performing geometric solving on the laparoscope image sequence by using the intra- and extra- imaging parameters to establish a correspondence between the image coordinates and the physical coordinates; performing distortion correction on the laparoscope image sequence by using the laparoscope end cap distortion model to obtain corrected images with consistent edge linearity and scale; performing photometric correction on the laparoscope image sequence by using the light source intensity response curve to obtain brightness-normalized image intensity distribution; jointly solving the instrument pose stream and the laparoscope image sequence processed by the hand-eye calibration matrix, the intra- and extra- imaging parameters, the laparoscope end cap distortion model and the light source intensity response curve in the unified coordinate system to generate an operating field imaging geometry model and output a cross-instrument transformation set.

[0033] Specifically, when mapping the instrument pose stream using the hand-eye calibration matrix, a homogeneous transformation chain is established in the unified coordinate system to transform the time-series pose of the instrument end in the end-effector coordinate system to the camera coordinate system, and then to the unified reference coordinate system, to obtain the instrument pose stream that can be directly connected with the image geometry solution. The following transformation relationship is used to complete the mapping: wherein represents the pose matrix of the instrument end in the unified coordinate system at time t, represents the external parameter homogeneous matrix of the camera coordinate system to the unified coordinate system, represents the hand-eye calibration matrix and gives the homogeneous transformation of the camera coordinate system to the end-effector coordinate system, represents the homogeneous transformation of the end-effector to the instrument end at time t. By applying this chain transformation to the instrument pose stream frame by frame, the instrument pose stream that is time-continuous and alignable with the image in the unified coordinate system is obtained, and consistent coordinate semantics are provided for subsequent image projection and three-dimensional back-projection.

[0034] When performing geometric solution on the laparoscope image sequence using the imaging internal and external parameters, a pinhole imaging model is used to establish the bidirectional mapping between the pixel coordinates and the physical three-dimensional points, and the corresponding relationship between the pixels and the three-dimensional points in the unified coordinate system is completed. The following forward projection model is used to describe the pixel generation process: wherein represents the pixel coordinates, represents the scale factor and is proportional to the depth of the point, represents the imaging internal parameter matrix and contains the focal length and the principal point coordinates, and represents the imaging external parameter and gives the rotation and translation of the unified coordinate system to the camera coordinate system, represents the three-dimensional point coordinates in the unified coordinate system. The back-projection is performed under the condition of having depth to recover the three-dimensional coordinates: wherein represents the three-dimensional coordinates in the camera coordinate system, and the bidirectional correspondence between the pixels and the three-dimensional points is completed by frame-by-frame solution, laying a geometric foundation for the unified fusion after distortion correction and photometric correction.

[0035] When performing distortion correction on the laparoscope image sequence using the laparoscope end cap distortion model, the pixels are first normalized to the imaging plane, and then the joint model of radial distortion and tangential distortion is applied to complete the pixel-level correction, and the corrected image with consistent edge straightness and proportion is output. The following distortion model is used to describe the mapping from the normalized coordinates to the distorted coordinates: wherein, with denotes the normalized distortion-free planar coordinate via imaging intrinsic, with denotes the distortion planar coordinate, denotes the radial distance to the optical axis, , , denotes the radial distortion coefficient, with denotes the tangential distortion coefficient, the inverse mapping from distortion coordinate to distortion-free coordinate is obtained by numerical solving, and resampling is performed on all frame pixels to restore straight line geometry and eliminate edge stretching and bending caused by end cap.

[0036] When the light source intensity response curve is used to correct the endoscopic image sequence, a luminance response function is established with respect to the incident angle and working distance, and the sensor response function is combined to complete the radiation calibration and corner vignetting compensation, and the luminance normalized image intensity distribution is output. A multiplicative and additive combined photometric model is used for normalization: wherein, denotes the normalized pixel intensity, denotes the original pixel intensity, with denotes the linear response gain and bias of the imaging sensor, denotes the light source intensity response curve and depicts the attenuation characteristics of illumination with respect to the incident angle and working distance , denotes the lens vignetting function and varies with the normalized radius , the parameters of which are fitted through calibration experiments and , and each frame of image is corrected pixel by pixel, so that regions of different depths and different incident angles obtain comparable luminance scales, and the robustness of subsequent feature matching and segmentation is significantly improved.

[0037] When the instrument pose stream processed through the hand-eye calibration matrix and the imaging intrinsic and extrinsic and the end cap distortion model of the endoscope and the light source intensity response curve and the endoscopic image sequence are jointly calculated in a unified coordinate system, a joint optimization method is used to simultaneously refine the extrinsic and instrument end trajectory, and an operating field imaging geometry model is generated which can uniformly describe the camera imaging geometry and the three-dimensional shape of the operating field. The cross-instrument transformation set is output to depict the stable relative relationship between multiple instruments. The reprojection residual and the pose prior are used as the target items to adjust and minimize the following cost function: wherein represents the distortion-free pixel position of the i-th feature at time t, represents the projection function that generates pixel positions from three-dimensional points, represents the coordinate of the i-th three-dimensional point in the unified coordinate system, represents the instrument pose increment at adjacent time and is subject to motion smoothing priors, represents the prior weight, represents the covariance weighted matrix of the pose increment. After optimization convergence, the surgical field imaging geometry model is obtained and contains camera extrinsic parameters in the unified coordinate system, corrected intrinsic parameters, key three-dimensional point cloud and visibility index. Based on the same optimization result, the inter-instrument transformation set is calculated and the relative pose relationship between different instrument end is given by the following formula: wherein represents the homogeneous transformation of instrument B relative to instrument A at time t and is used to describe the mutual position and attitude constraints of multiple instruments in the unified coordinate system. The surgical field imaging geometry model and the inter-instrument transformation set jointly constitute the geometric and motion priors for downstream occlusion estimation, view compensation and path conflict detection, and maintain consistent coordinate and photometric semantics in the whole link during the subsequent process.

[0038] S130, jointly solve the laparoscope image sequence and the surgical field imaging geometry model, extract the pixel masks of the instrument knife head, clamp end, catheter head and anatomical structure through semantic segmentation, combine depth estimation and inter-instrument transformation set for three-dimensional back projection, construct occlusion probability map and viewable domain map, and output view compensation candidate source set.

[0039] In a possible implementation, the endoscope image sequence is jointly solved with the surgical field imaging geometry model, the instrument tool tip, forceps end, catheter tip and anatomical structure pixel masks are extracted through semantic segmentation, three-dimensional back projection is performed combining depth estimation and inter-instrument transformation set, occlusion probability map and viewable domain map are constructed, and a view angle compensation candidate source set is output, specifically including: performing semantic segmentation on the endoscope image sequence in a unified coordinate system, extracting instrument tool tip pixel mask, forceps end pixel mask, catheter tip pixel mask and anatomical structure pixel mask, and performing three-dimensional back projection combining depth estimation parameters in the surgical field imaging geometry model and the inter-instrument transformation set to generate continuous surgical field point cloud and instrument end three-dimensional pose sequence; based on the continuous surgical field point cloud and the instrument end three-dimensional pose sequence, an occlusion probability map sequence is constructed, and a viewable domain map set is generated combining the occlusion probability map sequence, the viewable domain map set is used to quantify the potential visible coverage of the main channel endoscope invisible area on each auxiliary channel endoscope; based on the viewable domain map set and the occlusion probability map sequence, the candidate sources are sorted according to the coverage area ratio, the baseline angle, the luminosity consistency, the feature matching confidence and the time proximity, and the auxiliary channel information meeting the cross-channel registration accessibility and the structural consistency is screened out, and a view angle compensation candidate source set composed of auxiliary channel identifier, coverage area index, expected registration error, expected luminosity error and time stamp range is output.

[0040] Specifically, when performing semantic segmentation on the endoscope image sequence in a unified coordinate system, first, the endoscope image sequence is preprocessed, including luminosity equalization, noise filtering and edge enhancement, to ensure segmentation accuracy. Then the preprocessed endoscope image sequence is input into the depth segmentation network trained by the surgical scene data set, and the network outputs multi-class pixel masks in a unified coordinate system. The pixel mask refers to the class label result corresponding to each pixel in the image, wherein the instrument tool tip pixel mask is used to represent the area of the cutting or coagulation tool tip, the forceps end pixel mask is used to represent the area of the mechanical operating forceps end, the catheter tip pixel mask is used to represent the catheter port area entering the patient's body, and the anatomical structure pixel mask is used to represent the blood vessel, tissue and organ boundary in the surgical field. Combining the depth estimation parameters contained in the surgical field imaging geometry model, the pixels in each mask area are mapped to three-dimensional points in the unified coordinate system through back projection operation, to obtain continuous surgical field point cloud. At the same time, the pixels related to the instrument end in the point cloud are corresponded to specific instrument poses by using the inter-instrument transformation set, thereby forming an instrument end three-dimensional pose sequence. The surgical field point cloud describes the three-dimensional structure of the entire surgical field, and the instrument end three-dimensional pose sequence is used to depict the position and attitude change of the instrument operating end in space.

[0041] When constructing the sequence of occlusion probability maps based on the continuous intraoperative point cloud and the sequence of 3D pose of the instrument tip, the center of the endoscope imaging is taken as the viewpoint, and the rays are emitted to the point cloud for visibility determination in the unified coordinate system. For each pixel direction, if there are multiple depth points in the point cloud, the point with the smallest depth is selected as the visible point, and the remaining points are marked as potential occlusion points. The sequence of 3D pose of the instrument tip is further combined to statistically analyze the occlusion proportion of different instrument components to the anatomical structure region. By accumulating the occlusion frequency and occlusion confidence of different pixels over multiple time slices, a sequence of occlusion probability maps is generated, where the occlusion probability map represents the likelihood of each pixel being occluded in the time dimension. Using the sequence of occlusion probability maps and the visibility distribution of the intraoperative point cloud, a set of viewable domain maps is generated. The set of viewable domain maps is used to describe the coverage of the invisible region in the main channel endoscope image on other secondary channel endoscopes under multi-channel endoscope conditions, including coverage area proportion, coverage boundary continuity, and visible texture completeness.

[0042] When sorting the candidate sources based on the set of viewable domain maps and the sequence of occlusion probability maps, the coverage area proportion is taken as the first priority indicator, which represents the percentage of the invisible region of the main channel endoscope that can be compensated by the secondary channel endoscope. The baseline angle is taken as the second priority indicator, which refers to the optical axis angle between the main channel and the secondary channel endoscope. Too small an angle will result in information redundancy, and too large an angle may cause geometric distortion. The photometric consistency is taken as the third priority indicator, which represents the similarity in color and brightness distribution between different channel images. The higher the photometric consistency, the better the fusion effect. The feature matching confidence is taken as the fourth priority indicator, which refers to the reliability of cross-channel key point matching. High confidence indicates more stable cross-channel registration. The time proximity is taken as the fifth priority indicator, which refers to the proximity of the time stamp between the secondary channel image frame and the main channel image frame. The closer the proximity, the better the dynamic scene consistency. According to these indicators, the candidate sources are weighted and sorted, and the secondary channels with large errors or inconsistent structures in the cross-channel registration process are removed, and the secondary channel information that meets the requirements is selected. The final output of the view angle compensation candidate source set consists of secondary channel identification, coverage area index, expected registration error, expected photometric error, and time stamp range, providing a complete input set for subsequent cross-channel registration and content replacement of the occluded region.

[0043] S140, the main channel endoscope image and the view angle compensation candidate source set are spatio-temporally registered in the unified coordinate, the occluded region is replaced with the content from the secondary channel, and local reconstruction is performed with photometric consistency and edge consistency as constraints, outputting the compensated intraoperative image.

[0044] In a possible implementation, the main channel endoscope image and the perspective compensation candidate source set are spatio-temporally registered under a unified coordinate, the occluded area is replaced by cross-channel content, and local reconstruction is performed with luminosity consistency and edge consistency as constraints to output the compensated surgical field image, and specifically includes: determining candidate input of the main channel endoscope image by using the perspective compensation candidate source set, the candidate input being a secondary channel endoscope image adjacent in time to the main channel endoscope image and having coverage and visibility satisfying conditions; performing cross-channel spatio-temporal registration under a unified coordinate based on a surgical field imaging geometric model and imaging parameters and geometric constraints provided by the cross-instrument transformation set to output a first compensation image; after the spatio-temporal registration is completed, performing cross-channel content replacement on pixels in the occluded area of the main channel endoscope image, assigning weights to pixels at corresponding positions in the secondary channel endoscope image according to coverage, re-projection error and temporal proximity, and replacing the occluded area in the main channel endoscope image in a weighted fusion manner to output a second compensation image; after the cross-channel content replacement, ensuring smooth transition of local brightness and texture with luminosity consistency constraint, ensuring continuity of instrument boundaries and anatomical structure boundaries with edge consistency constraint, and performing local reconstruction in the occluded area to generate a third compensation image with luminosity balance and edge continuity; and taking the first compensation image and the second compensation image as the surgical field image.

[0045] Specifically, when determining the candidate input of the main channel endoscope image by using the perspective compensation candidate source set, first, compare the acquisition time of the main channel endoscope image with the timestamp range of the secondary channel endoscope images in the perspective compensation candidate source set under a unified timestamp mechanism, and filter out candidate frames with a time difference below a threshold. Then, judge the coverage of the secondary channel endoscope images in the invisible area of the main channel endoscope image based on the viewable domain graph set, the coverage being the area proportion of the secondary channel image that can compensate for the occluded area of the main channel after geometric projection, and only images with coverage exceeding a preset threshold are included in the candidate set. Further, the visibility confidence in the occlusion probability graph is combined to ensure that the selected secondary channel image has stable visibility in the occluded area of the main channel. Finally, the candidate secondary channel image satisfying the time proximity, coverage requirement and visibility requirement is obtained as the input for subsequent cross-channel registration.

[0046] When performing cross-channel spatio-temporal registration in a unified coordinate based on the imaging geometry model of the surgical field and the imaging parameters and geometric constraints provided by the inter-device transformation set, first, the imaging intrinsic matrix and the imaging extrinsic matrix in the imaging geometry model of the surgical field are used to perform projection transformation on the candidate secondary channel image, which is mapped to the unified coordinate system. Then, combined with the homogeneous transformation relationship in the inter-device transformation set, the optical center position, the imaging plane direction and the device end pose of the primary channel and the secondary channel are spatially aligned. In this process, photometric consistency and geometric residual minimization are used as the optimization objective function, and the registration parameters are adjusted through iterative optimization to reduce the reprojection error. The reprojection error here represents the difference between the pixel position predicted by the geometric model and the actual pixel position of the secondary channel. After optimization, the first compensation image is output, which is the secondary channel image that is spatio-temporally aligned with the primary channel image in the unified coordinate system.

[0047] After completing the spatio-temporal registration, when performing cross-channel content replacement for the pixels in the occluded area of the primary channel endoscope image, first, the occluded area in the primary channel endoscope image is locked using the occlusion probability map, and this area is taken as the candidate replacement target. For each occluded area pixel, the pixel value at the corresponding position in the aligned secondary channel image is obtained, and the fusion weight is calculated according to the coverage, reprojection error and temporal proximity. The coverage weight is used to measure the visible coverage proportion of the secondary channel image to the target area, the reprojection error weight is used to measure the registration accuracy, and the temporal proximity weight is used to measure the consistency of the secondary channel image acquisition time and the primary channel image acquisition time. The final fusion pixel value is obtained by weighted averaging, and the expression is: wherein represents the fusion value of pixel p, represents the pixel value at the corresponding position of the kth secondary channel after registration, represents the fusion weight. By pixel-by-pixel weighted replacement of the occluded area, the second compensation image is obtained, which retains the overall consistency of the primary channel image in structure while repairing the missing caused by occlusion.

[0048] After cross-channel content replacement, when performing local reconstruction with photometric consistency constraint and edge consistency constraint, first, a photometric smoothness model is established in the occluded area to ensure that the primary channel image and the secondary channel image maintain continuous transition in brightness and color. Photometric consistency constraint is realized by minimizing the local pixel brightness difference, so that the replacement area and the surrounding non-occluded area have no obvious mutation in brightness distribution. The edge consistency constraint is realized by maintaining the continuity of the image gradient, so that the device edge and the anatomical structure boundary remain clear and coherent after replacement. For this purpose, a joint reconstruction objective function is constructed: wherein represents the pixel value after local reconstruction, This represents the pixel value after weighted fusion. This represents the pixel value of the main channel image in the adjacent area. Represents the gradient of the image. and To balance the parameters, the optimized third-compensation image maintains a natural transition and boundary integrity in terms of brightness and edge structure.

[0049] Finally, the first compensated image and the second compensated image are combined to ensure that the main channel image and the sub-channel image are aligned in time and space under a unified coordinate system, so as to achieve photometric smoothing and continuous edge repair of the local occluded area. The result is output as a compensated surgical field image for subsequent path conflict detection and surgical procedure consistency judgment.

[0050] S150 discretizes the instrument pose flow and cross-instrument transformation set into a spatiotemporal occupancy voxel field, introduces the thermal diffusion safety radius of energy surgical instruments and the catheter advancement safety radius, and generates a conflict risk set based on short-time prediction.

[0051] In one possible implementation, the instrument pose flow and cross-instrument transformation set are discretized into a spatiotemporal occupied voxel field. The thermal diffusion safety radius of the energy surgical instrument and the catheter advancement safety radius are introduced. A conflict risk set is generated based on short-time prediction. Specifically, this includes: discretizing the instrument pose flow and cross-instrument transformation set in a unified coordinate system to map the instrument position and attitude in consecutive time slices into a spatiotemporal occupied voxel field; introducing the thermal diffusion safety radius parameter of the energy surgical instrument into the spatiotemporal occupied voxel field to form a thermal risk region for the energy surgical instrument, and introducing the catheter advancement safety radius parameter to form a catheter advancement risk region, thereby generating an extended spatiotemporal occupied voxel field; performing short-time prediction based on the motion trajectory of the instrument pose flow within consecutive time slices to obtain the predicted pose trajectory for several future time slices, and mapping the predicted pose trajectory to the extended spatiotemporal occupied voxel field to form a predicted occupied region; performing spatial intersection detection within the predicted occupied region; if spatial overlap is detected between the thermal risk region of the energy surgical instrument, the catheter advancement risk region, or other extended occupied regions of the instrument, a conflict risk index is generated; and the conflict risk index is aggregated to output a conflict risk set.

[0052] Specifically, when discretizing the instrument pose stream and the inter-instrument transformation set in a unified coordinate system, first, the three-dimensional position and three-dimensional attitude of each time in the instrument pose stream are represented in the form of a homogeneous transformation matrix, and all instrument end coordinates are unified to the same reference coordinate system through the inter-instrument transformation set. Then, the three-dimensional space is divided into cubic units of fixed size, i.e., voxel units, and the voxel units occupied by the instrument end position and attitude in the continuous time slice are labeled. The spatiotemporal occupancy voxel field generated by this process not only reflects the occupancy of the instrument in space, but also retains the dynamic trajectory of the instrument in the time dimension. The spatiotemporal occupancy voxel field here represents a four-dimensional data structure, where three dimensions are spatial coordinates and the other dimension is the time axis.

[0053] When introducing the energy surgical instrument heat diffusion safety radius parameter and the catheter advancement safety radius parameter into the spatiotemporal occupancy voxel field, first, a safety buffer zone is expanded around each instrument end voxel unit that has been labeled. For energy surgical instruments, based on their heat diffusion characteristics, all voxel units within the radius range are labeled as energy surgical instrument heat risk areas using the heat diffusion safety radius parameter, thereby reflecting the range of tissues that may be affected by the instrument when using energy. For catheters, the advancement safety radius parameter is used to expand around the catheter end position voxels to form a catheter advancement risk area, reflecting the potential risk of the catheter to the surrounding tissues during advancement in the body. After this process, the spatiotemporal occupancy voxel field is expanded into an extended spatiotemporal occupancy voxel field containing safety constraints, ensuring that subsequent prediction and risk discrimination can reflect the operation safety boundary.

[0054] When performing short-time prediction based on the motion trajectory of the instrument pose stream in the continuous time slice, first, the displacement vector and attitude change of the instrument end in the near time slice are extracted, and a short-time prediction model is established based on the estimation of velocity and acceleration. The prediction model obtains the predicted pose trajectory in the future time slice through kinematic extrapolation, which here represents the three-dimensional position and three-dimensional attitude that the instrument end may appear in the future time interval. Then, these predicted pose trajectories are projected into the extended spatiotemporal occupancy voxel field, and the voxel units covered by the predicted trajectory are labeled to form the predicted occupancy area. The predicted occupancy area is an estimate of the future spatial occupancy possibility, providing time foresight for conflict risk detection.

[0055] In the spatial intersection detection within the predicted occupancy region, the energy procedure instrument thermal risk region, the catheter advancement risk region, and the extended occupancy region of other instruments are voxel-by-voxel compared. If the extended occupancy regions of different instruments are detected to overlap in the same time slice, it means that a conflict may occur in the future. A conflict risk indicator is thus generated, which includes the instrument identification of the conflicting instruments, the time interval of the conflict, the spatial voxel set of the conflict, the conflict severity level, and the conflict probability. The conflict probability can be estimated based on the deviation statistics of the predicted trajectory and the historical trajectory, and the conflict severity level is divided according to the number of overlapping voxels and the position sensitivity of the conflict region.

[0056] In the collection processing of the conflict risk indicator, the conflict risk indicators of all predicted time slices and all instrument pairs are merged to form a conflict risk set. The conflict risk set here represents a structured set that contains the complete description of all potential conflicts of multiple instruments in the future time, including instrument identification, conflict time interval, spatial overlap range, risk level, and risk probability. The conflict risk set is directly input to the subsequent conflict resolution and scheduling module, and is used to generate scheduling instructions and safety time windows in real time during the operation, so as to ensure the operation safety and consistency of multi-instrument cooperation.

[0057] S160, the conflict risk set and the safety constraint set of the standardized surgical procedure node are jointly solved to generate a set of scheduling instructions and executable safety time windows according to the instrument function role and operation intention.

[0058] In a possible implementation, the conflict risk set is jointly solved with the safety constraint set of the standardized surgical procedure node, the scheduling instruction set and the executable safety time window are generated according to the instrument function role and the operation intention, and specifically, in the joint solving stage, the instrument identification, the time interval, the space range, the risk level and the risk probability in the conflict risk set are time axis aligned and identification consistent with the planned time, the allowed operation set and the disabled operation set in the standardized surgical procedure node, and the conflict risk is taken as a conflict constraint in the unified optimization domain; the safety constraint is comprehensively solved as a hard constraint and a soft constraint, and a joint solving result containing constraint satisfaction label and residual risk label is output; in the scheduling generation stage, the corresponding action elements are called in the action template library according to the joint solving result on the basis of the instrument function role and the operation intention, the action elements include speed upper limit, acceleration upper limit, path fine tuning amplitude, energy threshold, view priority, waiting condition, cancellation condition and fallback action, and the action elements are conflict mutually exclusive checked and execution order arranged according to the procedure node priority, and a structured scheduling instruction set is generated; in the safety time window generation stage, the constraint satisfaction evaluation provided by the joint solving result is taken as the basis, the continuous time slice meeting all hard safety boundaries and key soft limits is determined in the space-time occupation voxel field, and a warning buffer zone and a monitoring variable threshold are set at the boundary of each continuous time slice, forming an executable safety time window with time stamp, instrument identification, action parameter, trigger condition, cancellation condition and fallback action, and the executable safety time window and the scheduling instruction set are one-to-one corresponding output.

[0059] Specifically, in the joint solving stage, the time axis alignment and the identification consistency processing are first completed. The instrument identification and the time interval and the space range and the risk level and the risk probability are mapped to the planned time and the allowed operation set and the disabled operation set of the standardized surgical procedure node with a unified time reference. The conflict risk is taken as a conflict constraint into solving by establishing a unified optimization domain with a rolling time window, which is represented as , the instrument set is represented as , the action selection variable of each instrument at each time slice is represented as , the predicted occupation voxel overlapping volume is represented as , the corresponding risk probability is represented as . The conflict constraint residual is represented as: Wherein represents the acceptable minimum safety margin. If , it indicates that the conflict constraint is met within the time slice. The aligned constraint candidate set is obtained by simultaneously constraining all instrument pairs and all time slices, and is taken into comprehensive solving, and in the formula, represents the Boolean indication of any selected action of the instrument at the time slice, represents the volume of spatially overlapping voxels predicted to be occupied, represents the probability of occurrence given by the conflict risk set, is the penalty residual for conflict constraints, is the safety margin threshold.

[0060] The safety constraints are solved synthetically as hard constraints and soft constraints with a penalized constraint optimization model. The hard constraint set is denoted as . The soft constraint set is denoted as . The objective function is denoted as: where represents the positive part operator, represents the soft constraint penalty weight. The hard constraints cover the minimum inter- instrument spacing and the upper limit of energy surgical instrument thermal load and the upper limit of catheter pushing rate and the visual angle occupation mutual exclusion and the procedure node order and so on. The soft constraints cover the surgical rhythm deviation and the visual field comfort and the energy margin and the instrument motion smoothness and so on. The solving output is the constraint satisfaction label and the residual risk label. The constraint satisfaction label indicates the satisfaction state of each hard constraint and soft constraint at each time slice. The residual risk label quantifies the probabilistic risk size that still exists under the premise of satisfying the hard constraints.

[0061] In the scheduling generation phase, the structured scheduling instruction set is formed based on the joint solving result and according to the instrument function role and operation intention in the action template library. The instrument function role priority is denoted as . The operation intention priority is denoted as . The residual risk cost is denoted as . The resource contention cost is denoted as . The score function is established for the candidate action: where , , and are the normalized weights. The executable action is selected from high to low based on the score, and the conflict mutual exclusion check and the execution order arrangement are performed. The speed upper limit and the acceleration upper limit and the path fine-tuning amplitude and the energy threshold and the visual angle priority and the waiting condition and the cancellation condition and the backtracking action are written in each instruction entry, so as to generate the scheduling instruction set that can be directly executed by the robot control interface and the intraoperative interaction interface. In the formula, represents the responsibility level of the instrument at the current procedure node, represents the procedure importance of the operation intention, is the residual risk label from the joint solving, The competition degree of the view angle, path, and energy resources in the time slice is measured.

[0062] In the safety time window generation stage, the constraint satisfaction evaluation provided by the joint solution result is taken as the basis, and a set of continuous time slices that satisfy all hard safety boundaries and key soft constraints are determined in the space-time occupancy voxel field. The action feasibility indicator function is represented as: wherein is the indicator function, is the soft constraint tolerance margin, the maximum continuous merging of time slices that satisfy the indicator is one, to obtain the executable safety time window, and a time buffer band is added to both ends of each window . Monitoring variable thresholds are set for thermal load, propulsion rate, minimum distance, and view angle occupancy to trigger secondary strategies when approaching the thresholds. The timestamp, instrument identification, action parameters, trigger conditions, cancellation conditions, and fallback actions are written in each window and output one-to-one with the scheduling instruction set. In the formula represents the hard constraint function, represents the soft constraint function, represents the soft constraint acceptable upper limit, represents the warning buffer band length.

[0063] S170, the surgical field image, instrument pose stream, and patient physiological signal sequence are mapped to the surgical procedure knowledge graph and input into the procedure recognition network, the current procedure node and node confidence are output by comparing the standardized path template with time sequence attention, the instrument cooperation consistency discrimination result and correction suggestion are generated by combining the conflict risk set and the scheduling instruction set.

[0064] In a possible implementation, the surgical field image, the instrument pose stream, and the patient physiological signal sequence are mapped to the surgical procedure knowledge graph and input into a procedure recognition network to compare the standardized path template with time attention, output the current procedure node and node confidence, and generate the instrument cooperation consistency discrimination result and the correction suggestion in combination with the conflict risk set and the scheduling instruction set. Specifically, the surgical field image is encoded into a multi-modal feature vector containing anatomical structure boundaries, instrument position distribution, and luminosity features, and is mapped to the image semantic node in the surgical procedure knowledge graph. The instrument pose stream is expressed as a continuous three-dimensional position and three-dimensional attitude sequence in a unified coordinate, and is mapped to the instrument operation type node in the surgical procedure knowledge graph. The patient physiological signal sequence is normalized and synchronized with the timestamp, and is then mapped to the patient state type node in the surgical procedure knowledge graph. The mapping results in the surgical procedure knowledge graph are aggregated in terms of semantic relationship, and the continuity of the instrument pose stream and the patient physiological signal sequence is maintained by using a time weight, to output a time sequence embedding containing procedure features. The time sequence embedding is compared with the standardized path template by using a time attention mechanism, the similarity between the time sequence embedding and the standardized path template is calculated for each time slice, the attention weight of each time slice is assigned according to the similarity, and the current procedure node and node confidence are output. The current procedure node and node confidence are comprehensively discriminated in combination with the conflict risk set and the scheduling instruction set. When the node confidence is inconsistent with the conflict risk set or the scheduling instruction set, the instrument cooperation consistency discrimination result is inconsistent, and the correction suggestion is output as feedback information.

[0065] Specifically, when the surgical field image is encoded into a multi-modal feature vector containing anatomical structure boundaries, instrument position distribution, and luminosity features, the surgical field image is first subjected to luminosity equalization, noise suppression, and edge enhancement to make the texture and boundary information prominent and keep the brightness scale uniform. Subsequently, a multi-branch feature extraction network trained on medical scene data is used to simultaneously extract anatomical structure boundaries, instrument position distribution, and luminosity features. The three types of features are aligned in the channel dimension and spliced into a multi-modal feature vector in a unified coordinate. The multi-modal feature vector refers to the joint expression of geometry, appearance, and intensity of the same image frame within the same timestamp. The vector is written into the image semantic node in the surgical procedure knowledge graph according to the image space coordinate, and the spatial adjacency relationship between the anatomical structure node and the instrument node and the mapping relationship with the luminosity equalization parameter are recorded in the form of edges, so that the geometric semantics and luminosity semantics can be read simultaneously in the graph structure during subsequent retrieval.

[0066] When the instrument pose stream is represented as a sequence of continuous three-dimensional positions and three-dimensional poses in a unified coordinate, the three-dimensional position and three-dimensional pose of the instrument end at each timestamp are time-synchronized and arranged in time sequence to form a pose sequence, and the pose sequence retains the time sequence continuity of translation and rotation in the form of a homogeneous transformation chain. The pose sequence is written into the instrument operation type node in the surgical procedure knowledge graph, and the spatiotemporal alignment relationship with the image semantic node, the safe distance relationship with the anatomical structure node, and the load relationship with the energy constraint node are recorded according to the edge. The instrument operation type node is used to identify the operation categories such as cutting, coagulation, traction and positioning, and to ensure that the actions of the same instrument at different times can be uniformly interpreted by the procedure recognition network.

[0067] When the patient physiological signal sequence is normalized and synchronized with the timestamp and mapped to the patient state type node in the surgical procedure knowledge graph, the dimensions of the electrocardiogram, blood pressure, respiratory rate, blood oxygen saturation and other bedside monitoring signals are unified, outliers are removed, and missing values are completed, and resampled to a frequency consistent with the image frame rate and pose sampling rate on the common time axis. The patient state type node is used to represent state labels such as stable, stress, increased risk of blood loss, and insufficient sedation, and is connected to the risk factors of the procedure node through the edge, so that the procedure recognition network can refer to the dynamic constraints on the physiological side when judging the surgical link.

[0068] When the mapping results in the surgical procedure knowledge graph are performed semantic relationship aggregation and the continuity of the instrument pose stream and the patient physiological signal sequence is maintained by using a time weight, the features of the image semantic node, the instrument operation type node and the patient state type node are aggregated on the graph in a message passing manner, and the semantic relationship aggregation means propagating and weighting the node features along the edge on the graph structure to form a semantically consistent representation. Subsequently, a time weight is introduced to the aggregated time sequence, the time weight means a decay or enhancement coefficient applied to the feature according to the correlation of adjacent time points, which is used to emphasize the observations of the near time points and suppress the interference of the distant time points. Finally, a time sequence embedding containing procedure features is obtained, the time sequence embedding means a vector sequence of multi-modal features after graph structure encoding and time continuity modeling on a unified time axis, which is directly input as a subsequent template comparison.

[0069] When the time sequence attention mechanism is used to compare the standardized path template, the time sequence embedding and the standardized path template are aligned on the time axis to form two equal-length time sequences, and the semantic similarity is calculated for each time slice, the semantic similarity means the matching degree of the current observation and the template node in the feature space. The time sequence attention mechanism assigns attention weights to each time slice according to the similarity, and the higher the attention weight, the greater the contribution of the time slice to the current node determination. The weighted and summarized results are input into the discriminative head to obtain the current procedure node and the node confidence, the node confidence means the probabilistic measure of the correctness of the determined node by the procedure recognition network, and the higher the value, the more reliable the determination.

[0070] When the current flow node is combined with the node confidence, the conflict risk set and the scheduling instruction set are comprehensively judged. First, the current flow node is associated with the conflict risk entry and the corresponding scheduling instruction entry in the same time axis within the corresponding time window. The conflict risk set provides information on potential spatial overlap and safety boundary approximation. The scheduling instruction set gives the arranged action target and execution condition. The comprehensive judgment takes the node confidence as the dominant index and takes the conflict risk probability and the remaining safety margin as the constraint. If it is found that the node confidence is inconsistent with the conflict risk or the scheduling instruction, the generator generates a device cooperation consistency judgment result as inconsistent, and outputs a correction suggestion as feedback information. The correction suggestion includes executable entries such as action delay, path fine-tuning, view angle giving, energy derating, and priority adjustment, and is accompanied by trigger conditions, cancellation conditions, and fallback actions, so that the execution end can immediately land the correction under the same coordinate semantics and the same time semantics.

[0071] The embodiment also discloses a multi-sensor fusion surgical procedure automatic management device, referring to Figure 2 , comprising an acquisition module 201, a processing module 202, and an output module 203. The device is used to execute any one of the above multi-sensor fusion surgical procedure automatic management methods, wherein: The acquisition module 201 is used to collect a laparoscope image sequence, a catheter pose stream, an energy surgical instrument pose stream, an operating field environment depth, a patient physiological signal sequence, and an intraoperative speech event. A unified timestamp mechanism is used to complete cross-sensor time sequence alignment to obtain a synchronous data stream.

[0072] The processing module 202 is used to execute spatial geometric solving on the synchronous data stream based on a hand-eye calibration matrix, imaging internal and external parameters, a laparoscope end cap distortion model, and a light source intensity response curve, to generate an instrument pose stream and an operating field imaging geometric model in a unified coordinate system, and to output a cross-instrument transformation set.

[0073] The processing module 202 is used to jointly solve the laparoscope image sequence and the operating field imaging geometric model, extract pixel masks of an instrument knife head, a forceps end, a catheter head end, and an anatomical structure through semantic segmentation, perform three-dimensional back projection combined with depth estimation and the cross-instrument transformation set, construct an occlusion probability graph and a viewable domain graph, and output a view angle compensation candidate source set.

[0074] The processing module 202 is used to perform space-time registration on the main channel laparoscope image and the view angle compensation candidate source set in a unified coordinate, replace cross-channel content for an occluded area, and perform local reconstruction with luminosity consistency and edge consistency as constraints, and output a compensated operating field image.

[0075] The processing module 202 is configured to discretize the instrument pose stream and the set of cross-instrument transformations into a spatiotemporal occupancy voxel field, introduce an energy surgical instrument thermal diffusion safety radius and a catheter advancement safety radius, and generate a conflict risk set based on short-time prediction.

[0076] The processing module 202 is configured to jointly solve the conflict risk set and a safety constraint set of a standardized surgical procedure node, generate a set of scheduling instructions and an executable safety time window according to an instrument function role and an operation intention.

[0077] The output module 203 is configured to map a surgical field image, an instrument pose stream and a patient physiological signal sequence to a surgical procedure knowledge graph and input the procedure recognition network, compare a standardized path template with time sequence attention, output a current procedure node and a node confidence, and generate an instrument cooperation consistency discrimination result and a correction suggestion in combination with the conflict risk set and the scheduling instruction set.

[0078] It should be noted that the apparatus provided in the above embodiments is only used as an example to divide the above functional modules in realizing its functions, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0079] The embodiment further discloses an electronic device, which refers to Figure 3 The electronic device can include at least one processor 301, at least one communication bus 302, a user interface 303, a network interface 304, and at least one memory 305.

[0080] The communication bus 302 is configured to realize the connection and communication between the components.

[0081] The user interface 303 can include a display screen (Display) and a camera (Camera), and the optional user interface 303 can further include a standard wired interface and a wireless interface.

[0082] The network interface 304 can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0083] The processor 301 can include one or more processing cores. The processor 301 connects various parts within the server through various interfaces and lines, performs various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Alternatively, the processor 301 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 301 can integrate a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs. The GPU is responsible for rendering and drawing the content to be displayed on the display screen. The modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 301, but can be realized by a separate chip.

[0084] The memory 305 can include a random access memory (RAM) and a read-only memory (ROM). Alternatively, the memory includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area can store data related to the above-mentioned various method embodiments, etc. The memory 305 can also be at least one storage device located away from the aforementioned processor 301. As a computer storage medium, the memory 305 can include an operating system, a network communication module, a user interface 303 module, and an application program of a kind of multi-sensor fusion surgical procedure automatic management method.

[0085] In Figure 3In the electronic device shown, the user interface 303 is mainly used to provide an interface for user input, and obtain data input by the user. The processor 301 can be used to invoke an application program of a multi-sensor fusion surgical procedure automatic management method stored in the memory 305, and when executed by one or more processors 301, the electronic device performs the method of one or more of the above embodiments.

[0086] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0087] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0088] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different parts can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical or other forms.

[0089] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0090] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0091] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory 305 and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned memory 305 includes: a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0092] The present application also discloses a computer readable storage medium, which stores instructions. When executed by one or more processors 301, the instructions cause an electronic device to perform one or more methods as described in the above embodiments.

[0093] The above are only exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practicing the true principles of the present disclosure. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional technical means in the art that are not described in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A multi-sensor fusion based automatic management of surgical procedure method, characterized in that, The method comprises: Collecting a laparoscope image sequence, a catheter pose stream, an energy surgical instrument pose stream, an operating field environment depth, a patient physiological signal sequence, and an intraoperative speech event, performing cross-sensor time sequence alignment using a unified timestamp mechanism to obtain a synchronous data stream; Based on a hand-eye calibration matrix, an imaging internal and external parameter, a laparoscope end cap distortion model, and a light source intensity response curve, performing spatial geometric solving on the synchronous data stream to generate an instrument pose stream and an operating field imaging geometric model in a unified coordinate system, and outputting a cross-instrument transformation set; Jointly solving the laparoscope image sequence and the operating field imaging geometric model, extracting pixel masks of an instrument knife head, a forceps end, a catheter head end, and an anatomical structure through semantic segmentation, performing three-dimensional back projection combined with depth estimation and the cross-instrument transformation set, constructing an occlusion probability graph and a viewable domain graph, and outputting a view angle compensation candidate source set; Performing space-time registration on the main channel laparoscope image and the view angle compensation candidate source set in a unified coordinate to replace the occluded area with cross-channel content, and performing local reconstruction with photometric consistency and edge consistency as constraints to output a compensated operating field image; Discretizing the instrument pose stream and the cross-instrument transformation set to a space-time occupancy voxel field, introducing an energy surgical instrument thermal diffusion safety radius and a catheter advancement safety radius, and generating a conflict risk set based on short-term prediction; Jointly solving the conflict risk set and a safety constraint set of a standardized surgical procedure node, generating a scheduling instruction set and an executable safety time window according to an instrument function role and an operation intention; Mapping the operating field image, the instrument pose stream, and the patient physiological signal sequence to a surgical procedure knowledge graph and inputting them into a procedure recognition network to compare a standardized path template with time sequence attention, outputting a current procedure node and a node confidence, and combining the conflict risk set and the scheduling instruction set to generate an instrument cooperation consistency discrimination result and a correction suggestion.

2. The multi-sensor fusion based automatic management method of surgical procedure according to claim 1, wherein, The method comprises: projecting the instrument pose stream to a laparoscope imaging coordinate system and mapping it to a unified reference coordinate system using the hand-eye calibration matrix; performing geometric solving on the laparoscope image sequence using the imaging internal and external parameter to establish a corresponding relationship between image coordinates and physical coordinates; performing distortion correction on the laparoscope image sequence using the laparoscope end cap distortion model to obtain corrected images with consistent edge linearity and scale; performing photometric correction on the laparoscope image sequence using the light source intensity response curve to obtain brightness-normalized image intensity distribution; jointly solving the instrument pose stream and the laparoscope image sequence processed by the hand-eye calibration matrix, the imaging internal and external parameter, the laparoscope end cap distortion model, and the light source intensity response curve in a unified coordinate system to generate an operating field imaging geometric model and output a cross-instrument transformation set.

3. The multi-sensor fusion based automatic management method of surgical procedure as claimed in claim 1, wherein, The joint calculation of the endoscope image sequence and the surgical field imaging geometric model extracts the pixel mask of the instrument knife head, the end of the forceps, the end of the catheter and the anatomical structure through semantic segmentation, combines depth estimation and the cross-instrument transformation set for three-dimensional back projection, constructs an occlusion probability graph and a viewable area graph, and outputs a view angle compensation candidate source set, specifically including: Perform semantic segmentation on the endoscope image sequence in a unified coordinate system, extract the pixel mask of the instrument knife head, the end of the forceps, the end of the catheter and the anatomical structure, and combine the depth estimation parameters in the surgical field imaging geometric model and the cross-instrument transformation set to perform three-dimensional back projection to generate continuous surgical field point clouds and instrument end three-dimensional pose sequences; Based on the continuous surgical field point cloud and the instrument end three-dimensional pose sequence, an occlusion probability graph sequence is constructed, and a viewable area graph set is generated based on the occlusion probability graph sequence, which is used to quantify the potential visible coverage of the main channel endoscope invisible area on each auxiliary channel endoscope; Based on the viewable area graph set and the occlusion probability graph sequence, the candidate source is sorted according to the coverage area ratio, the baseline angle, the luminosity consistency, the feature matching confidence and the time proximity, and the auxiliary channel information that meets the cross-channel registration accessibility and structural consistency is filtered out, and a view angle compensation candidate source set composed of auxiliary channel identification, coverage area index, expected registration error, expected luminosity error and time stamp range is output.

4. The multi-sensor fusion based automatic management method of surgical procedure according to claim 1, wherein, The spatio-temporal registration of the main channel endoscope image and the view angle compensation candidate source set in a unified coordinate system replaces the occluded area across channels, and performs local reconstruction with luminosity consistency and edge consistency as constraints to output the compensated surgical field image, specifically including: Determine the candidate input of the main channel endoscope image and the view angle compensation candidate source set, which is the auxiliary channel endoscope image adjacent in time to the main channel endoscope image and having coverage and visibility that meet the conditions; Perform cross-channel spatio-temporal registration based on the imaging parameters and geometric constraints provided by the surgical field imaging geometric model and the cross-instrument transformation set in a unified coordinate system to output a first compensation image; After completing the spatio-temporal registration, perform cross-channel content replacement on the pixels in the occluded area of the main channel endoscope image, assign weights to the pixels at the corresponding position in the auxiliary channel endoscope image according to the coverage, re-projection error and time proximity, and replace the occluded area in the main channel endoscope image in a weighted fusion manner to output a second compensation image; After cross-channel content replacement, ensure smooth transition of local brightness and texture with luminosity consistency constraint, and ensure continuity of instrument boundary and anatomical structure boundary with edge consistency constraint, and perform local reconstruction in the occluded area to generate a third compensation image with equal luminosity and continuous edge; The first compensation image and the second compensation image are used as the surgical field image.

5. The multi-sensor fusion based automatic management of surgical procedure method as claimed in claim 1, wherein, Discretize the instrument pose stream and the cross-instrument transformation set to a spatio-temporal occupancy voxel field, introduce energy surgical instrument thermal diffusion safety radius and catheter advancement safety radius, and generate a conflict risk set based on short-term prediction, specifically including: Discretize the instrument pose stream and the set of cross-instrument transformations in a unified coordinate system to map the instrument position and pose in continuous time slices to a spatiotemporal occupancy voxel field; Introduce an energy surgical instrument thermal diffusion safety radius parameter in the spatiotemporal occupancy voxel field to form an energy surgical instrument thermal risk region, and introduce a catheter advancement safety radius parameter to form a catheter advancement risk region, thereby generating an extended spatiotemporal occupancy voxel field; Perform short-time prediction based on the motion trajectory of the instrument pose stream in continuous time slices to obtain a predicted pose trajectory in future time slices, and map the predicted pose trajectory to the extended spatiotemporal occupancy voxel field to form a predicted occupancy region; Perform spatial intersection detection in the predicted occupancy region, and if spatial overlap occurs between the energy surgical instrument thermal risk region, the catheter advancement risk region, or other instrument extended occupancy regions, generate a conflict risk indicator; Collect the conflict risk indicators to output the conflict risk set.

6. The multi-sensor fusion based automatic management of surgical procedure method as claimed in claim 1, wherein, The conflict risk set is jointly solved with the safety constraint set of the standardized surgical procedure node to generate a scheduling instruction set and an executable safety time window according to the instrument function role and operation intention, specifically including: In the joint solving stage, the instrument identifier, time interval, spatial range, risk level, and risk probability in the conflict risk set are time-axis aligned and identifier consistent with the planned time, allowed operation set, and disabled operation set in the standardized surgical procedure node, and the conflict risk is taken as a conflict constraint in a unified optimization domain; The safety constraint is comprehensively solved as a hard constraint and a soft constraint, and the joint solving result containing constraint satisfaction label and residual risk label is output; In the scheduling generation stage, based on the joint solving result, corresponding action elements are called from the action template library according to the instrument function role and operation intention, the action elements include speed upper limit, acceleration upper limit, path fine-tuning amplitude, energy threshold, view priority, waiting condition, cancellation condition, and fallback action, and the action elements are conflict mutually exclusive checked and execution order arranged according to the procedure node priority to generate a structured scheduling instruction set; In the safety time window generation stage, based on the constraint satisfaction evaluation provided by the joint solving result, continuous time slices that satisfy all hard safety boundaries and key soft limits are determined in the spatiotemporal occupancy voxel field, and a warning buffer and a monitoring variable threshold are set at the boundary of each continuous time slice to form an executable safety time window with a time stamp, an instrument identifier, an action parameter, a trigger condition, a cancellation condition, and a fallback action. The executable safety time window and the scheduling instruction set are one-to-one corresponding output.

7. The multi-sensor fusion based automatic management of surgical procedure method as claimed in claim 1, wherein, The surgical field image, the instrument pose stream, and the patient physiological signal sequence are mapped to the surgical procedure knowledge graph and input into the procedure recognition network to compare the standardized path template with the time sequence attention, output the current procedure node and node confidence, and generate an instrument cooperation consistency discrimination result and a correction suggestion in combination with the conflict risk set and the scheduling instruction set generator, specifically including: encoding the surgical field image as a multi-modal feature vector containing anatomical structure boundary, instrument position distribution and photometric feature, and mapping to the image semantic node in the surgical procedure knowledge graph; representing the instrument pose stream as a continuous sequence of three-dimensional position and three-dimensional attitude under a unified coordinate, and mapping to the instrument operation type node in the surgical procedure knowledge graph; normalizing and synchronizing the patient physiological signal sequence with timestamp, and mapping to the patient state type node in the surgical procedure knowledge graph; performing semantic relationship aggregation on the mapping results in the surgical procedure knowledge graph, and using time weight to maintain the continuity of the instrument pose stream and the patient physiological signal sequence, and outputting the time sequence embedding containing procedure features; comparing the standardized path template using the time sequence attention mechanism, calculating the similarity between the time sequence embedding and the standardized path template slice by slice, assigning attention weights to different time slices according to the similarity, and outputting the current procedure node and node confidence; combining the current procedure node and the node confidence with the conflict risk set and the scheduling instruction set for comprehensive judgment, when the node confidence and the conflict risk set or the scheduling instruction set are inconsistent, generating an instrument cooperation consistency judgment result of inconsistency, and outputting the rectification suggestion as feedback information.

8. A multi-sensor fusion based automatic management of surgical procedure apparatus, characterized in that, The device is used to perform a multi-sensor fusion surgical procedure automatic management method according to any one of claims 1-7, and the device comprises an acquisition module, a processing module and an output module, wherein: collecting laparoscope image sequence, catheter pose stream, energy surgical instrument pose stream, surgical field environment depth, patient physiological signal sequence and intraoperative speech event, and using a unified timestamp mechanism to complete cross-sensor time sequence alignment to obtain a synchronous data stream; based on the hand-eye calibration matrix, the imaging internal and external parameters, the laparoscope end cap distortion model and the light source intensity response curve, performing spatial geometric solving on the synchronous data stream to generate an instrument pose stream and a surgical field imaging geometric model under a unified coordinate system, and outputting a cross-instrument transformation set; jointly solving the laparoscope image sequence and the surgical field imaging geometric model, extracting the pixel mask of the instrument blade, clamp end, catheter head and anatomical structure through semantic segmentation, combining depth estimation and the cross-instrument transformation set to perform three-dimensional back projection, constructing an occlusion probability graph and a viewable domain graph, and outputting a view angle compensation candidate source set; performing space-time registration on the main channel laparoscope image and the view angle compensation candidate source set under a unified coordinate, replacing the occluded area across channels, and performing local reconstruction with luminosity consistency and edge consistency as constraints, and outputting the compensated surgical field image; discretizing the instrument pose stream and the cross-instrument transformation set to a space-time occupancy voxel field, introducing energy surgical instrument heat diffusion safety radius and catheter advancement safety radius, and generating a conflict risk set based on short-term prediction; jointly solving the conflict risk set and the safety constraint set of the standardized surgical procedure node, generating a scheduling instruction set and an executable safety time window according to the instrument function role and operation intention; The surgical field image, the instrument pose stream and the patient physiological signal sequence are mapped to a surgical procedure knowledge graph and input into a procedure recognition network to compare a standardization path template with time sequence attention, output a current procedure node and a node confidence, and combine the conflict risk set and the scheduling instruction set to generate an instrument collaboration consistency discrimination result and a correction suggestion.

9. An electronic device, comprising: The electronic device comprises a processor, a communication bus, a user interface, a network interface, and a memory, the memory is configured to store instructions, the user interface and the network interface are configured to communicate with other devices, the communication bus is configured to realize connection and communication between components in the electronic device, and the processor is configured to execute the instructions stored in the memory to enable the electronic device to perform the method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions which, when executed, perform the method in any one of claims 1-7.

Images