A multi-sensor fusion surgical procedure automatic management method and device

Abstract

The application provides a kind of multi-sensor fusion surgical procedure automatic management method and device, it is related to the technical field of automated process, method includes: by multi-source sensor data uniform timestamp alignment and space geometry solution, constructs the instrument pose flow and operating field geometric model under uniform coordinate system.Combining semantic segmentation and three-dimensional back projection generates occlusion probability graph and carries out cross-channel compensation reconstruction, improves image integrity.Through voxel field dispersion and short-time prediction, a conflict risk set is formed, and a standardized surgical procedure constraint is jointly solved to generate a scheduling instruction and a safety time window.Finally, the image, pose and physiological signal are mapped to the knowledge graph, the process node is identified by combining the timing attention, and the consistency discrimination and correction suggestion are output.The application can solve the problem of inaccurate surgical procedure management caused by the overlapping field of view of multi-source sensor data.

Description

Technical Field

[0001] This invention relates to the technical field of automated processes, and specifically to a method and apparatus for automated management of surgical processes using multi-sensor fusion. Background Technology

[0002] Currently, by constructing a closed-loop architecture of "perception-analysis-decision-execution," intelligent management of the surgical process is achieved. This involves acquiring multimodal data from operating table position sensors, surgical field image acquisition devices, physiological monitoring sensors, and surgical instrument tracking sensors. After cross-sensor synchronization via a unified timestamp and calibration algorithm, the data is input into a fusion processing model. Combined with spatiotemporal registration, feature extraction, and multimodal fusion, process node identification and prediction are completed. Then, by matching with standardized surgical paths, risks and deviations are identified, and prompts or correction plans are generated. Finally, the status and decision support are presented in real time in the multimodal interactive interface and auxiliary equipment, realizing the transformation from passive recording to active monitoring and intelligent scheduling.

[0003] In multi-instrument collaborative operation scenarios, different endoscopes, catheters, and energy surgical instruments may enter the patient's body simultaneously. The sensor data they carry may exhibit overlapping fields of view, signal interference, and intersecting operation trajectories. Therefore, multi-sensor fusion methods need to further introduce cross-instrument perspective compensation mechanisms on the basis of temporal alignment and feature fusion. By using geometric registration and spatial mapping algorithms, the occlusion and deviation of the sensing range of each instrument can be eliminated. At the same time, a path conflict detection model is established to predict and identify the pose trajectories of the multi-instrument instruments acquired in real time. Combined with the safety constraints of standardized surgical procedures, the operation is dynamically verified to ensure that it conforms to the established rules. This allows surgical procedure management to not only monitor the operation sequence but also ensure the safety and consistency of multi-instrument collaboration.

[0004] Therefore, a method is needed to solve the problem of insufficient precision in surgical procedure management caused by overlapping fields of view in multi-source sensor data. Summary of the Invention

[0005] This invention provides a method and apparatus for automatic management of surgical procedures using multi-sensor fusion, which can solve the problem of insufficient accuracy in surgical procedure management caused by overlapping fields of view in multi-source sensor data.

[0006] In a first aspect, the present invention provides a method for automated management of surgical procedures using multi-sensor fusion, the method comprising:

[0007] Acquire laparoscopic image sequences, catheter pose flow, energy surgical instrument pose flow, surgical field depth, patient physiological signal sequences, and intraoperative voice events. Use a unified timestamp mechanism to complete cross-sensor time alignment to obtain a synchronous data stream.

[0008] Based on the hand-eye calibration matrix, imaging intrinsic and extrinsic parameters, end-cap distortion model of the endoscope, and light source intensity response curve, spatial geometry solution is performed on the synchronous data stream to generate instrument pose flow and surgical field imaging geometric model in a unified coordinate system, and output cross-instrument transformation set.

[0009] The laparoscopic image sequence is jointly solved with the surgical field imaging geometric model. The pixel mask of the instrument tip, forceps tip, catheter tip and anatomical structure is extracted by semantic segmentation. The depth estimation and the cross-instrument transformation set are combined to perform three-dimensional back projection, construct the occlusion probability map and the field of view map, and output the viewpoint compensation candidate source set.

[0010] The main channel endoscopic image and the viewpoint compensation candidate source set are spatiotemporally registered under a unified coordinate system. The occluded area is replaced with cross-channel content, and local reconstruction is performed with light intensity consistency and edge consistency as constraints. The compensated surgical field image is then output.

[0011] The instrument pose flow and the cross-instrument transformation set are discretized into a spatiotemporal occupancy voxel field. The thermal diffusion safety radius of energy surgical instruments and the catheter advancement safety radius are introduced. A conflict risk set is generated based on short-time prediction.

[0012] The conflict risk set and the safety constraint set of the standardized surgical procedure nodes are jointly solved to generate a scheduling instruction set and an executable safe time window based on the instrument's functional role and operational intent.

[0013] The surgical field images, the instrument pose flow, and the patient's physiological signal sequence are mapped to the surgical procedure knowledge graph and input into the procedure recognition network. The standardized path template is compared with temporal attention, and the current procedure node and node confidence are output. The instrument collaboration consistency judgment result and correction suggestions are generated by combining the conflict risk set and the scheduling instruction set.

[0014] In a second aspect of the invention, a multi-sensor fusion-based automated surgical procedure management device is provided. The device is used to execute a multi-sensor fusion-based automated surgical procedure management method as described in any of the above embodiments. The device includes an acquisition module, a processing module, and an output module, wherein:

[0015] The acquisition module is used to collect laparoscopic image sequences, catheter pose flow, energy surgical instrument pose flow, surgical field environment depth, patient physiological signal sequences, and intraoperative voice events. It uses a unified timestamp mechanism to complete cross-sensor time alignment and obtain a synchronous data stream.

[0016] The processing module is used to perform spatial geometry solving on the synchronous data stream based on the hand-eye calibration matrix, imaging intrinsic and extrinsic parameters, end-cap distortion model of the endoscope and light source intensity response curve, generate instrument pose flow and surgical field imaging geometric model in a unified coordinate system, and output cross-instrument transformation set.

[0017] The processing module is used to jointly solve the laparoscopic image sequence and the surgical field imaging geometric model, extract the pixel mask of the instrument tip, forceps tip, catheter tip and anatomical structure through semantic segmentation, combine depth estimation and the cross-instrument transformation set to perform three-dimensional back projection, construct the occlusion probability map and the field of view map, and output the viewpoint compensation candidate source set.

[0018] The processing module is used to perform spatiotemporal registration of the main channel endoscopic image and the view compensation candidate source set under unified coordinates, perform cross-channel content replacement on the occluded area, and perform local reconstruction with photometric consistency and edge consistency as constraints, and output the compensated surgical field image.

[0019] The processing module is used to discretize the instrument pose flow and the cross-instrument transformation set into a spatiotemporal occupied voxel field, introduce the thermal diffusion safety radius of the energy surgical instrument and the catheter advancement safety radius, and generate a conflict risk set based on short-time prediction.

[0020] The processing module is used to jointly solve the conflict risk set and the safety constraint set of the standardized surgical procedure nodes, and generate a scheduling instruction set and an executable safe time window according to the instrument function role and operation intention.

[0021] The output module is used to map the surgical field image, the instrument pose flow and the patient physiological signal sequence to the surgical procedure knowledge graph and input them into the procedure recognition network. It compares the standardized path template with temporal attention, outputs the current procedure node and node confidence, and generates instrument collaboration consistency judgment results and correction suggestions by combining the conflict risk set and the scheduling instruction set.

[0022] In a third aspect of the invention, an electronic device is provided, including a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any of the preceding embodiments.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium storing instructions that, when executed, perform the method as described in any of the preceding claims.

[0024] In summary, one or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0025] 1. This invention introduces a unified timestamp mechanism and spatial geometry solution after multi-source sensor data acquisition, enabling the fusion and expression of different modal data under the same spatiotemporal coordinates. Combined with the joint solution of endoscopic images and surgical field geometric models and cross-channel perspective compensation, it effectively eliminates occlusion and information loss caused by overlapping fields of view. At the same time, it uses spatiotemporal occupancy voxel fields and safety radius parameters to predict and avoid potential conflicts, and maps surgical field images, instrument pose flow and patient physiological signals to a unified surgical procedure knowledge graph. Through temporal attention comparison and standardized path templates, it achieves accurate identification and consistency correction of process nodes, thereby ensuring that accurate and reliable surgical procedure management results can still be output even in the presence of overlapping fields of view and multi-sensor interference.

[0026] 2. Based on the hand-eye calibration matrix, imaging intrinsic and extrinsic parameters, end-cap distortion model of the endoscope, and light source intensity response curve, spatial geometric solution is performed, which can realize the spatial unification and photometric normalization of instrument pose flow and endoscope image. This allows subsequent multimodal data to be processed under the same coordinate and photometric scale, thereby significantly improving the accuracy of instrument pose and image registration and ensuring the stability of the surgical field geometric model.

[0027] 3. By jointly solving the laparoscopic image sequence and the surgical field imaging geometric model, and combining semantic segmentation and 3D retro-projection to generate occlusion probability maps and visible field maps, the visibility can be quantified and compensable sources can be screened when there are multiple instruments operating and local occlusion in the surgical field. This improves the information complementarity between multi-channel images and significantly reduces visual loss caused by occlusion.

[0028] 4. By performing cross-channel spatiotemporal registration and local reconstruction of the main channel image based on the viewpoint compensation candidate source set, the image content of the occluded area can be effectively restored. Furthermore, by constraining the photometric consistency and edge consistency, the brightness smoothness and boundary continuity of the compensated image are ensured, thereby enhancing the integrity and continuous visibility of the surgical field image.

[0029] 5. By mapping the instrument pose flow and cross-instrument transformation set to the spatiotemporal occupied voxel field and introducing the safety radius parameters of energy instruments and catheters, and combining short-term prediction to generate a conflict risk set, it is possible to identify potential future instrument collisions or energy diffusion risks in advance, thereby improving the forward-looking risk prediction and active protection capabilities of the surgical procedure.

[0030] 6. By jointly solving the conflict risk set and the safety constraint set of standardized surgical procedure nodes, the output scheduling instruction set and executable safe time window can realize dynamic coordination and resource allocation for different instrument operation intentions, and ensure that the execution of actions meets the safety boundaries and process specifications, thereby ensuring safety and process consistency under multi-instrument collaboration. Attached Figure Description

[0031] Figure 1This is a flowchart illustrating an automated surgical procedure management method based on multi-sensor fusion disclosed in an embodiment of the present invention.

[0032] Figure 2 This is a schematic diagram of a multi-sensor fusion-based automated surgical procedure management device disclosed in an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention.

[0034] Explanation of reference numerals in the attached drawings: 201, acquisition module; 202, processing module; 203, output module; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. Detailed Implementation

[0035] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0036] In the description of the embodiments of the present invention, words such as "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "for example" or "for instance" in the embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0037] In the description of the embodiments of the present invention, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0038] Existing intelligent surgical process management constructs a closed-loop architecture of "perception-analysis-decision-execution" to achieve the collection, synchronization, fusion, and process node identification of multimodal data. It also generates risk warnings and correction schemes in conjunction with standardized surgical paths, thereby completing the transformation from passive recording to proactive monitoring and intelligent scheduling. In multi-instrument collaborative operation scenarios, due to overlapping fields of view, signal interference, and trajectory intersections in the sensor data of laparoscopy, catheters, and energy surgical instruments, it is necessary to introduce cross-instrument view compensation mechanisms and path conflict detection models to predict and identify the pose trajectories of multiple instruments. Combined with safety constraints, the consistency and compliance of operations are dynamically verified to solve the problem of insufficient accuracy in surgical process management under multi-source sensor data fusion.

[0039] This embodiment discloses an automated surgical procedure management method based on multi-sensor fusion, referring to... Figure 1 This includes the following steps S110-S170:

[0040] S110 acquires laparoscopic image sequences, catheter pose flow, energy surgical instrument pose flow, surgical field depth, patient physiological signal sequences, and intraoperative voice events. It uses a unified timestamp mechanism to complete cross-sensor time alignment and obtain a synchronized data stream.

[0041] This invention discloses a multi-sensor fusion-based automated surgical procedure management method applied to a server. The server includes, but is not limited to, electronic devices such as mobile phones, tablets, wearable devices, and PCs (Personal Computers), and can also be a backend server running a multi-sensor fusion-based automated surgical procedure management method. The server can be implemented using a standalone server or a server cluster composed of multiple servers.

[0042] The process involves acquiring laparoscopic image sequences. A laparoscopic imaging device captures continuous video frames of the surgical field in real time, and these frames are input as a temporally sequenced image sequence. Here, a laparoscopic image sequence refers to a collection of laparoscopic images arranged in chronological order, typically including visible information in the surgical field such as anatomical structures, instrument tips, and forceps. During acquisition, it is crucial to ensure the resolution, frame rate, and illumination uniformity of the laparoscopic images. Simultaneously, data is transmitted to the fusion processing module via the image acquisition interface to ensure that the image sequence can be compared and fused with data from other sensors within the same time domain.

[0043] The catheter pose flow is acquired by using optical or electromagnetic tracking sensors to obtain the 3D position and orientation of the catheter at every moment during the procedure. These 3D positions and orientations from consecutive moments are then compiled into a temporal data stream. Here, catheter pose flow refers to the dynamic trajectory information of the catheter in a unified coordinate system, encompassing translational and rotational parameters, and characterizing the catheter's advancement, retraction, and rotation within the body. During acquisition, high sensitivity and low latency of the catheter tracking sensors are crucial for subsequent spatial registration with endoscopic image sequences.

[0044] The pose flow of energy-powered surgical instruments is acquired in real time via sensors on the surgical robot's end effector or an external optical tracking system. This data is continuously output as a temporal pose data stream. Here, the energy-powered surgical instrument pose flow refers to the spatial trajectory data describing the dynamic motion of energy-powered cutting instruments, energy-powered coagulation instruments, etc., during operation, including changes in the spatial position and orientation of the instrument's end effector. During acquisition, it is crucial to ensure that the tracking frequency matches the endoscopic image acquisition frequency to provide accurate input for subsequent path conflict detection and safety constraint verification.

[0045] The surgical field depth is acquired using a structured light depth camera, binocular stereo imaging device, or laser scanning device to obtain depth information for each pixel in the surgical field scene, forming a depth map sequence. The surgical field depth here refers to the set of spatial distances from the anatomical and instrument surfaces to the imaging plane within the surgical field; it is a necessary input for 3D reconstruction and occlusion detection. During acquisition, the stability of the depth sensor under endoscopic illumination conditions must be ensured, and errors caused by light reflection and tissue absorption must be corrected through a real-time calibration mechanism.

[0046] The patient's physiological signal sequence is collected. A multi-parameter physiological monitor acquires physiological parameters such as electrocardiogram (ECG), blood pressure, respiratory rate, blood oxygen saturation, and electroencephalogram (EEG) activity in real time, and arranges them into a physiological signal sequence in chronological order. Here, the patient's physiological signal sequence refers to a set of multidimensional physiological parameters arranged in a unified time domain, reflecting the patient's overall condition during the surgical procedure. During acquisition, it is necessary to ensure the continuity and low latency of physiological signal monitoring, and to remove electrical noise introduced by surgical procedures using an anti-interference filtering algorithm, thereby providing reliable input for risk identification in process management.

[0047] The procedure involves collecting intraoperative speech events. A speech acquisition device captures real-time verbal communication between the surgeon and assistant during surgery, and a speech recognition algorithm converts the speech stream into a sequence of semantic events. In this context, intraoperative speech events refer to the recognized and time-stamped speech content, such as instructions, prompts, and responses, directly reflecting the operator's intent. During acquisition, robustness of speech recognition must be ensured even in noisy operating room environments, and keyword recognition and contextual analysis are used to ensure the recognition results can be used for determining operational intent and annotating workflow nodes.

[0048] By embedding a unified time synchronization protocol at all sensor inputs, endoscopic image sequences, catheter pose streams, energy surgical instrument pose streams, surgical field depth, patient physiological signal sequences, and intraoperative speech events are uniformly labeled as multimodal data streams with the same reference time. The unified timestamp mechanism here means providing the same reference time base for different data sources, ensuring that different types of data maintain time consistency within milliseconds. The aligned dataset is called a synchronized data stream, representing a multimodal input sequence arranged under a unified time coordinate, providing a unified data foundation for subsequent spatial geometry solutions, semantic segmentation, path conflict detection, and process node identification.

[0049] S120, based on the hand-eye calibration matrix, imaging intrinsic and extrinsic parameters, end-cap distortion model of the endoscope, and light source intensity response curve, performs spatial geometric solution on the synchronous data stream, generates the instrument pose flow and surgical field imaging geometric model in a unified coordinate system, and outputs a cross-instrument transformation set.

[0050] In one possible implementation, based on the hand-eye calibration matrix, imaging intrinsic and extrinsic parameters, end-cap distortion model of the endoscope, and light source intensity response curve, spatial geometric solution is performed on the synchronous data stream to generate the instrument pose flow and surgical field imaging geometric model in a unified coordinate system, and output a cross-instrument transformation set. Specifically, this includes: projecting the instrument pose flow onto the endoscope imaging coordinate system and mapping it to a unified reference coordinate system using the hand-eye calibration matrix; performing geometric solution on the endoscope image sequence using imaging intrinsic and extrinsic parameters to establish a correspondence between image coordinates and physical coordinates; performing distortion correction on the endoscope image sequence using the end-cap distortion model of the endoscope to obtain a corrected image with consistent edge straightness and proportions; performing photometric correction on the endoscope image sequence using the light source intensity response curve to obtain a brightness-normalized image intensity distribution; and jointly solving the instrument pose flow and endoscope image sequence processed by the hand-eye calibration matrix, imaging intrinsic and extrinsic parameters, end-cap distortion model of the endoscope, and light source intensity response curve in a unified coordinate system to generate a surgical field imaging geometric model and output a cross-instrument transformation set.

[0051] Specifically, when mapping the instrument pose flow using the hand-eye calibration matrix, a homogeneous transformation chain is established within a unified coordinate system. This transforms the temporal pose of the instrument end effector in the end effector coordinate system to the camera coordinate system, and then to the unified reference coordinate system, resulting in an instrument pose flow that can be directly interfaced with image geometric calculations. The mapping is accomplished using the following transformation relationship:

[0052]

[0053] in Let represent the pose matrix of the instrument end in a unified coordinate system at time t. This represents the homogeneous extrinsic matrix from the camera coordinate system to the unified coordinate system. Represent the hand-eye calibration matrix and give the homogeneous transformation from the camera coordinate system to the end effector coordinate system. This represents the homogeneous transformation from the end effector to the instrument end at time t. By applying this chained transformation frame by frame to the instrument pose flow, a temporally continuous instrument pose flow that is aligned with the image in a unified coordinate system is obtained, providing consistent coordinate semantics for subsequent image projection and 3D retro-projection.

[0054] When performing geometric calculations on endoscopic image sequences using imaging intrinsic and extrinsic parameters, a pinhole imaging model is employed to establish a bidirectional mapping between pixel coordinates and physical 3D points. The correspondence between pixels and 3D points is then established within a unified coordinate system. The pixel generation process is described using the following forward projection model:

[0055]

[0056] in Represents pixel coordinates, This represents the scale factor and is proportional to the depth of the point. This represents the imaging intrinsic parameter matrix, which includes the focal length and principal point coordinates. and Indicate the imaging extrinsic parameters and provide the rotation and translation from the unified coordinate system to the camera coordinate system. This represents the coordinates of a 3D point in a unified coordinate system. A backprojection is performed to recover the 3D coordinates, provided depth is available.

[0057]

[0058] in It represents the three-dimensional coordinates in the camera coordinate system. By solving frame by frame, the bidirectional correspondence between pixels and three-dimensional points is completed, laying the geometric foundation for the unified fusion of distortion correction and photometric correction.

[0059] When using the end-cap distortion model to correct distortion in laparoscopic image sequences, pixels are first normalized to the imaging plane. Then, a joint model of radial and tangential distortion is applied to complete pixel-level correction, outputting a corrected image with consistent edge straightness and proportions. The following distortion model describes the mapping from normalized coordinates to distorted coordinates:

[0060]

[0061] in, and This represents the distortion-free plane coordinates after normalization using imaging intrinsic parameters. and Represents the coordinates of the distorted plane. Indicates the radial distance to the optical axis. , , Represents the radial distortion coefficient. and The tangential distortion coefficient is represented by a reverse mapping from distorted coordinates to undistorted coordinates obtained through numerical solution. The entire frame of pixels is then resampled to restore the linear geometry and eliminate edge stretching and bending caused by the end cap.

[0062] When performing photometric correction on endoscopic image sequences using the light source intensity response curve, a brightness response function with respect to the incident angle and working distance is established. This function, combined with the sensor response function, completes radiometric calibration and corner vignetting compensation, outputting a brightness-normalized image intensity distribution. Normalization is achieved using a combined multiplicative and additive photometric model.

[0063]

[0064] in, This represents the normalized pixel intensity. Indicates the original pixel intensity. and The linear response gain and bias of the imaging sensor are represented. Represent the light source intensity response curve and characterize the illumination as a function of the incident angle. work distance The attenuation characteristics The vignetting function is expressed as a function of the normalized radius. Changes, fitted through calibration experiments and After adjusting the parameters, each frame of the image is corrected pixel by pixel, so that regions with different depths and different incident angles can obtain comparable brightness scales, and significantly improve the robustness of subsequent feature matching and segmentation.

[0065] When jointly solving the instrument pose flow and endoscopic image sequence, processed by hand-eye calibration matrix, imaging internal and external participation endoscopic endcap distortion model, and light source intensity response curve in a unified coordinate system, a joint optimization method is used to simultaneously refine the trajectory of the externally participating instruments and generate a surgical field imaging geometry model that can uniformly describe the camera imaging geometry and the three-dimensional morphology of the surgical field. Simultaneously, a cross-instrument transformation set is output to characterize the stable relative relationships between multiple instruments. The reprojection residual and pose prior are used as objective terms to adjust and minimize the following cost function:

[0066]

[0067] in This represents the distortion-free pixel position of the i-th feature at time t. This represents the projection function that generates pixel positions from 3D points involving both internal and external imaging components. This represents the coordinates of the i-th 3D point in a unified coordinate system. This represents the increment of the device pose at adjacent time points and is subject to the prior constraint of motion smoothness. Indicates prior weights, This represents the covariance weighted matrix of pose increments. After optimization convergence, the surgical field imaging geometry model is obtained, including corrected intrinsic and extrinsic camera parameters in a unified coordinate system, along with key 3D point clouds and visibility indices. Based on the same optimization result, a cross-instrument transformation set is calculated, and the relative pose relationships between different instrument ends are given by the following formula:

[0068]

[0069] in The homogeneous transformation of instrument B relative to instrument A at time t is used to describe the mutual position and attitude constraints of multiple instruments in a unified coordinate system. The geometric and motion priors of downstream occlusion estimation, viewpoint compensation, and path conflict detection are jointly constituted by the surgical field imaging geometric model and the cross-instrument transformation set, and the consistent coordinate semantics and photometric semantics are maintained throughout the entire intraoperative process in subsequent stages.

[0070] S130 combines the laparoscopic image sequence with the surgical field imaging geometric model for joint calculation. It extracts pixel masks of instrument tips, forceps tips, catheter tips and anatomical structures through semantic segmentation. It then performs 3D back projection by combining depth estimation and cross-instrument transformation set to construct occlusion probability map and field of view map, and outputs a candidate source set for viewpoint compensation.

[0071] In one possible implementation, the laparoscopic image sequence is jointly solved with the surgical field imaging geometry model. Pixel masks for the instrument tip, forceps tip, catheter tip, and anatomical structures are extracted through semantic segmentation. Three-dimensional retrocasting is then performed using depth estimation and cross-instrument transformation sets to construct an occlusion probability map and a viewport map, outputting a viewpoint compensation candidate source set. Specifically, this includes: performing semantic segmentation on the laparoscopic image sequence in a unified coordinate system to extract pixel masks for the instrument tip, forceps tip, catheter tip, and anatomical structures; and performing three-dimensional retrocasting using depth estimation parameters from the surgical field imaging geometry model and cross-instrument transformation sets to generate a continuous surgical field point cloud and instrument tip three-dimensional transformation map. Three-dimensional pose sequence; Based on the continuous surgical field point cloud and the three-dimensional pose sequence of the instrument end, an occlusion probability map sequence is constructed, and a visible field atlas is generated by combining the occlusion probability map sequence. The visible field atlas is used to quantify the potential visible coverage of the invisible area of ​​the main channel endoscope on each secondary channel endoscope; Based on the visible field atlas and the occlusion probability map sequence, candidate sources are sorted according to the coverage area ratio, baseline angle, photometric consistency, feature matching confidence, and temporal proximity, and secondary channel information that meets the requirements of cross-channel registration accessibility and structural consistency is selected. The output is a view compensation candidate source set consisting of secondary channel identifier, coverage area index, expected registration error, expected photometric error, and timestamp range.

[0072] Specifically, when performing semantic segmentation using laparoscopic image sequences in a unified coordinate system, the laparoscopic image sequences are first preprocessed, including photometric equalization, noise filtering, and edge enhancement, to ensure segmentation accuracy. The preprocessed laparoscopic image sequences are then input into a depth segmentation network trained on a surgical scene dataset. The network outputs multi-class pixel masks in a unified coordinate system. A pixel mask refers to the category labeling result corresponding to each pixel in the image. Specifically, the instrument tip pixel mask represents the area for cutting or coagulating the instrument tip, the clamp tip pixel mask represents the area for mechanically manipulating the clamp tip, the catheter tip pixel mask represents the catheter port area entering the patient's body, and the anatomical structure pixel mask represents the boundaries of blood vessels, tissues, and organs in the surgical field. Combining the depth estimation parameters included in the surgical field imaging geometry model, the pixels in each mask region are mapped to 3D points in a unified coordinate system through back-projection, resulting in a continuous surgical field point cloud. Simultaneously, a cross-instrument transformation set is used to map pixels in the point cloud related to the instrument tip to specific instrument poses, thus forming a 3D pose sequence of the instrument tip. The surgical field point cloud describes the three-dimensional structure of the entire surgical field, while the three-dimensional pose sequence of the instrument end is used to characterize the position and posture changes of the instrument end in space.

[0073] When constructing an occlusion probability map sequence based on continuous surgical field point clouds and 3D pose sequences of instrument ends, the visibility of the point cloud is determined by emitting rays from the laparoscopic imaging center as the viewpoint in a unified coordinate system. For each pixel direction, if multiple depth points exist in the point cloud, the point with the smallest depth is selected as the visible point, and the remaining points are marked as potential occlusion points. The occlusion ratio of different instrument components on anatomical structures is further statistically analyzed using the 3D pose sequence of the instrument ends. By accumulating and calculating the occlusion frequency and occlusion confidence of different pixels across multiple time slices, an occlusion probability map sequence is generated. Here, the occlusion probability map represents the probability that each pixel is occluded in the time dimension. A visible field atlas is generated using the occlusion probability map sequence and the visibility distribution of the surgical field point cloud. The visible field atlas is used to describe the coverage of areas invisible in the main channel laparoscopic image under multi-channel laparoscopic conditions on other secondary channel laparoscopic images, including the coverage area ratio, coverage boundary continuity, and visible texture integrity.

[0074] When ranking candidate sources based on the visible area atlas and occlusion probability map sequence, the coverage area ratio is used as the first priority indicator. The coverage area ratio represents the percentage of the area that the secondary channel cavity mirror can compensate for that is not visible to the primary channel cavity mirror. The baseline angle is used as the second priority indicator. The baseline angle refers to the angle between the optical axes of the primary and secondary channel cavity mirrors; too small a value leads to information redundancy, while too large a value may cause geometric distortion. Photometric consistency is used as the third priority indicator. Photometric consistency represents the similarity in color and brightness distribution between images from different channels; higher photometric consistency results in better fusion. Feature matching confidence is used as the fourth priority indicator, referring to the reliability of cross-channel keypoint matching; high confidence indicates more stable cross-channel registration. Temporal proximity is used as the fifth priority indicator, referring to the proximity of the secondary channel image frame to the primary channel image frame in terms of timestamps; the closer the proximity, the better the dynamic scene consistency. Based on these indicators, candidate sources are weighted and ranked. Secondary channels with excessive errors or structural inconsistencies during cross-channel registration are eliminated, and secondary channel information that meets the requirements is selected. The final output view compensation candidate source set consists of sub-channel identifiers, coverage area indexes, expected registration errors, expected photometric errors, and timestamp ranges, providing a complete input set for subsequent cross-channel registration and content replacement of occluded areas.

[0075] S140 performs spatiotemporal registration of the main channel endoscopic image and the viewpoint compensation candidate source set under unified coordinates, performs cross-channel content replacement on the occluded area, and performs local reconstruction with photometric consistency and edge consistency as constraints, outputting the compensated surgical field image.

[0076] In one possible implementation, the main channel endoscopic image and the view compensation candidate source set are spatiotemporally registered under a unified coordinate system. Cross-channel content replacement is performed on the occluded area, and local reconstruction is performed using photometric consistency and edge consistency as constraints. The compensated surgical field image is then output. Specifically, this includes: determining candidate inputs from the view compensation candidate source set for the main channel endoscopic image; the candidate inputs are secondary channel endoscopic images that are temporally adjacent to the view compensation candidate source set and the main channel endoscopic image and meet the conditions for coverage and visibility; and performing cross-channel spatiotemporal registration under a unified coordinate system based on the surgical field imaging geometry model and the imaging parameters and geometric constraints provided by the cross-instrument transformation set, outputting a first compensated surgical field image. Compensation Image: After completing spatiotemporal registration, cross-channel content replacement is performed on pixels in the occluded area of ​​the main channel endoscopic image. Pixels at corresponding positions in the secondary channel endoscopic image are weighted according to coverage, reprojection error, and temporal proximity, and the occluded area in the main channel endoscopic image is replaced by weighted fusion to output the second compensation image. After cross-channel content replacement, photometric consistency constraints are used to ensure smooth transition of local brightness and texture, and edge consistency constraints are used to ensure the continuity of instrument boundaries and anatomical structure boundaries. Local reconstruction is performed in the occluded area to generate a third compensation image with photometric balance and continuous edges. The first compensation image and the second compensation image are used as surgical field images.

[0077] Specifically, when determining candidate inputs for the main channel endoscopic image using the viewpoint compensation candidate source set, the acquisition time of the main channel endoscopic image is first compared with the timestamp range of the secondary channel endoscopic images in the viewpoint compensation candidate source set under a unified timestamp mechanism, filtering out candidate frames with a time difference below a threshold. Then, based on the view area atlas, the coverage of the secondary channel endoscopic image in the invisible area of ​​the main channel endoscopic image is determined. Coverage refers to the proportion of the area that the secondary channel image can compensate for after geometric projection of the main channel occlusion area. Only images with coverage exceeding a preset threshold are included in the candidate set. Furthermore, the visibility confidence in the occlusion probability map is combined to ensure that the selected secondary channel image has stable visibility in the main channel occlusion area. Finally, candidate secondary channel images that meet the requirements of temporal adjacency, coverage, and visibility are obtained as inputs for subsequent cross-channel registration.

[0078] When performing cross-channel spatiotemporal registration in a unified coordinate system based on the imaging parameters and geometric constraints provided by the surgical field imaging geometry model and the cross-instrument transformation set, the candidate sub-channel images are first projected onto the unified coordinate system using the imaging intrinsic and extrinsic parameter matrices from the surgical field imaging geometry model. Then, by combining the homogeneous transformation relationships in the cross-instrument transformation set, the optical center positions, imaging plane directions, and instrument end poses of the main and sub-channels are spatially aligned. During this process, photometric consistency and minimizing geometric residuals are used as optimization objective functions, and the registration parameters are iteratively optimized to reduce reprojection errors. Reprojection error here represents the difference between the pixel positions predicted by the geometric model and the actual pixel positions in the sub-channels. After optimization, the first compensated image is output, which is the sub-channel image spatiotemporally aligned with the main channel image in the unified coordinate system.

[0079] After completing spatiotemporal registration, when performing cross-channel content replacement on pixels in occluded areas of the main channel endoscopic image, the occlusion probability map is first used to identify the occluded areas in the main channel endoscopic image, and these areas are then used as candidate replacement targets. For each pixel in an occluded area, the corresponding pixel value is obtained from the aligned secondary channel image, and a fusion weight is calculated based on coverage, reprojection error, and temporal proximity. The coverage weight measures the visible coverage ratio of the target area by the secondary channel image, the reprojection error weight measures the registration accuracy, and the temporal proximity weight measures the consistency between the acquisition time of the secondary channel image and the acquisition time of the main channel image. The final fused pixel value is obtained by weighted averaging, expressed as:

[0080]

[0081] in This represents the blending value of pixel p. This represents the pixel value at the corresponding position of the k-th sub-channel after registration. This indicates the fusion weight. By replacing the occluded areas pixel by pixel with weighted values, a second compensated image is obtained. This image retains the overall consistency of the main channel image in terms of structure, while repairing the defects caused by occlusion.

[0082] After cross-channel content replacement, when performing local reconstruction with photometric consistency and edge consistency constraints, a photometric smoothing model is first established within the occluded area to ensure a continuous transition in brightness and color between the main channel and sub-channel images. Photometric consistency constraints are achieved by minimizing local pixel brightness differences, ensuring no significant abrupt changes in brightness distribution between the replaced area and the surrounding unoccluded areas. Edge consistency constraints are achieved by maintaining image gradient continuity, ensuring that instrument edges and anatomical structure boundaries remain clear and coherent after replacement. Therefore, a joint reconstruction objective function is constructed:

[0083]

[0084] in This represents the pixel value after local reconstruction. This represents the pixel value after weighted fusion. This represents the pixel value of the main channel image in the adjacent area. Represents the gradient of the image. and To balance the parameters, the optimized third-compensation image maintains a natural transition and boundary integrity in terms of brightness and edge structure.

[0085] Finally, the first compensated image and the second compensated image are combined to ensure that the main channel image and the sub-channel image are aligned in time and space under a unified coordinate system, so as to achieve photometric smoothing and continuous edge repair of the local occluded area. The result is output as a compensated surgical field image for subsequent path conflict detection and surgical procedure consistency judgment.

[0086] S150 discretizes the instrument pose flow and cross-instrument transformation set into a spatiotemporal occupancy voxel field, introduces the thermal diffusion safety radius of energy surgical instruments and the catheter advancement safety radius, and generates a conflict risk set based on short-time prediction.

[0087] In one possible implementation, the instrument pose flow and cross-instrument transformation set are discretized into a spatiotemporal occupied voxel field. The thermal diffusion safety radius of the energy surgical instrument and the catheter advancement safety radius are introduced. A conflict risk set is generated based on short-time prediction. Specifically, this includes: discretizing the instrument pose flow and cross-instrument transformation set in a unified coordinate system to map the instrument position and attitude in consecutive time slices into a spatiotemporal occupied voxel field; introducing the thermal diffusion safety radius parameter of the energy surgical instrument into the spatiotemporal occupied voxel field to form a thermal risk region for the energy surgical instrument, and introducing the catheter advancement safety radius parameter to form a catheter advancement risk region, thereby generating an extended spatiotemporal occupied voxel field; performing short-time prediction based on the motion trajectory of the instrument pose flow within consecutive time slices to obtain the predicted pose trajectory for several future time slices, and mapping the predicted pose trajectory to the extended spatiotemporal occupied voxel field to form a predicted occupied region; performing spatial intersection detection within the predicted occupied region; if spatial overlap is detected between the thermal risk region of the energy surgical instrument, the catheter advancement risk region, or other extended occupied regions of the instrument, a conflict risk index is generated; and the conflict risk index is aggregated to output a conflict risk set.

[0088] Specifically, when discretizing the instrument pose flow and cross-instrument transformation set in a unified coordinate system, the 3D position and 3D orientation of the instrument pose flow at each moment are first represented in the form of a homogeneous transformation matrix. Then, the coordinates of all instrument ends are unified to the same reference coordinate system through the cross-instrument transformation set. Subsequently, the 3D space is divided into fixed-size cubic units, i.e., voxel units, and the voxel units occupied by the instrument ends are marked according to their position and orientation within consecutive time slices. The spatiotemporal occupancy voxel field generated by this process not only reflects the instrument's spatial occupancy but also preserves the instrument's dynamic trajectory in the time dimension. The spatiotemporal occupancy voxel field here represents a four-dimensional data structure, where three dimensions are spatial coordinates and the other dimension is the time axis.

[0089] When introducing the thermal diffusion safety radius parameter and catheter advancement safety radius parameter into the spatiotemporal occupancy voxel field, a safety buffer zone is first extended around each marked instrument tip voxel unit. For energy surgical instruments, based on their thermal diffusion characteristics, all voxel units within this radius are marked in three-dimensional space using the thermal diffusion safety radius parameter as the energy surgical instrument thermal risk region, thus reflecting the tissue range that the instrument may affect when using energy. For catheters, the advancement safety radius parameter is used to extend the voxels around the catheter tip position to form the catheter advancement risk region, reflecting the potential risk to surrounding tissues during catheter advancement in vivo. After this process, the spatiotemporal occupancy voxel field is extended into an extended spatiotemporal occupancy voxel field containing safety constraints, ensuring that subsequent predictions and risk assessments reflect the operational safety boundaries.

[0090] When performing short-time prediction based on the motion trajectory of a device's pose flow within consecutive time slices, the displacement vector and attitude change of the device's end effector over the most recent time slices are first extracted. A short-time prediction model is then established based on velocity and acceleration estimates. The prediction model uses kinematic extrapolation to obtain the predicted pose trajectory for several future time slices. This predicted pose trajectory represents the possible 3D position and 3D attitude of the device's end effector within the future time interval. Subsequently, these predicted pose trajectories are projected onto an extended spatiotemporal occupancy voxel field, and the voxel units covered by the predicted trajectories are marked, forming the predicted occupancy region. The predicted occupancy region is an estimate of the future spatial occupancy probability, providing a temporal lead time for conflict risk detection.

[0091] When performing spatial intersection detection within the predicted occupancy area, the thermal risk area of ​​energy surgical instruments, the catheter advancement risk area, and the expanded occupancy areas of other instruments are compared voxel-by-voxel. If spatial overlap of the expanded occupancy areas of different instruments is detected within the same time slice, it indicates that a conflict may occur in the future. A conflict risk index is thus generated, which includes the instrument identification of the conflict, the time interval of the conflict, the spatial voxel set of the conflict, the conflict severity level, and the probability of conflict. The probability of conflict can be estimated based on the statistical results of the deviation between the predicted trajectory and the historical trajectory, while the conflict severity level is classified according to the number of overlapping voxels and the location sensitivity of the conflict area.

[0092] When aggregating conflict risk indicators, the conflict risk indicators for all predicted time slices and all instrument pairs are merged to form a conflict risk set. This conflict risk set represents a structured collection containing a complete description of all potential conflicts between multiple instruments within a future timeframe, including instrument identification, conflict time interval, spatial overlap, risk level, and risk probability. This conflict risk set serves as direct input to the subsequent conflict resolution and scheduling modules, enabling real-time generation of scheduling instructions and safe time windows during surgery, thereby ensuring the operational safety and consistency of multi-instrument collaboration.

[0093] S160 solves the conflict risk set and the safety constraint set of standardized surgical procedure nodes together, and generates a scheduling instruction set and an executable safe time window based on the instrument's functional role and operational intent.

[0094] In one possible implementation, the conflict risk set and the safety constraint set of standardized surgical procedure nodes are jointly solved. A scheduling instruction set and an executable safe time window are generated based on the instrument's functional role and operational intent. Specifically, this includes: in the joint solution phase, aligning the instrument identifier, time interval, spatial range, risk level, and risk probability in the conflict risk set with the planned time, allowed operation set, and prohibited operation set in the standardized surgical procedure nodes using the time axis and ensuring consistent identifiers; treating conflict risks as conflict constraints within a unified optimization domain; solving the safety constraints as a combination of hard and soft constraints, outputting a joint solution result including constraint satisfaction labels and residual risk annotations; and in the scheduling generation phase, based on the joint solution result, generating a scheduling instruction set and an executable safe time window from the action template library according to the instrument's functional role and operational intent. The corresponding action elements are invoked, including speed limit, acceleration limit, path fine-tuning amplitude, energy threshold, viewpoint priority, waiting conditions, cancellation conditions, and rollback actions. The action elements are then checked for conflict and mutual exclusion and arranged in execution order according to the priority of process nodes to generate a structured scheduling instruction set. In the safe time window generation stage, based on the constraint satisfaction assessment provided by the joint solution results, continuous time slices that satisfy all hard safety boundaries and key soft constraints are determined in the spatiotemporal voxel field. Warning buffers and monitoring variable thresholds are set at the boundaries of each continuous time slice to form an executable safe time window with timestamps, device identifiers, action parameters, trigger conditions, cancellation conditions, and rollback actions. The executable safe time window and the scheduling instruction set are output one-to-one.

[0095] Specifically, in the joint solution phase, time axis alignment and identifier consistency are first completed. Using a unified time reference, instrument identifiers, time intervals, spatial ranges, risk levels, and risk probabilities are mapped to the planned times, permitted operation sets, and prohibited operation sets of standardized surgical procedure nodes. A unified optimization domain is established using a rolling time window, and conflict risks are incorporated into the solution as conflict constraints, represented as discrete-time sets. Represented by a set of instruments The action selection variable of each device in each time slice is represented as... The predicted voxel overlap volume is expressed as , expressed as the corresponding risk probability The conflict-constrained residuals are expressed as follows:

[0096]

[0097] in This indicates the minimum acceptable safety margin. If This indicates that conflict constraints are satisfied within the time slice. By simultaneously constraining all machine pairs and all time slices, an aligned set of candidate constraints is obtained and then used for synthesis. In the formula... A Boolean indication of any selected action of the instrument in the time slice. This represents the predicted space overlap volume of voxels. This represents the probability of occurrence given by the conflict risk set. Penalty residuals for conflict constraints This is the safety margin threshold.

[0098] When combining hard and soft constraints, a penalty-based constraint optimization model is used to solve for safety constraints. The set of hard constraints is represented as follows: Represented by a set of soft constraints The objective function is expressed as:

[0099]

[0100] in This indicates the positive part operator. This represents the penalty weight for soft constraints. Hard constraints cover mandatory conditions such as minimum instrument spacing, upper limits for energy and surgical instrument heat load, upper limits for catheter advancement rate, mutual exclusion of viewing angles, and sequence of workflow nodes. Soft constraints cover priority conditions such as surgical rhythm deviation, visual comfort, energy margin, and instrument movement smoothness. The solution outputs constraint satisfaction labels and residual risk labels. The constraint satisfaction label indicates the satisfaction status of each hard and soft constraint at each time slice. The residual risk label quantifies the probabilistic risk that still exists under the premise of satisfying hard constraints.

[0101] During the scheduling generation phase, based on the joint solution results and according to the instrument's functional role and operational intent, action elements are called from the action template library to form a structured scheduling instruction set. This is represented by the priority of the instrument's functional role. The priority of the operational intent is expressed as The residual risk cost is expressed as Expressed in terms of the cost of resource competition. Establish a scoring function for candidate actions:

[0102]

[0103] in , , and For normalized weights, executable actions are selected from high to low scores, and conflict and mutual exclusion checks and execution order scheduling are performed. Velocity limits, acceleration limits, path fine-tuning amplitude, energy thresholds, viewpoint priority, waiting conditions, cancellation conditions, and backtracking actions are written into each instruction entry, thereby generating a scheduling instruction set that can be directly executed by the robot control interface and the intraoperative interaction interface. In the formula... This indicates the level of responsibility of the instrument at the current process node. The importance of the process indicating the intent of the operation. Residual risk annotations from the joint solution. It measures the degree of competition among resources such as perspective, path, and energy in a time slice.

[0104] During the safe time window generation phase, based on the constraint satisfaction assessment provided by the joint solution results, a set of continuous time slices satisfying all hard safety boundaries and key soft constraints is determined in the spatiotemporal occupancy voxel field. This is expressed as an action feasibility indicator function:

[0105]

[0106] in For indicator functions, To allow for soft constraints, time slices that satisfy the indication of 1 are merged into a maximum consecutive time window to obtain an executable safe time window. Time buffers are added to both ends of each window. Furthermore, monitoring variable thresholds are set for heat load, propulsion rate, minimum spacing, and viewing angle occupancy to trigger secondary strategies when these thresholds are approached. Timestamps, instrument identifiers, action parameters, trigger conditions, cancellation conditions, and rollback actions are written into each window and output in a one-to-one correspondence with the scheduling instruction set. In the formula... Represents the hard constraint function. Represents the soft constraint function. This indicates the acceptable upper limit of the soft constraint. Indicates the length of the warning buffer zone.

[0107] S170 maps surgical field images, instrument pose flow, and patient physiological signal sequences to a surgical procedure knowledge graph and inputs them into a procedure recognition network. It compares standardized path templates with temporal attention, outputs the current procedure node and node confidence, and generates instrument collaboration consistency judgment results and correction suggestions by combining conflict risk set and scheduling instruction set.

[0108] In one possible implementation, surgical field images, instrument pose flow, and patient physiological signal sequences are mapped to a surgical procedure knowledge graph and input into a procedure recognition network. A standardized path template is compared using temporal attention, and the current procedure node and node confidence are output. Combined with a conflict risk set and a scheduling instruction set, instrument collaboration consistency judgment results and correction suggestions are generated. Specifically, this includes: encoding surgical field images into multimodal feature vectors containing anatomical structure boundaries, instrument position distribution, and photometric features, and mapping them to image semantic nodes in the surgical procedure knowledge graph; representing instrument pose flow as a continuous three-dimensional position and three-dimensional pose sequence in a unified coordinate system and mapping it to instrument operation type nodes in the surgical procedure knowledge graph; and normalizing the patient physiological signal sequence and synchronizing it with a timestamp. The process involves mapping to patient state type nodes in the surgical procedure knowledge graph; aggregating semantic relationships in the mapping results within the knowledge graph; maintaining the continuity between instrument pose flow and patient physiological signal sequences using temporal weights; outputting a temporal embedding containing procedure features; comparing the temporal embedding with the standardized path template using a temporal attention mechanism; calculating the similarity between the temporal embedding and the standardized path template on a time-slice basis; assigning attention weights to different time slices based on the similarity; and outputting the current procedure node and its confidence score; combining the current procedure node and its confidence score with the conflict risk set and scheduling instruction set for comprehensive judgment; when there is an inconsistency between the node confidence score and the conflict risk set or scheduling instruction set, generating an instrument collaboration consistency judgment result of inconsistency, and outputting correction suggestions as feedback information.

[0109] Specifically, when encoding surgical field images into multimodal feature vectors containing anatomical structure boundaries, instrument location distribution, and photometric features, photometric equalization, noise suppression, and edge enhancement are first performed on the surgical field images to make texture and boundary information significant while maintaining uniform brightness scale. Subsequently, a multi-branch feature extraction network trained on medical scene data simultaneously extracts anatomical structure boundaries, instrument location distribution, and photometric features. The three types of features are aligned in the channel dimension and concatenated under a unified coordinate system to form a multimodal feature vector. The multimodal feature vector refers to the joint expression of geometry, appearance, and intensity of the same image frame within the same timestamp. This vector is written into the image semantic nodes in the surgical procedure knowledge graph according to the image spatial coordinates, and the spatial adjacency relationships with anatomical structure nodes and instrument nodes, as well as the mapping relationships with photometric equalization parameters, are recorded in the form of edges, enabling subsequent retrieval to simultaneously read geometric and photometric semantics within the graph structure.

[0110] When representing the instrument pose flow as a continuous sequence of three-dimensional positions and postures in a unified coordinate system, the three-dimensional position and posture of the instrument end effector at each timestamp are time-synchronized and arranged into a pose sequence in chronological order. The pose sequence preserves the temporal continuity of translation and rotation in the form of a homogeneous transformation chain. The pose sequence is written into the instrument operation type node in the surgical procedure knowledge graph, and the spatiotemporal alignment relationship with image semantic nodes, the safe distance relationship with anatomical structure nodes, and the load relationship with energy constraint nodes are recorded by edge. The instrument operation type node is used to identify operation categories such as cutting and coagulation, traction and repositioning, ensuring that the actions of the same instrument at different times can be uniformly interpreted by the procedure recognition network.

[0111] After normalizing and synchronizing the patient's physiological signal sequence with timestamps, and mapping it to the patient state type node in the surgical procedure knowledge graph, the system performs dimensional unification, outlier removal, and missing value completion on ECG, blood pressure, respiratory rate, blood oxygen saturation, and other bedside monitoring signals. The signals are then resampled along a common time axis to a frequency consistent with the image frame rate and pose sampling rate. The patient state type node represents states such as stability, stress, increased risk of blood loss, and insufficient sedation. It is connected to risk factors in the procedure nodes via edges, enabling the procedure recognition network to simultaneously consider dynamic constraints from the physiological side when determining surgical steps.

[0112] When performing semantic relation aggregation on the mapping results in the surgical procedure knowledge graph and using temporal weights to maintain the continuity of instrument pose flow and patient physiological signal sequences, the features of image semantic nodes, instrument operation type nodes, and patient state type nodes are first aggregated on the graph using a message-passing method. Semantic relation aggregation refers to propagating and weighting node features along the edges of the graph structure to form a semantically consistent representation. Subsequently, temporal weights are introduced into the aggregated time series. Temporal weights refer to attenuation or enhancement coefficients applied to features based on the correlation of nearby time points, used to emphasize recent observations and suppress interference from distant time points. Finally, a temporal embedding containing procedure features is obtained. Temporal embedding refers to a vector sequence of multimodal features encoded by graph structure and modeled for temporal continuity on a unified time axis, which serves as the direct input for subsequent template comparison.

[0113] When comparing standardized path templates using a temporal attention mechanism, the temporal embedding and the standardized path template are first aligned on the timeline as two time series of equal length. Semantic similarity is then calculated for each time slice, where semantic similarity measures the degree of matching between the current observation and the template node within the feature space. The temporal attention mechanism assigns attention weights to each time slice based on the similarity; a higher attention weight indicates a greater contribution of that time slice to the current node's determination. The weighted summation result is input into the discriminator to obtain the current process node and its confidence score. The node confidence score is a probabilistic measure of the correctness of the determined node by the process recognition network; a higher value indicates a more reliable determination.

[0114] When combining the current process node and node confidence with the conflict risk set and scheduling instruction set for comprehensive judgment, the current process node is first associated with the conflict risk items and corresponding scheduling instruction items within the corresponding time window on the same time axis. The conflict risk set provides information on potential spatial overlap and safety boundary approximation, while the scheduling instruction set provides the scheduled action objectives and execution conditions. The comprehensive judgment uses node confidence as the primary indicator and is constrained by the conflict risk probability and remaining safety margin. If a contradiction is found between the node confidence and the conflict risk or the scheduling instruction, the device collaboration consistency judgment result is generated as inconsistent, and corrective suggestions are output as feedback information. The corrective suggestions include executable items such as action delay, path fine-tuning, perspective relocation, energy reduction, and priority adjustment, along with triggering conditions, cancellation conditions, and rollback actions, enabling the execution end to immediately implement corrections under the same coordinate semantics and the same temporal semantics.

[0115] This embodiment also discloses an automated surgical procedure management device based on multi-sensor fusion, referring to... Figure 2 The device includes an acquisition module 201, a processing module 202, and an output module 203. It is used to execute any of the multi-sensor fusion-based automated surgical procedure management methods described above, wherein:

[0116] The acquisition module 201 is used to acquire endoscopic image sequences, catheter pose flow, energy surgical instrument pose flow, surgical field environment depth, patient physiological signal sequences, and intraoperative voice events. It uses a unified timestamp mechanism to complete cross-sensor time alignment and obtain a synchronous data stream.

[0117] The processing module 202 is used to perform spatial geometry solving on the synchronous data stream based on the hand-eye calibration matrix, imaging intrinsic and extrinsic parameters, end-cap distortion model of the endoscope, and light source intensity response curve, to generate the instrument pose flow and surgical field imaging geometric model in a unified coordinate system, and output the cross-instrument transformation set.

[0118] Processing module 202 is used to jointly solve the laparoscopic image sequence and the surgical field imaging geometric model. It extracts pixel masks of instrument tip, forceps tip, catheter tip and anatomical structure through semantic segmentation, performs three-dimensional back projection by combining depth estimation and cross-instrument transformation set, constructs occlusion probability map and field of view map, and outputs view compensation candidate source set.

[0119] Processing module 202 is used to perform spatiotemporal registration of the main channel endoscopic image and the viewpoint compensation candidate source set under unified coordinates, perform cross-channel content replacement on the occluded area, and perform local reconstruction with photometric consistency and edge consistency as constraints, and output the compensated surgical field image.

[0120] Processing module 202 is used to discretize the instrument pose flow and cross-instrument transformation set into a spatiotemporal occupancy voxel field, introduce the thermal diffusion safety radius of energy surgical instruments and the catheter advancement safety radius, and generate a conflict risk set based on short-time prediction.

[0121] The processing module 202 is used to jointly solve the set of conflict risks and the set of safety constraints of standardized surgical procedure nodes, and generate a set of scheduling instructions and an executable safe time window based on the instrument function role and operation intention.

[0122] The output module 203 is used to map surgical field images, instrument pose flow and patient physiological signal sequence to the surgical process knowledge graph and input them into the process recognition network. It compares the standardized path template with temporal attention, outputs the current process node and node confidence, and generates instrument collaboration consistency judgment results and correction suggestions by combining the conflict risk set and scheduling instruction set.

[0123] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0124] This embodiment also discloses an electronic device, as shown in the reference. Figure 3 The electronic device may include: at least one processor 301, at least one communication bus 302, user interface 303, network interface 304, and at least one memory 305.

[0125] The communication bus 302 is used to enable communication between these components.

[0126] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0127] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0128] The processor 301 may include one or more processing cores. The processor 301 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 305, and by calling data stored in memory 305. Optionally, the processor 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 301 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications. The GPU is responsible for rendering and drawing the content required for display. The modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 301 and may be implemented as a separate chip.

[0129] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory may include a non-transitory computer-readable storage medium. The memory 305 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described above, etc. The data storage area may store data involved in the various method embodiments described above. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned processor 301. As a computer storage medium, the memory 305 may include an operating system, a network communication module, a user interface 303 module, and an application program for a multi-sensor fusion-based automated surgical procedure management method.

[0130] exist Figure 3In the illustrated electronic device, the user interface 303 is primarily used to provide an input interface for the user and acquire user input data. The processor 301 can be used to call an application program stored in the memory 305 that represents a multi-sensor fusion-based automated surgical procedure management method. When executed by one or more processors 301, the electronic device performs one or more methods as described in the above embodiments.

[0131] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0133] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory 305 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned memory 305 includes various media capable of storing program code, such as a USB flash drive, external hard drive, magnetic disk, or optical disk.

[0137] The present invention also discloses a computer-readable storage medium storing instructions. When executed by one or more processors 301, these instructions cause an electronic device to perform one or more methods as described in the above embodiments.

[0138] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the specification and the disclosure of practical truths. This invention is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A multi-sensor fusion-based automated surgical procedure management device, characterized in that, The device includes an acquisition module, a processing module, and an output module, wherein: The acquisition module is used to collect laparoscopic image sequences, catheter pose flow, energy surgical instrument pose flow, surgical field environment depth, patient physiological signal sequences, and intraoperative voice events. It uses a unified timestamp mechanism to complete cross-sensor time alignment and obtain a synchronous data stream. The processing module is used to perform spatial geometry solving on the synchronous data stream based on the hand-eye calibration matrix, imaging intrinsic and extrinsic parameters, end-cap distortion model of the endoscope, and light source intensity response curve, generating an instrument pose flow and surgical field imaging geometry model in a unified coordinate system, and outputting a cross-instrument transformation set; jointly solving the endoscope image sequence and the surgical field imaging geometry model, extracting pixel masks of instrument tips, forceps tips, catheter tips, and anatomical structures through semantic segmentation, combining depth estimation and the cross-instrument transformation set for 3D retroprojection, constructing an occlusion probability map and a field of view map, and outputting a viewpoint compensation candidate source set; and processing the main channel endoscope image... Spatiotemporal registration is performed with the aforementioned viewpoint compensation candidate source set under unified coordinates. Cross-channel content replacement is performed on the occluded area, and local reconstruction is performed with photometric consistency and edge consistency as constraints to output the compensated surgical field image. The instrument pose flow and the cross-instrument transformation set are discretized into a spatiotemporal occupancy voxel field. The thermal diffusion safety radius of energy surgical instruments and the catheter advancement safety radius are introduced, and a conflict risk set is generated based on short-time prediction. The conflict risk set is jointly solved with the safety constraint set of standardized surgical procedure nodes, and a scheduling instruction set and an executable safe time window are generated according to the instrument function role and operation intention. The output module is used to map the surgical field image, the instrument pose flow and the patient physiological signal sequence to the surgical process knowledge graph and input them into the process recognition network. It compares the standardized path template with temporal attention, outputs the current process node and node confidence, and generates instrument collaboration consistency judgment results and correction suggestions by combining the conflict risk set and the scheduling instruction set. The process involves jointly solving the laparoscopic image sequence with the surgical field imaging geometry model, extracting pixel masks for the instrument tip, forceps tip, catheter tip, and anatomical structures through semantic segmentation, and performing 3D retrocasting using depth estimation and the cross-instrument transformation set to construct an occlusion probability map and a view area map, and outputting a viewpoint compensation candidate source set. Specifically, this includes: performing semantic segmentation on the laparoscopic image sequence in a unified coordinate system to extract pixel masks for the instrument tip, forceps tip, catheter tip, and anatomical structures; and performing 3D retrocasting using depth estimation parameters from the surgical field imaging geometry model and the cross-instrument transformation set to generate a continuous surgical field point cloud and a 3D pose sequence for the instrument tip. Based on the continuous surgical field point cloud and the three-dimensional pose sequence of the instrument end, an occlusion probability map sequence is constructed, and a visible field atlas is generated by combining the occlusion probability map sequence. The visible field atlas is used to quantify the potential visible coverage of the invisible area of ​​the main channel laparoscope on each secondary channel laparoscope. Based on the visible field atlas and the occlusion probability map sequence, candidate sources are sorted according to coverage area ratio, baseline angle, photometric consistency, feature matching confidence, and temporal proximity. Secondary channel information that meets the requirements of cross-channel registration accessibility and structural consistency is selected, and a viewpoint compensation candidate source set consisting of secondary channel identifier, coverage area index, expected registration error, expected photometric error, and timestamp range is output.

2. The multi-sensor fusion-based automated surgical procedure management device according to claim 1, characterized in that, Based on the hand-eye calibration matrix, imaging intrinsic and extrinsic parameters, end-cap distortion model of the endoscope, and light source intensity response curve, spatial geometry is solved on the synchronous data stream to generate an instrument pose flow and surgical field imaging geometry model in a unified coordinate system, and outputs a cross-instrument transformation set, specifically including: The instrument pose flow is projected onto the endoscopic imaging coordinate system and mapped to a unified reference coordinate system using the hand-eye calibration matrix. Geometric calculations are performed on endoscopic image sequences using imaging intrinsic and extrinsic parameters to establish a correspondence between image coordinates and physical coordinates; The laparoscopic image sequence was distorted using the aforementioned end-cap distortion model to obtain a corrected image with consistent edge straightness and proportions. The light source intensity response curve is used to perform photometric correction on the endoscopic image sequence to obtain a brightness-normalized image intensity distribution. The instrument pose flow processed by the hand-eye calibration matrix, the imaging intrinsic and extrinsic parameters, the end-cap distortion model of the laparoscopy, and the light source intensity response curve are jointly solved with the laparoscopic image sequence in a unified coordinate system to generate a surgical field imaging geometric model and output a cross-instrument transformation set.

3. The multi-sensor fusion-based automated surgical procedure management device according to claim 1, characterized in that, The process of performing spatiotemporal registration of the main channel endoscopic image and the viewpoint compensation candidate source set under unified coordinates, performing cross-channel content replacement on the occluded area, and performing local reconstruction with light intensity consistency and edge consistency as constraints, and outputting the compensated surgical field image specifically includes: The candidate inputs of the main channel endoscopic image are determined using the viewpoint compensation candidate source set. The candidate inputs are secondary channel endoscopic images that are temporally adjacent to the main channel endoscopic image and have satisfactory coverage and visibility. Based on the surgical field imaging geometric model and the imaging parameters and geometric constraints provided by the cross-instrument transformation set, cross-channel spatiotemporal registration is performed in a unified coordinate system to output the first compensated image. After completing the spatiotemporal registration, cross-channel content replacement is performed on the pixels in the occluded area of ​​the main channel endoscopic image. The pixels in the corresponding position of the sub-channel endoscopic image are weighted according to coverage, reprojection error and temporal proximity, and the occluded area in the main channel endoscopic image is replaced by weighted fusion to output the second compensation image. After cross-channel content replacement, the light intensity consistency constraint ensures a smooth transition of local brightness and texture, the edge consistency constraint ensures the continuity of instrument boundaries and anatomical structure boundaries, and local reconstruction is performed in the occluded area to generate a third compensation image with light intensity balance and edge continuity. Finally, the first compensated image and the second compensated image are combined to ensure that the main channel image and the sub-channel image are aligned in time and space under a unified coordinate system, so as to achieve photometric smoothing and continuous edge repair of the local occluded area. The result is output as a compensated surgical field image for subsequent path conflict detection and surgical procedure consistency judgment.

4. The multi-sensor fusion-based automated surgical procedure management device according to claim 1, characterized in that, The process of discretizing the instrument pose flow and the cross-instrument transformation set into a spatiotemporal occupancy voxel field, introducing the thermal diffusion safety radius of energy surgical instruments and the catheter advancement safety radius, and generating a conflict risk set based on short-time prediction specifically includes: By discretizing the instrument pose flow and the cross-instrument transformation set in a unified coordinate system, the instrument position and instrument posture of continuous time slices are mapped into a spatiotemporal occupancy voxel field. In the spatiotemporal occupied voxel field, a thermal diffusion safety radius parameter for energy surgical instruments is introduced to form a thermal risk region for energy surgical instruments, and a catheter advancement safety radius parameter is introduced to form a catheter advancement risk region, thereby generating an extended spatiotemporal occupied voxel field; Short-term prediction is performed based on the motion trajectory of the device pose flow in a continuous time slice to obtain the predicted pose trajectory in a few future time slices, and the predicted pose trajectory is mapped to the extended spatiotemporal occupied voxel field to form a predicted occupied region. Spatial intersection detection is performed within the predicted occupancy area. If spatial overlap is detected between the thermal risk area of ​​energy surgical instruments, the catheter advancement risk area, or the occupancy area of ​​other instruments, a conflict risk index is generated. The conflict risk indicators are aggregated to output the conflict risk set.

5. The multi-sensor fusion-based automated surgical procedure management device according to claim 1, characterized in that, The process of jointly solving the conflict risk set and the safety constraint set of standardized surgical procedure nodes, and generating a scheduling instruction set and an executable safe time window based on the instrument's functional role and operational intent, specifically includes: During the joint solution phase, the instrument identification, time interval, spatial range, risk level and risk probability of the conflict risk set are aligned with the planned time, allowed operation set and prohibited operation set in the standardized surgical procedure node and the identification is made consistent. The conflict risk is treated as a conflict constraint in the unified optimization domain. The safety constraints are solved as a combination of hard and soft constraints, and the output is a joint solution that includes constraint satisfaction labels and residual risk labels. In the scheduling generation phase, based on the joint solution results, the corresponding action elements are called from the action template library according to the functional role and operation intention of the device. The action elements include speed limit, acceleration limit, path fine-tuning range, energy threshold, view priority, waiting condition, cancellation condition and backtracking action. The action elements are then checked for conflict and mutual exclusion and arranged in the execution order according to the priority of process nodes to generate a structured scheduling instruction set. During the safe time window generation phase, based on the constraint satisfaction assessment provided by the joint solution results, continuous time slices that satisfy all hard safety boundaries and key soft constraints are determined in the spatiotemporal occupancy voxel field. Warning buffers and monitoring variable thresholds are set at the boundaries of each continuous time slice to form an executable safe time window with timestamps, device identifiers, action parameters, trigger conditions, cancellation conditions, and rollback actions. The executable safe time window is output in a one-to-one correspondence with the scheduling instruction set.

6. The multi-sensor fusion-based automated surgical procedure management device according to claim 1, characterized in that, The process of mapping the surgical field image, the instrument pose flow, and the patient's physiological signal sequence to a surgical procedure knowledge graph and inputting it into a procedure recognition network, comparing standardized path templates with temporal attention, outputting the current procedure node and node confidence, and combining the conflict risk set and the scheduling instruction set to generate instrument collaboration consistency judgment results and correction suggestions, specifically includes: The surgical field images are encoded into multimodal feature vectors containing anatomical structure boundaries, instrument location distribution, and photometric features, and mapped to image semantic nodes in the surgical procedure knowledge graph. The instrument pose flow is represented as a continuous three-dimensional position and three-dimensional pose sequence in a unified coordinate system and mapped to the instrument operation type node in the surgical procedure knowledge graph. After the patient's physiological signal sequence is normalized and synchronized with a timestamp, it is mapped to the patient state type node in the surgical procedure knowledge graph. Semantic relationship aggregation is performed on the mapping results in the surgical procedure knowledge graph, and time weights are used to maintain the continuity between the instrument pose flow and the patient physiological signal sequence, outputting a temporal embedding containing procedure features; The time-series attention mechanism is used to compare the standardized path template. The similarity between the time-series embedding and the standardized path template is calculated time-by-time. Attention weights are assigned to different time-time slices based on the similarity, and the current process node and node confidence are output. The current process node and the node confidence are combined with the conflict risk set and the scheduling instruction set for comprehensive judgment. When the node confidence is inconsistent with the conflict risk set or the scheduling instruction set, the instrument cooperation consistency judgment result is generated as inconsistent, and the correction suggestion is output as feedback information.

Images