An intraoperative real-time navigation and early warning system and method for colorectal cancer surgery

Abstract

The application relates to the technical field of automatic control of surgical navigation, and discloses an intraoperative real-time navigation and early warning system and method for colorectal cancer surgery, which comprises the following modules: a collection module, which is used for collecting preoperative data before the execution of the colorectal cancer surgery and collecting intraoperative data during the execution of the colorectal cancer surgery; a preprocessing module, which is used for processing the preoperative data to obtain preoperative risk data and processing the intraoperative data to obtain an intraoperative point cloud set; a registration module, which is used for performing spatial registration of the semantic dimension on the intraoperative data and the preoperative data based on the preoperative risk data and the intraoperative point cloud set to obtain a registration result; and a navigation and early warning module, which is used for generating navigation data for a surgical instrument based on the registration result and generating and outputting early warning data associated with the surgical instrument during the execution of the colorectal cancer surgery. The scheme can meet the actual needs of improving the safety of the colorectal cancer surgery and improving the surgical effect.

Description

Technical Field

[0001] This application relates to the field of automatic control technology for surgical navigation, specifically to, but not limited to, an intraoperative real-time navigation and early warning system and method for colorectal cancer surgery. Background Technology

[0002] During colorectal cancer surgery, blurred tumor boundaries, incomplete lymph node removal, and high risk of damage to adjacent tissues are the main factors affecting the safety and outcome of the surgery. Therefore, precise real-time navigation of surgical instruments is crucial during colorectal cancer surgery.

[0003] In practical applications, in order to improve the safety and surgical outcomes of colorectal cancer surgery, static computed tomography (CT) images or magnetic resonance imaging (MRI) images of the lesion site of the colorectal region need to be acquired before colorectal cancer surgery is performed. Based on the CT images or MRI images and the patient's condition reflected therein, the surgical strategy is determined and the lesion site is treated according to the surgical strategy.

[0004] However, in actual colorectal cancer surgery, factors such as colorectal tissue deformation and surgical instrument movement may change dynamically at any time. This means that the surgical strategies determined by the above methods cannot meet the actual needs of improving the safety and surgical outcomes of colorectal cancer surgery. Summary of the Invention

[0005] Based on the above technical problems, this application provides an intraoperative real-time navigation and early warning system and method for colorectal cancer surgery, which can meet the actual needs of improving the safety and surgical outcomes of colorectal cancer surgery.

[0006] The technical solution provided in this application is as follows: This application first provides an intraoperative real-time navigation and early warning system for colorectal cancer surgery, including: The data acquisition module is used to acquire preoperative data before the colorectal cancer surgery is performed and to track and acquire intraoperative data during the colorectal cancer surgery; wherein, the preoperative data includes images of at least a portion of the lesion site in the colorectal region; the intraoperative data includes image data of the colorectal cancer surgery process; and the lesion site is associated with the colorectal cancer. The preprocessing module is used to process the preoperative data to obtain preoperative risk data and to process the intraoperative data to obtain an intraoperative point cloud set. The preoperative risk data includes the distribution status of the lesion site and the first semantic category of the site associated with the lesion site. The intraoperative point cloud set includes the point cloud set corresponding to the intraoperative data and the second semantic category corresponding to the point cloud in the point cloud set. The registration module is used to perform semantic-dimensional spatial registration on the intraoperative data and the preoperative data based on the preoperative risk data and the intraoperative point cloud set, and obtain the registration result; The navigation and early warning module is used to generate navigation data for surgical instruments based on the registration results, and to generate and output early warning data associated with the surgical instruments during the colorectal cancer surgery.

[0007] This application also provides an intraoperative real-time navigation and early warning method for colorectal cancer surgery, including: Preoperative data is collected before the colorectal cancer surgery is performed, and intraoperative data is collected during the colorectal cancer surgery; wherein, the preoperative data includes images of at least a portion of the lesion site in the colorectal region; the intraoperative data includes image data of the colorectal cancer surgery process; the lesion site is associated with the colorectal cancer; The preoperative data is processed to obtain preoperative risk data, and the intraoperative data is processed to obtain an intraoperative point cloud set; wherein, the preoperative risk data includes the distribution status of the lesion site and the first semantic category of the site associated with the lesion site; the intraoperative point cloud set includes the point cloud set corresponding to the intraoperative data and the second semantic category corresponding to the point cloud in the point cloud set. Based on the preoperative risk data and the intraoperative point cloud set, spatial registration of the semantic dimension is performed on the intraoperative data and the preoperative data to obtain the registration result; Based on the registration results, navigation data for the surgical instruments is generated, and during the colorectal cancer surgery, warning data associated with the surgical instruments is generated and output.

[0008] The intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application has at least the following beneficial effects: In the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the acquisition module collects preoperative data before the colorectal cancer surgery is performed and tracks and collects intraoperative data during the colorectal cancer surgery. The preoperative data includes image data of at least part of the lesion site in the colorectal region, and the intraoperative data includes image data of the colorectal cancer surgery process, thus improving the comprehensiveness of data acquisition by the acquisition module. Furthermore, the preprocessing module processes the preoperative data to obtain preoperative risk data, which includes the distribution status of the lesion site and the first semantic category of the associated sites. Thus, the preoperative risk data can achieve a comprehensive representation of the distribution status of the lesion site, the semantic category of the lesion site and the associated sites. At the same time, the preprocessing module processes the intraoperative data to obtain an intraoperative point cloud set, which includes the point cloud set corresponding to the intraoperative data and the point cloud set. The point cloud in the cloud set corresponds to the second semantic category. Thus, the intraoperative point cloud set enables a three-dimensional and intuitive semantic description of the imaging data during the colorectal cancer surgery process. Based on this, the registration module performs spatial registration of the intraoperative and preoperative data in the semantic dimension based on the preoperative risk data and the intraoperative point cloud set, obtaining the registration result. This achieves dual registration of intraoperative and preoperative data in both spatial and semantic dimensions, allowing the registration result to comprehensively and accurately represent the degree of difference between the intraoperative and preoperative data. On the other hand, the navigation and warning module generates navigation data for surgical instruments based on the registration result and generates and outputs warning data associated with the surgical instruments. This improves the accuracy of navigation and warning data, thereby enhancing the safety and surgical outcome of colorectal cancer surgery and meeting the actual surgical needs of colorectal cancer surgery. Attached Figure Description

[0009] Figure 1 A schematic diagram of the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in an embodiment of this application; Figure 2 A schematic diagram of the registration module in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in the embodiments of this application; Figure 3 Another structural schematic diagram of the intraoperative real-time navigation and early warning system provided in the embodiments of this application; Figure 4 This is a flowchart illustrating the intraoperative real-time navigation and early warning method provided in an embodiment of this application. Detailed Implementation

[0010] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0011] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0012] During colorectal cancer surgery, blurred tumor boundaries, incomplete lymph node removal, and high risk of damage to adjacent tissues are the main factors affecting the safety and outcome of the surgery. Therefore, precise real-time navigation of surgical instruments is crucial during colorectal cancer surgery.

[0013] In practical applications, in order to improve the safety and surgical outcomes of colorectal cancer surgery, static CT or MRI scans of the lesion site in the colorectal region are required before the surgery is performed. Based on the CT or MRI images and the patient's condition reflected therein, a surgical strategy is determined, and the lesion site is treated accordingly.

[0014] However, in actual colorectal cancer surgery, factors such as colorectal tissue deformation and surgical instrument movement may change dynamically at any time. This means that the surgical strategies determined by the above methods cannot meet the actual needs of improving the safety and surgical outcomes of colorectal cancer surgery.

[0015] Based on the above technical problems, this application provides an intraoperative real-time navigation and early warning system and method for colorectal surgery.

[0016] Figure 1 This is a schematic diagram of the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in the embodiments of this application, as shown below. Figure 1 As shown, the intraoperative real-time navigation and early warning system 100 may include: The data acquisition module 101 is used to acquire preoperative data before colorectal cancer surgery and to track and acquire intraoperative data during colorectal cancer surgery; wherein, the preoperative data includes images of at least part of the lesion site of colorectal cancer; the intraoperative data includes image data during the colorectal cancer surgery; the lesion site is associated with colorectal cancer; The preprocessing module 102 is used to process preoperative data to obtain preoperative risk data and to process intraoperative data to obtain an intraoperative point cloud set. The preoperative risk data includes the distribution status of the lesion site and the first semantic category of the site associated with the lesion site. The intraoperative point cloud set includes the point cloud set corresponding to the intraoperative data and the second semantic category corresponding to the point cloud in the point cloud set. The registration module 103 is used to spatially register intraoperative data and preoperative data based on preoperative risk data and intraoperative point cloud set to obtain registration results. The navigation and early warning module 104 is used to generate navigation data for surgical instruments based on the registration results, and to generate and output early warning data associated with surgical instruments during the colorectal cancer surgery.

[0017] In some embodiments, patient images may include CT images or MRI images; exemplaryly, patient images may three-dimensionally characterize the pixel features of the patient site and its associated sites; exemplaryly, associated sites may include at least one tissue that is directly or indirectly adjacent to the patient site; exemplaryly, at least one tissue may include tumors, arteries, veins, nerves, intestinal walls, ureters, and adipose tissue, etc.

[0018] In some embodiments, the lesion site may include the location of the tumor corresponding to colorectal cancer, and may also include at least a portion of the colorectal tissue adjacent to the tumor.

[0019] In some embodiments, the image data may include video data or multiple frames of continuous image data for real-time tracking and recording of the execution status of colorectal cancer surgery; for example, the execution status may include the position and / or posture of surgical instruments, the morphology of the lesion site, the morphology of related sites, and the amount of bleeding during the execution of colorectal cancer surgery.

[0020] In some embodiments, intraoperative data can be tracked and collected in real time during colorectal cancer surgery.

[0021] In some embodiments, the distribution of the lesion site may include at least one of the location, area, shape, and size of the lesion site in the colon and rectum.

[0022] In some embodiments, the first semantic category may include the semantic category of the tissue to which the associated site belongs; for example, the semantic category of the tissue may include fat, artery, vein, nerve, ureter, and intestinal wall, etc.

[0023] In some embodiments, preoperative risk data can be represented in matrix form, characterizing the distribution of lesion sites, the distribution of associated sites, and the first semantic category, etc.; specifically, preoperative risk data can be obtained in the following ways: Preoperative risk data is obtained by performing fine segmentation of the lesion site and related sites in preoperative data using a 3D medical image segmentation network and performing semantic recognition on the lesion site and related sites. For example, the 3D medical image segmentation network may include 3D U-Net++.

[0024] Specifically, preoperative data is identified and segmented at the voxel level using 3D U-Net++ to obtain a probability set corresponding to the first semantic category. Then, the probabilities in the probability set are processed by thresholding and connected component analysis to obtain a discrete 3D voxel mask with the first semantic category as the semantic label. Next, the discrete 3D voxel mask is converted into a continuous triangular mesh model. Based on the mapping relationship between the vertices in the triangular mesh model and the voxels in the 3D voxel mask, and based on the first semantic category corresponding to the voxels in the 3D voxel mask, the corresponding semantic labels are set for the vertices in the triangular mesh model. Finally, the corresponding risk coefficients are set for the vertices in the triangular mesh model according to the risk coefficients corresponding to the first semantic category, thereby obtaining preoperative risk data composed of the vertex set, edge set, triangular facet set of the triangular mesh model, and the first semantic category.

[0025] The probability set may include the probabilities of various semantic categories contained in the first semantic category; each voxel in the three-dimensional voxel mask contains the semantic category in its corresponding first semantic category.

[0026] In some embodiments, the semantic categories contained in the second semantic category may be the same as those contained in the first semantic category.

[0027] In some embodiments, the intraoperative point cloud set can be obtained in the following ways: By combining the Multi-View Stereo (MVS) method with visual inertial odometry, real-time reconstruction of the acquired image data is performed to obtain a dense 3D point cloud of the current field of view, which is then defined as a point cloud set. Simultaneously, the image data is processed using DeepLabV3+ Mobile to obtain pixel-level semantic segmentation results, which are then defined as the second semantic category. Next, using the depth information of the image data acquired by the image acquisition device used to acquire intraoperative data, the second semantic category is back-mapped to each point cloud in the point cloud set, thus obtaining the intraoperative point cloud set.

[0028] In some embodiments, spatial registration may include mapping and transforming the preoperative data of the k-th semantic in a spatial dimension, such that the transformed preoperative data of the k-th semantic can be associated with or consistent with the intraoperative data of the k-th semantic; or, shifting and rotating the intraoperative data of the k-th semantic in a spatial dimension, such that the transformed intraoperative data of the k-th semantic can be associated with or consistent with the preoperative data of the k-th semantic; wherein, k can be an integer greater than or equal to 1, and the k-th semantic can be the k-th semantic category in the first semantic category and the second semantic category.

[0029] In some embodiments, the registration result can characterize the intraoperative data after performing spatial registration in the semantic dimension and the differences between it and the preoperative data.

[0030] In some embodiments, the registration result can be obtained in the following way: The preoperative data is segmented according to the first semantic category in the preoperative risk data to obtain the preoperative segmentation result. The intraoperative data is segmented according to the second semantic category in the intraoperative point cloud set to obtain the intraoperative segmentation result. Then, the spatial dimension mapping association and transformation are performed on the preoperative segmentation result and / or the intraoperative segmentation result, and the transformed preoperative segmentation result and / or the transformed intraoperative segmentation result are determined as the registration result.

[0031] In some embodiments, navigation data may include data used to dynamically guide the displacement and / or rotation of surgical instruments in at least one direction during colorectal cancer surgery.

[0032] In some embodiments, the warning data may include data on the potential threat that the displacement and / or rotation of surgical instruments may pose to the target tissue during colorectal cancer surgery. For example, the target tissue may include at least one tissue in which the severity of post-injury complications is greater than or equal to a severity threshold, the mortality and disability rate is greater than or equal to a first probability threshold, the repair difficulty is greater than or equal to a difficulty threshold, and the probability of accidental injury during surgery is greater than or equal to a second probability threshold. For example, the target tissue may include arteries and urinary catheters.

[0033] In some embodiments, navigation data can be generated in the following ways: During colorectal cancer surgery, the instrument status of the surgical instruments is dynamically tracked, and a first relative state and a second relative state with respect to the target tissue are evaluated based on the instrument status. Then, based on the first and second relative states, first control data for controlling the surgical instruments to continuously approach the patient site and second control data for controlling the distance between the surgical instruments and the target tissue to be greater than or equal to a distance threshold are determined. Navigation data is generated based on the first and second control data. For example, the instrument status may include the instrument pose, the first relative state may include the relative position between the surgical instruments and the patient site in at least one direction, and the second relative state may include the relative position between the surgical instruments and the target tissue in at least one direction.

[0034] Specifically, navigation data can be generated in the following ways: First, preoperative data is processed to obtain preoperative point cloud data. If the registration result involves transformation of the preoperative data, the preoperative point cloud data is transformed to obtain a transformed preoperative semantic point cloud. The truncated signed distance function (TSDF) is then used to map the transformed preoperative point cloud data and the intraoperative point cloud set to a unified voxel space. The above process may include the following steps: Step S1: Process the preoperative point cloud. Details are shown below: The preoperative semantic point cloud carrying the first semantic category is converted into a preoperative voxel mesh. An SDF value is assigned to each preoperative voxel in the mesh using the Signed Distance Function (SDF), initialized to the TSDF value from the center of the voxel to the surface of its nearest tissue or organ. The distance weights corresponding to the signed distances of each voxel are initialized to 1. Next, the semantic category with the highest frequency in the preoperative voxels is determined as the semantic category corresponding to the voxel, and the risk coefficient corresponding to the aforementioned semantic category is determined as the risk coefficient corresponding to the voxel. The TSDF value is truncated to... The risk coefficient can be measured by at least one of the following factors: the severity of complications after injury, the mortality and disability rate, the difficulty of repair, and the probability of accidental injury during surgery. For example, the repair difficulty of arteries can be greater than that of adipose tissue. Therefore, the risk coefficient of arteries can be greater than that of adipose tissue.

[0035] Step S2: Process the intraoperative point cloud dataset. Details are as follows: The point cloud in the intraoperative point cloud set acquired in real time is projected onto the preoperative voxel grid, and the temporary SDF value corresponding to the voxel in the preoperative voxel grid of the projected intraoperative point cloud set is calculated. The SDF value of the voxel can also be updated based on the temporary SDF value. The specific process of updating the SDF value can be shown in Equation (1): (1) in, Voxels in the preoperative voxel grid. For the j-th point cloud in the intraoperative point cloud set, for Confidence weights For The weight value, This represents the update of the aforementioned weight values; that is, the weight value can be updated once a voxel v is observed. voxels The voxel centers are defined, where t is the current time and t+1 is the next time step. This is a temporary SDF value. The SDF value at time t+1 This is the currently stored SDF value.

[0036] For example, the weighted average processing shown in Equation (1) allows the geometric position of the voxels to gradually converge to the tissue surface represented by the intraoperative point cloud set.

[0037] Step S3, Semantic Category and Risk Inheritance. Details are as follows: For example, the first semantic category corresponding to the preoperative semantic point cloud is set to a voxel in the preoperative voxel grid of the projected intraoperative point cloud set; and if a single voxel contains multiple different semantic categories, the semantic category corresponding to a single voxel can be determined by the majority voting method based on the volume.

[0038] Step S4: Zero-intersection surface extraction. Details are as follows: After performing steps S1 to S3 on the intraoperative point cloud set corresponding to multiple frames of image data, the marchingcubes algorithm is used to extract zero isosurfaces from the TSDF field composed of TSDF values ​​to obtain the fused dynamic surface geometry. At the same time, each surface vertex in the dynamic surface set inherits the semantic label and risk value of its voxel, forming a dynamic intraoperative map. The dynamic intraoperative map is stored in the form of a voxel grid, with each voxel containing the fused voxel center, semantic category and its corresponding risk coefficient.

[0039] Real-time path planning is performed based on the dynamic intraoperative map, and the direction, speed, and risk represented by the path planning are determined as navigation data. Specifically, the current tip position of the surgical instrument can be used as the starting point of the path and the patient site as the ending point. The Fast Marching Method (FMM) is used to calculate the dynamic intraoperative map, and the wavefront propagation process from the starting point is simulated by solving the equation. In order to automatically avoid high-risk areas in the path planning, the speed function is defined as a decreasing function of the risk coefficient as shown in equation (2). (2) in, The base velocity constant is used to control the wavefront propagation speed in non-disease tissues or structures. A larger value makes path planning tend to be shorter. This is a risk adjustment coefficient used to control the degree to which risk inhibits speed. The larger the value, the stronger the tendency of the path planning to avoid high-risk organizations. Indicates position The risk coefficient at the location can be obtained by interpolation from the dynamic intraoperative map.

[0040] Equation (2) enables the location The higher the risk coefficient, the lower the wavefront propagation speed, and the higher the corresponding path cost. Thus, navigation data enables safe, stable, and precise navigation of surgical instruments during the surgical process.

[0041] In some embodiments, warning data can be generated in the following ways: During colorectal cancer surgery, the position of surgical instruments is dynamically tracked in real time using a dynamic intraoperative map. The spatial position of the instruments relative to the lesion and target tissue is assessed in real time based on their position. Then, based on the spatial position and a set of safety thresholds, a variety of warning levels are determined and warning data corresponding to the warning levels is output.

[0042] Specifically, risky tissues with risk coefficients greater than or equal to a threshold are marked on a dynamic intraoperative map. The minimum distance between the surgical instrument and the risky tissue is determined based on the instrument's position. Then, the current risk level of the surgical instrument is determined based on the relationship between the minimum distance and the safety thresholds in the safety threshold set, and warning data corresponding to the risk level is output. For example, the safety threshold set may include a first distance threshold, a second distance threshold, and a third distance threshold. The first distance threshold may be greater than the second distance threshold, and the second distance threshold may be greater than the third distance threshold. If the minimum distance is greater than or equal to the second distance threshold and less than the first distance threshold, the risk level of the surgical instrument is level one. If the minimum distance is less than the second distance threshold and greater than or equal to the third distance threshold, the risk level of the surgical instrument is level two. If the minimum distance is less than the third distance threshold, the risk level may be level three. The risk levels represented by the first, second, and third levels may show an increasing trend.

[0043] In some embodiments, the warning data may be presented in at least one of the following forms: sound, text, video, image, and vibration. This application does not limit the specific form of the warning data.

[0044] As can be seen from the above, in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the acquisition module collects preoperative data before the colorectal cancer surgery is performed and tracks and collects intraoperative data during the colorectal cancer surgery. The preoperative data includes image data of at least part of the lesion site in the colorectal region, and the intraoperative data includes image data of the colorectal cancer surgery process. This improves the comprehensiveness of data acquisition by the acquisition module. Furthermore, the preprocessing module processes the preoperative data to obtain preoperative risk data, which includes the distribution status of the lesion site and the first semantic category of the associated sites. Thus, the preoperative risk data can achieve a comprehensive representation of the distribution status of the lesion site, the semantic category of the lesion site and the associated sites. At the same time, the preprocessing module processes the intraoperative data to obtain an intraoperative point cloud set, which includes the point cloud set corresponding to the intraoperative data. The intraoperative point cloud set, along with the corresponding second semantic category, enables a three-dimensional and intuitive semantic description of the imaging data during colorectal cancer surgery. Based on this, the registration module performs spatial registration of the intraoperative and preoperative data in the semantic dimension, using preoperative risk data and the intraoperative point cloud set. This achieves dual registration of intraoperative and preoperative data in both spatial and semantic dimensions, allowing the registration results to comprehensively and accurately represent the degree of difference between the two data. Furthermore, the navigation and warning module generates navigation data for surgical instruments based on the registration results and generates and outputs warning data associated with the surgical instruments. This improves the accuracy of both navigation and warning data, thereby enhancing the safety and outcome of colorectal cancer surgery and meeting the actual surgical needs.

[0045] Figure 2 A schematic diagram of the registration module for an intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment is shown below. Figure 2 As shown, the registration module of the intraoperative real-time navigation and early warning system 100 includes a coarse registration unit 1031, which is used to extract global features from the intraoperative point cloud set to obtain intraoperative global features, and to extract features from the triangular mesh model corresponding to the preoperative risk data to obtain the preoperative local feature set. The coarse registration unit 1031 is also used to determine a candidate feature set from the preoperative local feature set based on the degree of matching between the intraoperative global features and the local features in the preoperative local feature set, and to determine the first transformation matrix between the preoperative risk data and the intraoperative point cloud set based on the intraoperative global features and the candidate feature set. The coarse registration unit 1031 is also used to transform the preoperative risk data based on the first transformation matrix to obtain the first registration data; wherein the registration result includes the first registration data.

[0046] In some embodiments, intraoperative global features can characterize the overall three-dimensional morphological features of the structure or tissue corresponding to the intraoperative data at the current moment; for example, the structure or tissue may include the intestinal segment, tumor outline, and blood vessels processed in colorectal surgery.

[0047] In some embodiments, intraoperative global features can be obtained in the following ways: The first extraction network performs global feature extraction on the intraoperative point cloud set to obtain intraoperative global features; for example, the first extraction network may include PointNet.

[0048] In some embodiments, before obtaining the preoperative local feature set, a second extraction network can be used to extract global features from the triangular mesh model corresponding to the preoperative risk data to obtain preoperative global features; for example, the second extraction network can be 3D U-Net++.

[0049] In some embodiments, the preoperative local feature set may include a set of features corresponding to local regions in the local region set; exemplarily, the local region set can be obtained by spatially partitioning a triangular mesh model; correspondingly, the local features in the preoperative local feature set can be obtained in the following ways: Max pooling is performed on the set of vertex features contained in the m-th local region to obtain the m-th local features; where m is an integer greater than or equal to 1.

[0050] In some embodiments, the candidate feature set can be determined in the following ways: The degree of global matching between preoperative and intraoperative global features is determined. If the degree of global matching is greater than or equal to the first degree threshold, the m-th cosine similarity between the m-th local feature in the preoperative local feature set and the intraoperative global feature is calculated. If the m-th cosine similarity is greater than or equal to the second degree threshold, and the semantic category corresponding to the m-th local feature matches the semantic category corresponding to the intraoperative global feature, then the m-th local feature is determined as a candidate feature. By traversing and calculating the local features in the preoperative local feature set in the above manner, a candidate feature set can be obtained.

[0051] In some embodiments, the first transformation matrix may include a transformation matrix that transforms the intraoperative point cloud set so that the transformed intraoperative point cloud set is in the same spatial coordinate system as the preoperative risk data; exemplarily, the first transformation matrix may include a first displacement parameter for displacing the intraoperative point cloud set and a first rotation parameter for rotating the intraoperative point cloud set; exemplarily, the first transformation matrix may be referred to as the initial rigid transformation matrix.

[0052] For example, the first transformation matrix can be obtained in the following way: The local regions corresponding to the candidate feature set are determined as candidate matching regions. The Random Sample Consensus (RANSAC) method is used to analyze the candidate matching regions and the intraoperative point cloud set to determine the correspondence between the data points in the candidate matching regions and the point clouds in the intraoperative point cloud set. The above correspondence is then parsed to obtain the first transformation matrix.

[0053] In some embodiments, the first registration data can be implemented in the following ways: Perform matrix multiplication on the first transformation matrix and the preoperative risk data to obtain the first registration data.

[0054] As can be seen from the above, in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the registration module includes a coarse registration unit that performs global feature extraction on the intraoperative point cloud set to obtain intraoperative global features, and performs feature extraction on the triangular mesh model corresponding to the preoperative risk data to obtain a preoperative local feature set, thereby realizing separate feature extraction for corresponding intraoperative and preoperative data; furthermore, based on the degree of matching between the intraoperative global features and the local features in the preoperative local feature set, a candidate feature set is determined from the preoperative local feature set, thereby realizing targeted screening of features in the preoperative local feature set; at the same time, based on the intraoperative global features and the candidate feature set, a first transformation matrix is ​​determined, which can improve the accuracy of the first transformation matrix; on this basis, the preoperative risk data is transformed based on the first transformation matrix to obtain the first registration data, which can improve the accuracy of the first registration data.

[0055] Based on the foregoing embodiments, the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment includes preoperative data including risk coefficients corresponding to the lesion site and related sites respectively; and intraoperative data including the instrument positions of surgical instruments. The registration module includes a fine registration unit 1032, which is used to determine a first point cloud set associated with the instrument position from the first registration data, determine a second point cloud set associated with the instrument position from the intraoperative point cloud set, determine a second transformation matrix between the first point cloud set and the second point cloud set based on the first point cloud set, the second point cloud set and the risk coefficient, and transform the first point cloud set based on the second transformation matrix to obtain the second registration data.

[0056] The first registration data is obtained by converting the preoperative risk data through the coarse registration unit of the registration module. The registration result includes the first registration data and the second registration data.

[0057] In some embodiments, the risk coefficient may be associated with a semantic category in the first semantic category, or the risk coefficient may change with the semantic category in the first semantic category, or the risk coefficient may be measured by at least one of the following: the severity of complications after injury, mortality and disability rate, repair difficulty, and probability of accidental injury during surgery of the tissue corresponding to the semantic category in the first semantic category. For example, the repair difficulty of an artery may be greater than that of adipose tissue, and therefore, the risk coefficient of an artery may be greater than that of adipose tissue.

[0058] In some embodiments, the instrument location may include the contact point between the surgical instrument and a structure or tissue.

[0059] In some embodiments, the first point cloud set may include a first spherical region centered on the instrument position in the first registration data; for example, the first spherical region may be a first region of interest (ROI); and the volume of the first ROI can be flexibly adjusted.

[0060] In some embodiments, the second point cloud set may include a second spherical region centered on the instrument position in the intraoperative point cloud set, and the second spherical region may be defined as a second ROI.

[0061] Accordingly, the first point cloud set can be determined in the following way: Determine the first center in the first registration data that corresponds to the instrument position, and then construct the first ROI with the first center as the center.

[0062] Accordingly, the second point cloud set can be determined in the following way: A second ROI is constructed using the location in the intraoperative point cloud set corresponding to the instrument position as the second center.

[0063] In some embodiments, the second transformation matrix can be determined in the following ways: If the risk coefficient is greater than or equal to the coefficient threshold, a correspondence analysis is performed on the first number of data points in the first ROI and the first number of data points in the second ROI to obtain a second transformation matrix. If the risk coefficient is less than the coefficient threshold, a correspondence analysis is performed on the second number of data points in the first ROI and the second number of data points in the second ROI to obtain a second transformation matrix. The first number can be greater than the second number, and the first risk level can be higher than the second risk level. The first risk level can include the risk level corresponding to the risk coefficient greater than the coefficient threshold, and the second risk level can include the risk level corresponding to the risk coefficient less than the risk coefficient.

[0064] In some embodiments, the second registration data can be obtained in the following ways: Perform matrix multiplication on the second transformation matrix and the first point cloud set to obtain the second registration data.

[0065] In some embodiments, the second registration data may include data that finely maps the data corresponding to the first ROI at the instrument position in the preoperative data to the intraoperative point cloud set.

[0066] As can be seen from the above, in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the preoperative data also includes risk coefficients corresponding to the patient site and related sites, and the intraoperative data also includes the instrument positions of the surgical instruments. The fine registration unit in the registration module is used to determine a first set of point clouds associated with the instrument positions from the first registration data and a second set of point clouds associated with the instrument positions from the intraoperative point cloud set. In this way, targeted screening of point clouds in the first registration data and the intraoperative point cloud set is achieved, so that the first set of point clouds and the second set of point clouds are respectively associated with the instrument positions. Furthermore, based on the first set of point clouds, the second set of point clouds, and the risk coefficients, a second transformation matrix between the first set of point clouds and the second set of point clouds is determined. In this way, the mutual correlation between the second transformation matrix and the risk coefficients is achieved, so that the second transformation matrix can intuitively represent the transformation relationship between the first set of point clouds and the second set of point clouds from the perspective of risk coefficients.

[0067] Based on the foregoing embodiments, in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the fine registration unit is used to determine the first fusion feature corresponding to the first point cloud set, determine the second fusion feature corresponding to the second point cloud set, and determine the second transformation matrix based on the first fusion feature, the second fusion feature, and the risk coefficient.

[0068] The first fusion feature includes at least the semantic category of the point cloud in the first point cloud set and the curvature of the point cloud in the first point cloud set, and the second fusion feature includes at least the semantic category of the point cloud in the second point cloud set and the curvature of the point cloud in the second point cloud set.

[0069] In some embodiments, the first fusion feature may further include the position and normal vector of the point cloud in the first point cloud set; correspondingly, the second fusion feature may further include the position and normal vector of the point cloud in the second point cloud set; exemplary, the above normal vector is used to characterize the orientation of the local surface where the point cloud is located, specifically to characterize the tilt direction of the tissue surface or structural surface where the local surface is located, and in the process of determining the second transformation matrix, the normal vector is used to constrain the second rotation parameter in the second rotation matrix.

[0070] In some embodiments, the curvatures included in the first fusion feature and the second fusion feature are used to quantify the degree of bending of the structural surface or tissue surface.

[0071] In some embodiments, the normal vector in the first fused feature can be obtained in the following way: For the i-th point cloud in the first point cloud set, obtain its k nearest neighbor set. Fit a local plane to the k nearest neighbor set using Principal Component Analysis (PCA), and determine the minimum principal component direction of the above plane as the normal vector of the i-th point cloud; where i can be an integer greater than or equal to 1, and k can be 20.

[0072] In some embodiments, the curvature in the first fusion feature can be obtained in the following manner: For the k nearest neighbor set corresponding to the i-th point cloud in the first point cloud set, calculate the covariance matrix of the three-dimensional coordinates of the point cloud corresponding to the k nearest neighbor set, and perform eigenvalue decomposition on the covariance matrix. Then, determine the curvature of the i-th point cloud based on at least three eigenvalues ​​with the largest values ​​in the eigenvalue decomposition. Specifically, it can be shown in Equation (3): (3) in, Let be the curvature of the i-th point cloud. , and These are the three largest eigenvalues, and their values ​​decrease sequentially.

[0073] In some embodiments, the second transformation matrix can be determined in the following ways: The first point cloud set is divided into sub-point cloud sets with different risk levels based on the risk coefficient. The priority level of the sub-point cloud is determined according to the risk coefficient of the sub-point cloud in the sub-point cloud set. The fusion features of the sub-point cloud in the sub-point cloud set are analyzed in turn according to the priority level. The correspondence between the fusion features and the second fusion features in terms of curvature, semantic category, position and normal vector dimension is obtained to obtain the second transformation matrix.

[0074] As can be seen from the above, in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the fine registration unit determines the first fusion feature corresponding to the first point cloud set and the second fusion feature corresponding to the second point cloud set. The first fusion feature includes the semantic category and curvature of the point cloud in the first point cloud set, and the second fusion feature includes the semantic category and curvature of the point cloud in the second point cloud set. Thus, through the first and second fusion features, the first and second point cloud sets can be intuitively represented from spatial, semantic, and morphological dimensions. Based on this, a second transformation matrix is ​​determined based on the first and second fusion features and the risk coefficient, so that the second transformation matrix can intuitively and accurately represent the mapping relationship between the first and second point cloud sets from the risk coefficient dimension, semantic dimension, and morphological dimension.

[0075] Based on the foregoing embodiments, in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the fine registration unit is used to determine the first fusion feature corresponding to the first point cloud set, determine the second fusion feature corresponding to the second point cloud set, and analyze the features in the first fusion feature and the features in the second fusion feature based on the risk coefficient through an attention mechanism to obtain the second transformation matrix.

[0076] In some embodiments, the second transformation matrix can be obtained in the following way: By placing the attention layer after the Multi-Layer Perceptron (MLP) module and before the global pooling module of PointNetLK, an improved PointNetLK is obtained. Then, the MLP module extracts features from the first fused features to obtain the first extraction result. At the same time, the MLP module extracts features from the second fused features to obtain the second extraction result. After that, the attention layer analyzes the risk coefficients associated with the first point cloud set to determine the attention weights corresponding to the high-risk level. Then, based on the above attention weights, weighted feature aggregation and iterative LK optimization are performed on the first and second extraction results to obtain the second transformation matrix.

[0077] Specifically, attention weights can be obtained in the following way: Construct an attention layer consisting of two layers of MLP and a cascaded Sigmoid function, and control the attention layer to perform the operation process shown in Equation (4) to obtain the attention weights: (4) in, It is a combination of the first fusion feature and the second fusion feature. To and The corresponding attention weights The weights are the weights of the first layer of a two-layer MLP, used to map combined features to a lower-dimensional space and extract discriminative patterns related to the importance of points. This is the bias vector for the first-layer MLP, used to set an independent activation threshold for each output channel. The weights are those of the second layer MLP in a two-layer MLP. This is the bias vector for the second-layer MLP, used to adjust the range of activation values. This is the Sigmoid function.

[0078] As can be seen from the above, in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the fine registration unit determines the first fusion feature corresponding to the first point cloud set and the second fusion feature corresponding to the second point cloud set. The first fusion feature includes the semantic category and curvature of the point clouds in the first point cloud set, and the second fusion feature includes the semantic category and curvature of the point clouds in the second point cloud set. Thus, through the first and second fusion features, an intuitive representation of the first and second point cloud sets can be achieved from spatial, semantic, and morphological dimensions. On this basis, the first and second fusion features are analyzed based on the risk coefficient using an attention mechanism to obtain a second transformation matrix. In this way, targeted tracking and analysis of the features in the first and second fusion features that correspond to the specified risk coefficient can be achieved, thereby improving the correlation between the second transformation matrix and the risk coefficient and improving the accuracy of the second transformation matrix.

[0079] Based on the foregoing embodiments, in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the fine registration unit is further used to determine the position error set based on the coordinate data in the second registration data and the coordinate data in the intraoperative point cloud set; The registration module also includes a deformation compensation unit, which is used to determine the deformation region based on the set of position errors.

[0080] In some embodiments, the i-th position error in the position error set can be obtained in the following way: Calculate the coordinate data of the i-th point in the second registration data and the i-th distance between the coordinate data of the point cloud corresponding to the i-th point in the intraoperative point cloud set, and determine the i-th distance as the i-th position error; where i is an integer greater than or equal to 1.

[0081] In some embodiments, the deformation region may include the region obtained by connecting the locations of the deformation points; for example, the deformation points may include points in the set of position errors whose position errors are greater than or equal to an error threshold.

[0082] In some embodiments, the deformation region can be determined in the following ways: The positional errors in the set of positional errors are analyzed, and the points whose positional errors are greater than or equal to the error threshold are identified as deformation points. Then, the deformation points are connected to obtain the deformation region.

[0083] As can be seen from the above, in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the fine registration unit determines the set of position errors based on the coordinate data in the second registration data and the coordinate data in the intraoperative point cloud set, and the deformation compensation unit determines the deformation area based on the position error. In this way, the accuracy of the deformation area can be improved and the effectiveness of the deformation area can be improved.

[0084] Based on the foregoing embodiments, in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the deformation compensation unit is used to determine the set of displacement vectors corresponding to the deformation region; wherein, the registration result includes the set of displacement vectors; The navigation and warning module is used to generate navigation data based on a set of displacement vectors.

[0085] In some embodiments, the i-th displacement vector corresponding to the i-th point cloud in the deformation region can be calculated in the following way: A graph structure consisting of point clouds in the deformed region is constructed. Then, a multi-layer graph convolutional network is used to perform neighbor feature aggregation on the i-th vertex in the graph structure inside the deformed region to obtain the i-th aggregation result. Then, the i-th aggregation result is 3D mapped by MLP to obtain the i-th displacement vector corresponding to the i-th vertex.

[0086] For vertices outside the deformation region, the radial basis function (RBF) interpolation method can be used to calculate the corresponding i-th displacement vector based on the known displacement at the boundary of the deformation region, as shown in Equation (5): (5) in, Let i be the i-th displacement vector. Let i be the point cloud. For deformation region Control points evenly distributed on the edge , as well as These are the interpolation coefficients. Let i be the kernel function of RBF; where i and k are both integers greater than or equal to 1.

[0087] For example, after determining the set of displacement vectors, the position of the i-th point cloud in the set of point clouds corresponding to the deformed region contained in the preoperative semantic point cloud data can be superimposed with the i-th displacement vector to obtain the i-th corrected displacement; i is traversed based on the number of point clouds contained in the deformed region to obtain the set of corrected displacements of all point clouds corresponding to the deformed region in the preoperative semantic point cloud, and the preoperative voxel mesh described in the previous embodiment is constructed based on the set of corrected displacements; then, the navigation data is determined by the SDF update method provided in the previous embodiment.

[0088] As can be seen from the above, in the intraoperative real-time navigation and early warning system for colorectal cancer surgery provided in this application embodiment, the deformation compensation unit determines the set of displacement vectors corresponding to the deformation region, and the navigation and early warning module generates navigation data based on the set of displacement vectors. Thus, the set of displacement vectors enables intuitive quantification of the displacement and deformation of the point cloud in the deformation region, and also improves the accuracy and detail of the navigation data.

[0089] It should be noted that the neural network model used in the aforementioned embodiments can be trained in advance using sample data.

[0090] For example, sample data can be collected in advance; for example, the sample data may include a first sample set, which may include preoperative three-dimensional image data of colorectal cancer that has been finely annotated by experts, and the semantic categories annotated by experts may include key structures such as tumors, arteries, veins, nerves, ureters, intestinal walls and adipose tissue.

[0091] For example, preoperative 3D image data can be segmented using an initialized 3D U-Net++ to obtain sample segmentation results. Then, forward propagation computation is performed on the sample segmentation results using the initialized 3D U-Net++ to obtain the semantic category and probability value of the voxels corresponding to the preoperative 3D image data. Finally, the following process is recursively executed using the backpropagation algorithm to obtain the 3D U-Net++: Calculate the gradient of the first loss between the semantic category and the semantic category annotated by the expert with respect to the parameters of the initialized 3D U-Net++, and update the above parameters using the Adam optimizer.

[0092] For example, the sample data may also include a second sample set, which may include real video frames from colorectal cancer surgery and pixel-level semantic categories annotated by experts.

[0093] For example, enhancement processing, including rotation, scaling, and color jitter, can be performed on the video frames in the second sample set first; then, the enhanced video frames are forward-propagated through the initialized DeepLabV3+ Mobile to output the semantic category of each pixel. The gradient of the second loss between the above semantic category and the pixel-level semantic category with respect to the initialized DeepLabV3+ Mobile network weights is calculated, and the network weights are iteratively updated using the backpropagation algorithm and optimizer to obtain DeepLabV3+ Mobile.

[0094] For example, the sample data may also include a third sample set, which may include a first preoperative point cloud sample corresponding to the preoperative three-dimensional image data, and may also include a second preoperative point cloud sample obtained by performing random perturbation on the preoperative point cloud sample; wherein, random perturbation may include perturbation such as random rigid transformation, applying noise, and random adoption.

[0095] For example, the first preoperative point cloud sample can be determined as the source point cloud, the second preoperative point cloud sample as the target point cloud, and the point cloud corresponding to a different structure or tissue from the preoperative 3D image can be determined as the negative sample. Then, based on the source point cloud, the target point cloud, and the negative sample, the parameters of the initial PointNet are adjusted to obtain the PointNet.

[0096] For example, the sample data may also include a fourth sample set, which may include a first sample ROI containing the surgical instrument contact area represented by the preoperative three-dimensional image, and a second sample ROI containing the contact area between the surgical instrument and the tissue in the real video frame; it should be noted that the first sample features of the first sample ROI may include the position, curvature, semantic category and risk coefficient of the point cloud contained in the first sample ROI, and the second sample features of the second sample ROI may include the position, curvature, semantic category and risk coefficient of the point cloud contained in the second sample ROI.

[0097] At this point, the first sample ROI and the second sample ROI can be used as inputs to the initial improved PointNetLK. After the initial improved PointNetLK fusion feature extraction, semantic weight calculation, weighted feature aggregation and iterative LK optimization, the first sample transformation result corresponding to the point cloud in the first sample ROI and the first sample transformation matrix can be obtained. The first transformation loss can be calculated by Equation (6): (6) in, For the first conversion loss, This is the result of the first sample conversion. This represents the actual transformation result in the second sample ROI corresponding to the transformation result of the first sample. To compare with the point cloud in the first sample ROI The corresponding semantic category, For the second sample ROI and Point clouds with the same semantic category semantic categories, To balance hyperparameters, Used to quantify the difference between the transformation result of the first sample and the actual transformation result. Used via indicator function The point cloud in the second sample ROI is filtered to obtain the result. Point clouds that share the same semantic category.

[0098] For example, after obtaining the first transformation loss, the loss gradient can be calculated using the backpropagation algorithm, and the network weights of the improved PointNetLK, including the attention module, can be iteratively updated using the optimizer, driving the improved PointNetLK to prioritize and accurately align high-risk semantic regions within a given ROI.

[0099] For example, the sample data may also include a fifth sample set, which may be a set of sample vectors; wherein, the set of sample vectors may include a set of point clouds corresponding to the preoperative three-dimensional image data and a set of real displacement vectors between corresponding point clouds in the set of point clouds corresponding to real video frames.

[0100] For example, a sample deformation field can be constructed based on the above sample vector set, and then the vertex set V of the point cloud set corresponding to the preoperative three-dimensional image data can be used as the graph node. Based on the spatial proximity relationship of the vertex set, an undirected edge set E can be constructed to build a graph structure (V, E). Each vertex in the vertex set contains the three-dimensional coordinates, unit normal vector, curvature and semantic category of the point cloud corresponding to the pixel in the preoperative three-dimensional image data.

[0101] Specifically, firstly, the neighbor features of the k-th vertex in the vertex set can be aggregated through an initial multi-layer graph convolutional network to obtain the k-th feature aggregation result. Then, the k-th feature aggregation result is feature-mapped through an initial MLP to obtain the k-th sample displacement vector. Then, the network parameters, including the initial multi-layer graph convolutional network and the initial MLP, are iteratively adjusted using an optimizer based on the displacement error between the k-th sample displacement vector and the k-th actual displacement vector corresponding to the k-th vertex in the sample vector set, to obtain the multi-layer graph convolutional network and the MLP.

[0102] Figure 3 This is another structural schematic diagram of the intraoperative real-time navigation and early warning system provided in the embodiments of this application, as shown below. Figure 3As shown, the intraoperative real-time navigation and early warning system may include: The preoperative data processing module and the risk map generation module are used to jointly process preoperative data to obtain preoperative risk data.

[0103] The intraoperative data processing module is used to process intraoperative data to obtain an intraoperative point cloud set.

[0104] The instrument positioning data processing module is used to track and convert the instrument position.

[0105] The data registration and fusion module can perform the operations performed by the configuration module in the aforementioned embodiments.

[0106] The real-time route planning module is used to generate navigation data.

[0107] The risk assessment module, the early warning module, and the navigation display module are used to perform the data processing functions of the navigation and early warning module in the aforementioned embodiments.

[0108] Based on the foregoing embodiments, this application also provides a method for intraoperative real-time navigation and early warning in colorectal cancer surgery. Figure 4 This is a flowchart illustrating the intraoperative real-time navigation and early warning method provided in the embodiments of this application, as shown below. Figure 4 As shown, the process may include the following steps: Step 401: Collect preoperative data before colorectal cancer surgery and collect intraoperative data during colorectal cancer surgery.

[0109] The preoperative data includes images of at least part of the lesion site of colorectal cancer; the intraoperative data includes imaging data of the colorectal cancer surgery process; and the lesion site is associated with the colorectal cancer. Step 402: Process the preoperative data to obtain preoperative risk data, and process the intraoperative data to obtain an intraoperative point cloud set.

[0110] The preoperative risk data includes the distribution of the lesion site and the first semantic category of the site associated with the lesion site; the intraoperative point cloud set includes the point cloud set corresponding to the intraoperative data and the second semantic category of the point cloud in the point cloud set. Step 403: Based on the preoperative risk data and the intraoperative point cloud set, perform spatial registration of the semantic dimension on the intraoperative data and the preoperative data to obtain the registration result; Step 404: Generate navigation data for surgical instruments based on the registration results, and generate and output early warning data associated with surgical instruments during the colorectal cancer surgery.

[0111] In some embodiments, based on preoperative risk data and intraoperative point cloud sets, spatial registration of the intraoperative data and preoperative data is performed along the semantic dimension to obtain registration results, including: Global features are extracted from the intraoperative point cloud set to obtain the intraoperative global features, and features are extracted from the triangular mesh model corresponding to the preoperative risk data to obtain the preoperative local feature set. Based on the degree of matching between intraoperative global features and local features in the preoperative local feature set, a candidate feature set is determined from the preoperative local feature set. Based on the intraoperative global features and the candidate feature set, the first transformation matrix between the preoperative risk data and the intraoperative point cloud set is determined. The preoperative risk data is transformed based on the first transformation matrix to obtain the first registration data; wherein, the registration result includes the first registration data.

[0112] In some embodiments, preoperative data further includes risk coefficients corresponding to the lesion site and associated sites respectively; intraoperative data further includes the instrument positions of surgical instruments; based on preoperative risk data and intraoperative point cloud sets, semantic spatial registration is performed on the intraoperative data and preoperative data to obtain registration results, including: A first set of point clouds associated with the instrument position is determined from the first registration data, and a second set of point clouds associated with the instrument position is determined from the intraoperative point cloud set; wherein, the first registration data is obtained by transforming preoperative risk data through the coarse registration unit of the registration module; Based on the first point cloud set, the second point cloud set, and the risk coefficient, determine the second transformation matrix between the first point cloud set and the second point cloud set; The first point cloud set is transformed based on the second transformation matrix to obtain the second registration data; wherein, the registration result includes the first registration data and the second registration data.

[0113] In some embodiments, a second transformation matrix between the first point cloud set and the second point cloud set is determined based on the first point cloud set, the second point cloud set, and a risk coefficient, including: A first fusion feature corresponding to the first point cloud set is determined, and a second fusion feature corresponding to the second point cloud set is determined. Based on the first fusion feature, the second fusion feature, and the risk coefficient, a second transformation matrix is ​​determined. The first fusion feature includes at least the semantic category and curvature of the point clouds in the first point cloud set. The second fusion feature includes at least the semantic category and curvature of the point clouds in the second point cloud set.

[0114] In some embodiments, a second transformation matrix between the first point cloud set and the second point cloud set is determined based on the first point cloud set, the second point cloud set, and a risk coefficient, including: Determine a first fusion feature corresponding to a first point cloud set, and determine a second fusion feature corresponding to a second point cloud set; wherein, the first fusion feature includes at least the semantic category of the point cloud in the first point cloud set and the curvature of the point cloud in the first point cloud set; the second fusion feature includes at least the semantic category of the point cloud in the second point cloud set and the curvature of the point cloud in the second point cloud set. The second transformation matrix is ​​obtained by analyzing the features in the first fusion feature and the features in the second fusion feature based on the risk coefficient through the attention mechanism.

[0115] In some embodiments, the above method further includes: Based on the coordinate data in the second registration data and the coordinate data in the intraoperative point cloud set, the set of positional errors is determined; The deformation region is determined based on the set of positional errors.

[0116] In some embodiments, navigation data for surgical instruments is generated based on the registration results, including: Determine the set of displacement vectors corresponding to the deformation region; the registration result includes the set of displacement vectors. Navigation data is generated based on the set of displacement vectors.

[0117] The following provides statistical data on the surgical outcomes of this procedure and the comparison procedure. The comparison procedure may include the following: First option: A rigid registration and navigation scheme based on Scale-Invariant Feature Transform (SIFT) features; The second approach: Non-rigid registration navigation based on the Iterative Closest Point (ICP) algorithm; The third approach: Net based on deep learning point cloud registration.

[0118] The comparison scenarios include: large-scale rigid displacement with a displacement amplitude greater than 50 mm, local nonlinear deformation amplitude greater than or equal to 30 mm, and high-risk close-range operations.

[0119] The evaluation metrics comprehensively cover the core dimensions of clinical performance: target registration error, tumor coverage, early warning accuracy, system latency, and success rate under strong deformation.

[0120]

[0121] Table 1 As shown in Table 1, compared with the comparative scheme, this scheme has improved in terms of target registration error, tumor coverage, early warning accuracy, system latency, and success rate under strong deformation.

[0122] Table 2 presents the statistical results of the ablation experiments provided in the embodiments of this application.

[0123]

[0124] Table 2 As shown in Table 2, after removing the semantic category processing steps in the deformation compensation unit and the fine registration unit, the targeted registration error, tumor coverage, and high-risk structure protection rate all decreased. Therefore, the semantic category-related data processing steps included in this scheme are crucial to improving its performance.

[0125] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0126] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0127] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0128] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0129] It should be noted that the aforementioned computer-readable storage media can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0130] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0131] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware nodes. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0133] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0134] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0135] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0136] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A real-time intraoperative navigation and early warning system for colorectal cancer surgery, characterized in that, include: The data acquisition module is used to acquire preoperative data before the colorectal cancer surgery is performed and to track and acquire intraoperative data during the colorectal cancer surgery; wherein, the preoperative data includes images of at least a portion of the lesion site in the colorectal region; the intraoperative data includes image data of the colorectal cancer surgery process; and the lesion site is associated with the colorectal cancer. The preprocessing module is used to process the preoperative data to obtain preoperative risk data and to process the intraoperative data to obtain an intraoperative point cloud set. The preoperative risk data includes the distribution status of the lesion site and the first semantic category of the site associated with the lesion site. The intraoperative point cloud set includes the point cloud set corresponding to the intraoperative data and the second semantic category corresponding to the point cloud in the point cloud set. The registration module is used to perform semantic-dimensional spatial registration on the intraoperative data and the preoperative data based on the preoperative risk data and the intraoperative point cloud set, and obtain the registration result; The navigation and early warning module is used to generate navigation data for surgical instruments based on the registration results, and to generate and output early warning data associated with the surgical instruments during the colorectal cancer surgery.

2. The system according to claim 1, characterized in that, The registration module includes a coarse registration unit, which is used to extract global features from the intraoperative point cloud set to obtain intraoperative global features, and to extract features from the triangular mesh model corresponding to the preoperative risk data to obtain a preoperative local feature set. The coarse registration unit is further configured to determine a candidate feature set from the preoperative local feature set based on the degree of matching between the intraoperative global features and the local features in the preoperative local feature set, and to determine a first transformation matrix between the preoperative risk data and the intraoperative point cloud set based on the intraoperative global features and the candidate feature set. The coarse registration unit is further configured to transform the preoperative risk data based on the first transformation matrix to obtain first registration data; wherein the registration result includes the first registration data.

3. The system according to claim 1, characterized in that, The preoperative data also includes risk coefficients corresponding to the lesion site and the associated sites respectively; the intraoperative data also includes the instrument positions of the surgical instruments; The registration module includes a fine registration unit, used to determine a first point cloud set associated with the instrument position from the first registration data, and to determine a second point cloud set associated with the instrument position from the intraoperative point cloud set; wherein, the first registration data is obtained by converting the preoperative risk data through the coarse registration unit of the registration module; The fine registration unit is further configured to determine a second transformation matrix between the first point cloud set and the second point cloud set based on the first point cloud set, the second point cloud set, and the risk coefficient; The fine registration unit is further configured to transform the first point cloud set based on the second transformation matrix to obtain second registration data; wherein the registration result includes the first registration data and the second registration data.

4. The system according to claim 3, characterized in that, The fine registration unit is used to determine a first fusion feature corresponding to the first point cloud set, determine a second fusion feature corresponding to the second point cloud set, and determine a second transformation matrix based on the first fusion feature, the second fusion feature, and the risk coefficient; wherein, the first fusion feature includes at least the semantic category of the point cloud in the first point cloud set and the curvature of the point cloud in the first point cloud set; the second fusion feature includes at least the semantic category of the point cloud in the second point cloud set and the curvature of the point cloud in the second point cloud set.

5. The system according to claim 3, characterized in that, The fine registration unit is used to determine a first fusion feature corresponding to the first point cloud set and a second fusion feature corresponding to the second point cloud set; wherein, the first fusion feature includes at least the semantic category of the point cloud in the first point cloud set and the curvature of the point cloud in the first point cloud set; the second fusion feature includes at least the semantic category of the point cloud in the second point cloud set and the curvature of the point cloud in the second point cloud set. The fine registration unit is further configured to analyze the features in the first fusion feature and the features in the second fusion feature based on the risk coefficient using an attention mechanism to obtain the second transformation matrix.

6. The system according to claim 3, characterized in that, The fine registration unit is also used to determine a set of positional errors based on the coordinate data in the second registration data and the coordinate data in the intraoperative point cloud set; The registration module further includes a deformation compensation unit, used to determine the deformation region based on the set of position errors.

7. The system according to claim 6, characterized in that, The deformation compensation unit is used to determine the set of displacement vectors corresponding to the deformation region; wherein, the registration result includes the set of displacement vectors. The navigation and early warning module is used to generate the navigation data based on the set of displacement vectors.

8. A method for intraoperative real-time navigation and early warning in colorectal cancer surgery, characterized in that, The method includes: Preoperative data is collected before the colorectal cancer surgery is performed, and intraoperative data is collected during the colorectal cancer surgery; wherein, the preoperative data includes images of at least a portion of the lesion site in the colorectal region; the intraoperative data includes image data of the colorectal cancer surgery process; the lesion site is associated with the colorectal cancer; The preoperative data is processed to obtain preoperative risk data, and the intraoperative data is processed to obtain an intraoperative point cloud set; wherein, the preoperative risk data includes the distribution status of the lesion site and the first semantic category of the site associated with the lesion site; the intraoperative point cloud set includes the point cloud set corresponding to the intraoperative data and the second semantic category corresponding to the point cloud in the point cloud set. Based on the preoperative risk data and the intraoperative point cloud set, spatial registration of the semantic dimension is performed on the intraoperative data and the preoperative data to obtain the registration result; Based on the registration results, navigation data for the surgical instruments is generated, and during the colorectal cancer surgery, warning data associated with the surgical instruments is generated and output.

Images