Surgical visual segmentation method, system, device and storage medium across device generalization

Abstract

The present application relates to the technical field of medical image processing, in particular to a cross-device generalization surgical visual segmentation method, system, device and storage medium, the method comprising: acquiring image data, constructing a training sample set; applying image signal processing intervention to the image data, generating intervention image data and constructing same-frame intervention pairing samples; inputting the same-frame intervention pairing samples into a multi-branch surgical visual segmentation network, extracting semantic features, boundary detail features and instrument features; applying selective domain invariant constraints to a preset feature subset in the semantic features, and applying multi-scale structure consistency constraints to the boundary detail features; determining instrument region masks and reliability indicators, and adaptively modulating and fusing the semantic features and the boundary detail features after the constraints are applied; updating network parameters; and inputting surgical operation image data to be processed into the trained network. The present scheme can improve the generalization ability and segmentation stability of the surgical visual segmentation model under heterogeneous devices and complex surgical fields.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a surgical visual segmentation method, system, device, and storage medium that can be generalized across devices. Background Technology

[0002] With the development of minimally invasive surgical techniques, deep learning-based semantic segmentation models have been widely applied to tasks such as key anatomical structure identification, instrument localization, and surgical area segmentation in laparoscopic surgical videos. However, existing surgical visual segmentation models still face two major challenges in practical clinical deployment.

[0003] First, current mainstream models are typically trained on datasets acquired by endoscopic devices of specific brands, models, or with fixed imaging parameters. Because different imaging systems exhibit significant differences in image signal processing aspects such as spectral response, color mapping, sharpening intensity, denoising strategies, and dynamic range, the model's segmentation accuracy can drastically decrease when applied across devices due to feature distribution shifts. For example, when a segmentation model trained on one device is transferred to another brand of device, organ boundaries may become blurred, or even targets may be lost. Existing technologies often employ domain adaptation methods, but these typically require target domain data for retraining and lack systematic decoupling strategies to address physical differences in the imaging process, making it difficult to achieve true zero-shot cross-device generalization.

[0004] Secondly, real surgical scenarios involve complex factors such as instrument occlusion, tissue reflection, smoke interference, and soft tissue deformation. When faced with instrument occlusion, models tend to use statistical co-occurrence relationships between instrument positions and anatomical structures to form shortcut learning. However, once the occlusion pattern changes or the appearance of the instruments changes, the segmentation results degrade. Furthermore, existing methods lack quantitative evaluation and adaptive adjustment mechanisms for the reliability of segmentation results, and cannot avoid performance collapse in the worst case under strong interference conditions.

[0005] In summary, improving the generalization ability of surgical visual segmentation models on heterogeneous devices and enhancing their segmentation stability under complex surgical field conditions are urgent technical problems that need to be solved. Summary of the Invention

[0006] This invention provides a surgical visual segmentation method, system, device, and storage medium that can generalize across devices, thereby improving the generalization ability and segmentation stability of surgical visual segmentation models under heterogeneous devices and complex surgical field conditions.

[0007] The basic solution provided by this invention is as follows:

[0008] A cross-device generalized surgical visual segmentation method includes the following steps:

[0009] S100: Acquire surgical image data, as well as segmentation labels and device domain information corresponding to the image data, and construct a training sample set;

[0010] S200, apply a preset image signal processing intervention to each image data in the training sample set, adjust the imaging style of the image data, and generate corresponding intervention image data; pair each image data with its corresponding intervention image data to construct same-frame intervention paired samples;

[0011] S300, construct a multi-branch surgical visual segmentation network, the multi-branch surgical visual segmentation network including semantic branches, boundary detail branches and instrument branches;

[0012] S400, input the image data in the same frame intervention pairing sample and its corresponding intervention image data into the multi-branch surgical visual segmentation network, and extract semantic features, boundary detail features and instrument features respectively;

[0013] S500, apply selective domain invariance constraints to a preset subset of semantic features to suppress the predictability of device domain information; and apply multi-scale structural consistency constraints to the boundary detail features to ensure that the image data and the intervention image data are consistent in boundary morphology and topology.

[0014] S600: Determine the instrument region mask and the reliability index of the instrument region mask based on the extracted instrument features, and perform adaptive modulation fusion on the constrained semantic features and constrained boundary detail features based on the reliability index to generate output features.

[0015] S700, generate training segmentation results based on the output features, and update the parameters of the multi-branch surgical visual segmentation network based on a preset loss function to obtain the trained multi-branch surgical visual segmentation network.

[0016] The S800 inputs the surgical image data to be processed into the trained multi-branch surgical visual segmentation network to generate the target segmentation result.

[0017] Furthermore, in S200, the image signal processing intervention includes one or more of the following: gamma adjustment, brightness perturbation, contrast perturbation, sharpening perturbation, denoising perturbation, color matrix perturbation, dynamic range perturbation, and quantization range perturbation.

[0018] Furthermore, in S500, the selective domain invariance constraint is implemented through a domain discriminator, and its optimization objective is:

[0019]

[0020] In the formula, To segment network parameters, For domain discriminator parameters, This refers to a predefined subset of features selected from semantic features. For device domain information, This is the probability that the domain discriminator predicts for device domain information based on the preset feature subset. For the expected operation, The main segmentation loss term, For the loss weights that are selected by the invariant constraint of the selection field, , This is the loss term for the selection field invariance constraint.

[0021] Furthermore, in S500, the multi-scale structural consistency constraint is implemented at at least two scales, and the loss based on the multi-scale structural consistency constraint is expressed as:

[0022]

[0023] In the formula, For the loss term based on multi-scale structural consistency constraints, Weights for boundary topological continuity loss. For the boundary topological continuity loss term, For pairwise structural consistency loss weights, For the index of intervention types, For the expected operation of the index for all intervention types, For structural description operators, and The image data and the first Boundary detail features of interventional image data Represents image data, Indicates the first Interventional image data.

[0024] Furthermore, the output information of the device branch includes confidence level, prediction entropy, and calibration score; in S600, the reliability index of the device region mask is one or more of confidence level, prediction entropy, and calibration score.

[0025] Furthermore, in S600, the adaptive modulation fusion is represented as:

[0026]

[0027] In the formula, For output features, As a reliability indicator, The occlusion suppression coefficient is a monotonically varying factor that changes with reliability indicators. For the instrument area mask, This indicates element-wise operations. This refers to the fusion of semantic features after applying selective domain invariance constraints and boundary detail features after applying multi-scale structural consistency constraints. Indicates the semantic compensation coefficient. The semantic features are defined by applying selective domain invariance constraints.

[0028] further, .

[0029] Furthermore, the occlusion suppression coefficient is positively correlated with the reliability index, and the semantic compensation coefficient is negatively correlated with the reliability index.

[0030] Furthermore, the preset loss function includes a main segmentation loss term, a loss term based on multi-scale structural consistency constraints, a device branch supervision loss term, and a selective domain invariant constraint loss term.

[0031] Furthermore, the preset loss function Represented as:

[0032]

[0033] In the formula, The main segmentation loss term, Weights for structural consistency loss. For the loss term based on multi-scale structural consistency constraints, For the weight of the monitoring loss of the device branch, For the device branch supervision loss item, For the loss weights that are selected by the invariant constraint of the selection field, This is the loss term for the selection field invariance constraint.

[0034] The second basic solution provided by this invention is a cross-device generalized surgical visual segmentation system for performing the above-mentioned cross-device generalized surgical visual segmentation method.

[0035] Furthermore, it includes a data acquisition module, an intervention generation module, a network construction module, a feature extraction module, a domain invariant constraint module, a structural consistency constraint module, an instrument region and reliability estimation module, a feature fusion module, a training and update module, and an output module;

[0036] The data acquisition module is used to acquire surgical image data, as well as segmentation labels and device domain information corresponding to the image data, and to construct a training sample set.

[0037] The intervention generation module is used to apply a preset image signal processing intervention to each image data in the training sample set, adjust the imaging style of the image data, and generate corresponding intervention image data; and pair each image data with its corresponding intervention image data to construct a same-frame intervention pairing sample.

[0038] The network construction module is used to construct a multi-branch surgical visual segmentation network, which includes a semantic branch, a boundary detail branch, and an instrument branch.

[0039] The feature extraction module is used to input the image data in the same frame intervention pairing sample and its corresponding intervention image data into the multi-branch surgical visual segmentation network constructed by the network construction module, and extract semantic features, boundary detail features and instrument features respectively.

[0040] The domain invariant constraint module is used to apply selective domain invariant constraints to a preset subset of semantic features to suppress the predictability of device domain information.

[0041] The structural consistency constraint module is used to apply multi-scale structural consistency constraints to the boundary detail features, so that the image data and the intervention image data are consistent in boundary morphology and topology.

[0042] The instrument region and reliability estimation module is used to determine the instrument region mask and the reliability index of the instrument region mask based on the extracted instrument features.

[0043] The feature fusion module is used to adaptively modulate and fuse the constrained semantic features and the constrained boundary detail features based on the reliability index to generate output features;

[0044] The training update module is used to generate training segmentation results based on the output features and update the parameters of the multi-branch surgical visual segmentation network based on a preset loss function to obtain the trained multi-branch surgical visual segmentation network.

[0045] The output module is used to input the surgical image data to be processed into the trained multi-branch surgical visual segmentation network to generate the target segmentation result.

[0046] The third basic solution provided by the present invention is an electronic device comprising a computer-readable storage medium and a processor, wherein the computer-readable storage medium stores a computer program, and when the processor executes the computer program, it implements the above-mentioned cross-device generalized surgical visual segmentation method.

[0047] The present invention provides a fourth basic solution: a computer-readable storage medium for storing computer-executable instructions, which, when executed, implement the above-mentioned cross-device generalized surgical visual segmentation method.

[0048] The principles and advantages of this invention are as follows:

[0049] In this scheme, by applying pre-defined image signal processing interventions to each image data in the training sample set, corresponding intervention image data is generated, and same-frame intervention paired samples are constructed. This enables the multi-branch surgical visual segmentation network to learn invariant features under imaging style changes. By applying selective domain invariant constraints to a pre-defined subset of semantic features, the predictability of device domain information is suppressed, reducing the model's dependence on specific device imaging links, thereby improving the generalization ability of the surgical visual segmentation model in heterogeneous device environments.

[0050] Furthermore, this scheme applies multi-scale structural consistency constraints to boundary detail features, ensuring consistency in boundary morphology and topology between the image data and the interventional image data. This suppresses boundary jitter and fine structure breakage caused by imaging differences. Simultaneously, by determining the reliability index of the instrument region mask, adaptive modulation and fusion are performed on the constrained semantic features and boundary detail features based on this index. When the reliability index is low, the dependence on the instrument region mask is reduced and the compensation of semantic feature branches is enhanced; when the reliability index is high, the suppression of occluded regions is strengthened. This mechanism provides quantitative evaluation and dynamic adjustment capabilities for the segmentation results, avoiding performance collapse under strong interference conditions, thereby enhancing the segmentation stability of the surgical visual segmentation model under complex surgical field conditions.

[0051] Finally, by adopting this scheme, during the inference stage, only the surgical image data to be processed needs to be input into the trained multi-branch surgical visual segmentation network to generate the target segmentation result. No additional temporal modeling or domain adaptation calculation is required, which can meet the real-time requirements of surgical navigation. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating an embodiment of the cross-device generalized surgical visual segmentation method of the present invention.

[0053] Figure 2 This is a schematic diagram illustrating the generation and influence relationship of image data in an embodiment of the cross-device generalized surgical visual segmentation method of the present invention.

[0054] Figure 3 This is a schematic diagram illustrating the construction of paired intervention samples within the same frame in an embodiment of the cross-device generalized surgical visual segmentation method of the present invention.

[0055] Figure 4 This is a block diagram of a multi-branch surgical visual segmentation network structure in an embodiment of the cross-device generalized surgical visual segmentation method of the present invention.

[0056] Figure 5 This is a schematic diagram of reliability modulation and fusion under instrument occlusion in an embodiment of the cross-device generalized surgical visual segmentation method of the present invention. Detailed Implementation

[0057] The following detailed description illustrates the specific implementation method:

[0058] Example 1:

[0059] This embodiment provides a cross-device generalized surgical visual segmentation method, such as... Figure 1 As shown, it includes the following steps.

[0060] S100: Acquire surgical image data, as well as segmentation labels and device domain information corresponding to the image data, and construct a training sample set.

[0061] Specifically, surgical images or video frames are acquired from laparoscopic, endoscopic, or microsurgical devices of different brands, models, and image signal processing parameter configurations. The data covers at least two different device domains to ensure that the model can learn generalized features across devices.

[0062] For the acquired image data Preprocessing is performed, including image black border removal, timeline cropping, resolution unification, color space conversion, frame sampling, outlier frame removal, annotation region verification, and label consistency checking. This allows raw video frames acquired from different devices to be uniformly adjusted to a preset input resolution, for example... , or In other embodiments of this application, higher resolution images may also be retained for high-precision training or distillation training to improve the model's ability to perceive details.

[0063] Then, clinical experts use medical image annotation tools to perform pixel-level annotations, generating segmentation labels corresponding to the original image data. The annotation objects include key anatomical structures such as blood vessels, nerves, and ureters; surgical areas such as cutting planes or anastomoses; high-risk areas such as bleeding points or tumor boundaries; and surgical instruments. Device domain information for each image is also recorded. The encoding can be done using the device model, center number, or ISP link type.

[0064] Thus, through meticulous preprocessing and resolution strategies, the quality and diversity of training data are ensured; retaining high-resolution images makes it possible to conduct subsequent high-precision training or knowledge distillation, further enhancing the applicability of the model.

[0065] S200, apply a preset image signal processing intervention to each image data in the training sample set, adjust the imaging style of the image data, and generate corresponding intervention image data; pair each image data with its corresponding intervention image data to construct a same-frame intervention pairing sample.

[0066] The core of this step is to simulate the differences in imaging style caused by different devices or different image signal processing parameters, without changing the scene content in the image data. The formation process of surgical image data can be abstracted into the following relationship:

[0067]

[0068] In the formula, Represents the observed image data, It indicates the state of anatomical structures, specifically including the actual spatial state of tissues, blood vessels, nerves, surgical planes, or risk areas in the current surgical field; The instrument factors specifically include the instrument's type, location, shape, motion state, and degree of obstruction. This refers to equipment or imaging link factors, specifically including differences in spectral response, color mapping, sharpening, noise reduction, quantization range, etc., between imaging systems of different brands, models, or parameter configurations. This indicates noise disturbance.

[0069] like Figure 2 As shown, the goal of this solution is to leverage image data. Predict the corresponding segmentation result At the same time, suppress device factor The resulting cross-device distribution shift, and the instrument factor Anatomical structure status The statistical shortcut learning caused by the coupling relationship between them Figure 2 The dashed lines in the diagram represent the "target-determining relationship," indicating the segmentation result. It should reflect the anatomical state as much as possible. The corresponding real semantic information.

[0070] The insufficient cross-device generalization ability of existing models mainly stems from two aspects: firstly, device or imaging link factors. This will introduce cross-device distribution offset; on the other hand, the device factor Anatomical structure status Often exhibiting statistical correlation, models can easily exploit the co-occurrence relationship between instruments and anatomical structures during training to form statistical shortcuts, leading to significant degradation under conditions of unknown equipment, unknown instrument appearance, or complex occlusion. Based on this, this invention designs a modulation fusion mechanism that incorporates same-frame intervention pairing, selective domain invariance constraints, multi-scale structural consistency constraints, and reliability indices based on instrument region masks. The following steps all revolve around this core objective.

[0071] For each original image data Apply several preset image signal processing interference parameters ,generate Interventional image data The image signal processing interventions include one or more of the following: gamma adjustment, brightness perturbation, contrast perturbation, sharpening perturbation, denoising perturbation, color matrix perturbation, dynamic range perturbation, and quantization range perturbation. In this embodiment, four interventions are used: gamma adjustment, sharpening perturbation, denoising perturbation, and color matrix perturbation. To avoid excessive intervention damaging the recognizability of the original image data, the intervention intensity is controlled within a preset range.

[0072] The gamma coefficient is set between 0.7 and 1.3. In this embodiment, a value of 1.2 is selected. In other embodiments of this application, the value can also be 0.8 or 1.0.

[0073] The sharpening perturbation intensity is set to three levels: weak, medium, and strong. In other embodiments of this application, a continuous weighting form can also be used.

[0074] The noise reduction perturbation intensity is achieved by using different smoothing kernel scales or different filtering intensities;

[0075] The color matrix perturbation uses a preset color offset matrix, and in other embodiments of this application, a random sampling matrix may also be used.

[0076] In other embodiments of this application, the intensity of brightness perturbation, contrast perturbation, dynamic range perturbation, and quantization range perturbation is also set according to a preset range. For example, the brightness adjustment coefficient is set between 0.8 and 1.2, the contrast adjustment coefficient is set between 0.8 and 1.2, and the dynamic range and quantization range perturbations are achieved by changing the upper and lower limits of pixel values. An intervention method can be randomly selected during training, or multiple intervention image data can be generated simultaneously and used together with the original image data for training.

[0077] like Figure 3 As shown, since the intervention does not change the semantic content of the image data, the original image data... With all its interventional imaging data Sharing the same segmentation label The reason is that these images originate from the same frame of scene content, with only the imaging style changing, while the actual anatomical structure and the spatial semantic location of the target area remain unchanged. and Pairing is used to form paired samples for intervention within the same frame.

[0078] This approach significantly increases the diversity of training samples without requiring additional annotations and explicitly guides the model to focus on invariant structural and semantic information under image signal processing perturbations, thereby effectively suppressing cross-device distribution shifts. Fine-grained intensity control ensures the rationality of the intervention, preventing unrecognizable image data due to over-intervention.

[0079] S300, a multi-branch surgical visual segmentation network is constructed. In this embodiment, an improved real-time segmentation architecture is adopted, specifically a multi-branch network based on PIDNet. In other embodiments of this application, other real-time segmentation architectures with multi-scale semantic and boundary joint modeling capabilities can also be used. The multi-branch surgical visual segmentation network includes a semantic branch: using a larger receptive field to extract global semantic features. Used to identify the category of anatomical structure; Boundary detail branch: preserves high-resolution spatial information and extracts boundary and detail features. Used for fine segmentation and edge localization; Instrument branch: specifically learns the features of the instrument region. It also outputs the instrument region mask and reliability estimate.

[0080] As a concrete implementation example, this embodiment improves upon PIDNet, a pixel-based image deep network, as its foundational architecture. Specifically: First, the detail branch and semantic branch of PIDNet are retained, serving as the boundary detail branch and semantic branch of this invention, respectively. The boundary detail branch maintains a high feature map resolution, for example, one-quarter or one-eighth of the original image size, and extracts edge and texture information through several convolutional layers; the semantic branch uses downsampled feature maps, for example, one-sixteenth or one-thirty-second of the original image size, and extracts global contextual semantics through residual blocks and pyramid pooling modules. Second, a new lightweight convolutional branch is added in parallel alongside the PIDNet backbone network as the instrumentation branch. This branch receives feature maps from intermediate layers of the backbone network, such as the output of the second residual block from the semantic branch, with a resolution one-eighth of the original image. These feature maps are sequentially passed through two 3×3 convolutional layers, each followed by batch normalization and a ReLU activation function, and then through a 1×1 convolutional layer. The final output consists of two results: the first is a binary mask of the same size as the input, activated by a sigmoid function, representing the probability that each pixel belongs to the instrument region; the second is an image of the same size as the mask or a scalar representing the reliability index of the prediction, normalized to the 0-1 range by a sigmoid function. Third, the encoder parts of the three branches share the convolutional layer parameters from the first two downsampling stages to reduce computation and promote collaborative learning of basic features. However, the deep parameters of each branch are learned independently thereafter, optimizing their respective task objectives.

[0081] The multi-branch design decouples semantic information from spatial details and provides a dedicated channel for explicit modeling of instrument factors, avoiding over-coupling of instrument features and semantic features in the early stages.

[0082] S400, the image data in the same frame intervention pairing sample and its corresponding intervention image data are input into the multi-branch surgical visual segmentation network to extract semantic features, boundary detail features and instrument features respectively.

[0083] like Figure 4 As shown, during training, the original image data... and the interventional image data it generates Input into the network simultaneously or sequentially. For and each The three branches of the network extract their respective semantic features in parallel. Boundary detail features Instrument characteristics These features will be used in subsequent different constraint and fusion modules. By extracting multi-dimensional features in parallel, a sufficient feature foundation is provided for subsequent selective domain invariant constraints, structural consistency constraints, and reliability modulation fusion, ensuring that each branch can optimize independently and work collaboratively.

[0084] S500, apply selective domain invariance constraints to a preset subset of semantic features to suppress the predictability of device domain information; and apply multi-scale structural consistency constraints to the boundary detail features to ensure that the image data and the intervention image data are consistent in boundary morphology and topology.

[0085] First, the selective domain invariance constraint does not uniformly apply domain adversarial treatment to all network features, but rather selects a preset subset of features from the intermediate layers or specific channels of the semantic branches. Device domain information suppression is applied only to the feature subset. The selective domain invariant constraint is implemented through a domain discriminator. To achieve this, the optimization objective is:

[0086]

[0087] In the formula, To segment network parameters, For domain discriminator parameters, This refers to a predefined subset of features selected from semantic features. For device domain information, This is the probability that the domain discriminator predicts for device domain information based on the preset feature subset. For the expected computation on training samples or mini-batch samples, The main segmentation loss term, Let the weights of the selective region-invariant constraint loss be denoted as... , This is the loss term for the selection field invariance constraint.

[0088] In one specific embodiment, the preset feature subset The selection method is as follows: For the multi-channel feature map output by the semantic branch, for example, the feature map with C channels output by the last residual block of the semantic branch, firstly, the feature map of each channel is subjected to global average pooling to obtain a C-dimensional feature vector; then, this feature vector and the device domain information d (discrete label) are input into a mutual information estimator to calculate the mutual information value between each channel and the device domain. The feature maps corresponding to the top K channels with the highest mutual information values ​​(K=C / 4, i.e., one-quarter of the number of channels) are selected as... A domain-invariant constraint is applied to it. In another embodiment, a lightweight learnable mask module is used, which consists of two 1×1 convolutional layers and a sigmoid activation function. The input is semantic features, and the output is a mask vector of the same length as the number of channels, with values ​​between 0 and 1. This mask vector is multiplied channel by channel with the semantic features to obtain the weighted mask. and utilize By constraining the weighted feature, a selection domain invariance constraint is achieved through soft selection.

[0089] This constraint encourages The device origin cannot be accurately identified by the domain discriminator, thus suppressing the leakage of device-related information. Because the constraints are selective, the rest of the network, such as boundary detail branches, can still retain the ability to express local visibility differences and fine structures, avoiding the decline in semantic recognition ability caused by excessive dedomaining.

[0090] Secondly, to enhance the model's robustness to boundary jitter and structural fracture, multi-scale structural features are output from the boundary detail branch. Apply consistency constraints. This is performed at at least two scales, such as 1 / 8 and 1 / 16 of the feature map resolution, to calculate the structural similarity between the original image data and the intervention image data. The structural consistency constraints can be based on a structural tensor, orientation consistency descriptor, normalized gradient field, refined skeleton description, or boundary topology preservation index. The loss based on the multi-scale structural consistency constraints is expressed as:

[0091]

[0092] In the formula, For the loss term based on multi-scale structural consistency constraints, Weights for boundary topological continuity loss. For the boundary topological continuity loss term, For pairwise structural consistency loss weights, For the index of intervention types, For the expected operation of the index for all intervention types, For structural descriptors, structural tensors, orientation-consistent descriptors, gradient orientation fields, normalized boundary responses, or combinations thereof can be used. and The image data and the first Boundary detail features of interventional image data Represents image data, Indicates the first Interventional image data.

[0093] In this embodiment, the boundary topological continuity loss term adopts a topological difference loss based on persistent homology. The specific calculation process is as follows:

[0094] First, feature values ​​are extracted from the boundary detail feature map output from the boundary detail branch, corresponding to the original image data, and from the boundary detail feature map corresponding to any intervention image data. To reduce computational complexity, the two feature maps can be downsampled to one-quarter or one-eighth of the original image size.

[0095] Then, the Vietoris-Rips filtering method is used to construct a persistence graph from each feature map. The persistence graph records the threshold information for the appearance and disappearance of topological features—including connected components and holes). Each topological feature is represented as a point in the persistence graph, with the x-axis representing the threshold for the feature's appearance and the y-axis representing the threshold for the feature's disappearance.

[0096] Next, the 1-Wasserstein distance between the two persistent graphs is calculated. This distance is defined as the minimum sum of distances between all topological feature points in one persistent graph and those in the other, when performing optimal matching. A smaller distance indicates that the two feature graphs are more similar in terms of the number of connected components, the number of holes, and their persistence.

[0097] The calculated 1-Wasserstein distance is used as the value of the boundary topological continuity loss term. This loss term forces the boundary detail feature maps generated by the model under different imaging styles to have a consistent topological structure, thereby avoiding boundary breaks, hole disappearances, or the appearance of redundant connected components due to image signal processing intervention.

[0098] In practice, open-source persistent homology computation libraries, such as Ripser++, can be used to efficiently construct a persistent graph and the Sinkhorn-Knopp algorithm can be used to approximate the optimal transport problem, thereby reducing computational complexity. This loss term is jointly optimized with the main segmentation loss and structural consistency loss during training, and is calculated every few training steps to balance efficiency and effectiveness.

[0099] As a feasible example, the structure descriptor operator employs a structure tensor. The specific process is as follows: For the feature map output by the boundary detail branch, which has been extracted from the original image data by the network, the gradients of this feature map along the horizontal and vertical directions at each spatial location are first calculated. Then, a 2×2 structure tensor matrix is ​​constructed using these two gradient values. This matrix is ​​then subjected to eigenvalue decomposition, yielding two eigenvalues. The larger eigenvalue represents the intensity of the principal direction at that location, and the smaller eigenvalue represents the intensity in the vertical direction. The difference between the two eigenvalues ​​is divided by their sum. To prevent division by zero, a very small positive number, such as 0.00001, can be added. The result is the output of the structure descriptor operator, which is between 0 and 1, representing the coherence or directional consistency of the local boundary structure. For the boundary detail feature map of the k-th intervention image data, its structure descriptor operator output is calculated in the same way. Finally, the Euclidean distance between the structure descriptor operator output corresponding to the original image data and the structure descriptor operator output corresponding to the intervention image data is calculated, i.e., the square root of the sum of the squares of the differences calculated pixel by pixel, which serves as a component of the pairwise structure consistency loss. In another embodiment, the structural description operator may employ an oriented gradient histogram or a normalized gradient orientation field.

[0100] This constraint ensures that the original image data and the intervention image data remain consistent in terms of boundary location, fine structural connectivity, and topological morphology.

[0101] Selective domain adversarial approaches enhance cross-device robustness while avoiding the decline in semantic recognition capabilities caused by excessive domain removal. Multi-scale structural consistency constraints explicitly guide the model to learn boundary and topological representations that remain invariant under different image signal processing perturbations, effectively solving problems such as boundary jitter, fine structure breakage, or topological distortion.

[0102] S600, such as Figure 5 As shown, a device region mask and its reliability index are determined based on the extracted device features. Adaptive modulation fusion is then performed on the constrained semantic features and constrained boundary detail features based on the reliability index to generate output features. The output information of the device branch includes confidence, prediction entropy, and calibration score. The reliability index of the device region mask is one or more of confidence, prediction entropy, and calibration score. In this embodiment, the reliability index of the device region mask includes confidence, prediction entropy, and calibration score.

[0103] First, the output of the instrument branch. After passing through a prediction head, the instrument region mask is obtained. and reliability indicators In this embodiment, This is a spatial distribution map calculated based on the confidence level, prediction entropy, and calibration score of the instrument branch output. The value ranges from [0,1], with higher values ​​indicating more reliable instrument region identification at that location. The reliability index can also be normalized and taken within the interval [0,1], where larger values ​​indicate higher reliability. This indicates that the instrument region judgment result has high reliability and is relatively small. This indicates a low level of credibility.

[0104] Specifically, this embodiment provides a method for calculating a reliability index. The prediction confidence map output by the device branch represents the probability that each pixel belongs to the device region, ranging from 0 to 1. The prediction entropy map is calculated from the soft prediction probability of the device branch and normalized to the 0-1 range; a larger value indicates higher prediction uncertainty. The calibration score map, ranging from 0 to 1, represents the degree of calibration between the model's predicted probability and the actual accuracy, and can be pre-calculated using methods such as temperature scaling or ordinal-preserving regression on the validation set. The comprehensive reliability index is calculated pixel-by-pixel using the following weighted fusion method: the confidence level, the result of subtracting the prediction entropy from 1, and the calibration score are weighted and summed, where the weight of the confidence level is set to 0.5, the weight of the prediction entropy complement is set to 0.3, and the weight of the calibration score is set to 0.2. The final reliability index value is still in the 0-1 range; a higher value indicates a more reliable identification of the device region at that location. In other embodiments of this application, the product of the confidence level and the calibration score can also be directly used as the reliability index. The confidence map, prediction entropy map, and calibration score map mentioned above all maintain the same spatial resolution as the original input image, or are calculated after being upsampled and aligned to the same resolution.

[0105] Then, feature fusion is performed. The semantic features after the selective domain invariance constraint are then... Boundary detail features after structural consistency constraints By performing basic fusion, we obtain:

[0106]

[0107] In the formula, This refers to the fusion of semantic features after applying selective domain invariance constraints and boundary detail features after applying multi-scale structural consistency constraints. The feature fusion operation can be performed by convolution after feature concatenation, channel-weighted summation, attention fusion, residual fusion, or other equivalent fusion methods.

[0108] Finally, a reliability modulation fusion is introduced, dynamically adjusting the fusion strategy based on the reliability of device identification. The adaptive modulation fusion is expressed as:

[0109]

[0110] In the formula, For output features, As a reliability indicator, The occlusion suppression coefficient is a monotonic variable that changes with the reliability index. For the instrument area mask, This indicates element-wise operations. This refers to the fusion of semantic features after applying selective domain invariance constraints and boundary detail features after applying multi-scale structural consistency constraints. Represents the semantic compensation coefficient. The semantic features are those after applying the selective domain invariance constraint.

[0111] In this embodiment, the following settings are provided: Furthermore, the occlusion suppression coefficient is positively correlated with the reliability index, and the semantic compensation coefficient is negatively correlated with the reliability index. Specifically, ,when When it is a scalar, and Broadcast to the spatial dimension and then participate in the calculation; when When it is a spatial distribution map, and These are the pixel-by-pixel or region-by-region modulation coefficients corresponding to the spatial location of the feature map.

[0112] When the device identification result is reliable, that is When it is large, Larger It strongly suppresses the instrument region, effectively eliminating the interference of instrument occlusion on segmentation, while semantically compensating for path weights. Reduced. When the instrument recognition result is unreliable. When smaller, Approaching 0, the suppression of fusion features is very weak, while... The larger feature size leads to a regression in the fusion process to a more robust pure semantic feature path. This avoids catastrophic segmentation errors caused by instrument misdiagnosis. This mechanism is a quantifiable risk control strategy that provides a failure-safe mode for the model under adverse conditions such as complex occlusion, reflection, and smoke, significantly improving the stability of the system's clinical deployment and overcoming catastrophic degradation in the worst-case scenario.

[0113] S700, generate training segmentation results based on the output features, and update the parameters of the multi-branch surgical visual segmentation network based on a preset loss function to obtain the trained multi-branch surgical visual segmentation network; the preset loss function includes a main segmentation loss term, a loss term based on multi-scale structural consistency constraints, a device branch supervision loss term, and a selective domain invariance constraint loss term.

[0114] Output features The final segmentation prediction result is generated after passing through a segmentation head, such as a 1×1 convolutional layer. During the training phase, this segmentation result is called the training segmentation result. The preset loss function... Represented as:

[0115]

[0116] In the formula, The main segmentation loss term can be cross-entropy loss, Dice loss, IoU loss, or a combination thereof, to measure... With real labels Differences; Weights for structural consistency loss. For the loss term based on multi-scale structural consistency constraints, For the weight of the monitoring loss of the device branch, The instrument branch supervision loss term can be the instrument region binary classification loss, focus loss, or boundary supervision loss, used to supervise the generation of the instrument region mask; For the loss weights that are selected by the invariant constraint of the selection field, This is a loss term that selects the domain invariant constraint, which is implemented through domain adversarial methods.

[0117] The parameters of the segmentation network, including the semantic branch, boundary detail branch, instrument branch, and fusion module, are iteratively updated using the backpropagation algorithm. and the parameters of the domain discriminator The training continues until the total loss converges, resulting in a fully trained cross-device generalized segmentation model. This joint optimization design ensures that multiple constraint modules work together, achieving an optimal balance between segmentation accuracy, cross-device robustness, and resistance to complex interference in the final model.

[0118] The S800 inputs the surgical image data to be processed into the trained multi-branch surgical visual segmentation network to generate the target segmentation result.

[0119] During the inference phase, the surgical image data or video frames to be processed are input into the trained model. This image data can come from any new, untrained imaging device. The model performs forward computation, skipping steps related to the domain discriminator and in-frame intervention pairing, and directly generates the segmentation result through semantic branching, boundary detail branching, instrument branching, and reliability modulation fusion modules. This segmentation result is called the target segmentation result.

[0120] In practical deployments, the model can be converted to ONNX, TensorRT, or other inference engine executable formats to achieve lower latency and higher throughput. For 4K video input, depending on hardware resources and clinical real-time requirements, whole-frame inference, sliding-window inference, scaled inference, or region-first inference methods can be employed. Finally, the generated pixel-level segmentation results are overlaid and displayed in real-time on the original surgical video for intraoperative navigation, visualization of key structures, risk area warnings, or to assist surgical decision-making.

[0121] The system can dynamically adjust the occlusion suppression intensity and semantic compensation intensity based on changes in the instrument area reliability index during the inference phase, so as to enhance the stability of the model under conditions of surgical field reflection, smoke, rapid instrument movement, or severe occlusion.

[0122] The efficient deployment strategy during the inference phase ensures real-time performance, and the dynamic application of reliability indicators further enhances stability in complex surgical fields, giving this method direct clinical value.

[0123] To verify the model's cross-device generalization ability and guide model tuning, this method may also include an auxiliary device proxy feature evaluation step. After training, this step extracts proxy features reflecting device imaging characteristics, such as Lab color space statistics, color gamut distribution differences, gray-level co-occurrence matrix texture features, gradient magnitude spectrum, high-frequency energy ratio, or other statistics that characterize sharpening, denoising, and color mapping differences, and evaluates the features extracted by the model. The degree of decline in the predictability of these device agent characteristics is assessed. A higher degree of decline indicates a better suppression effect of the model on device-related factors. This step can be used for model screening, system debugging, or quantitative verification of cross-device generalization ability. This section can serve as an auxiliary step in training evaluation, model screening, or system debugging. Adopting the above approach helps to objectively screen and optimize the model before actual deployment, enhancing the interpretability and engineering reliability of the system.

[0124] Example 2:

[0125] A cross-device generalized surgical visual segmentation system is used to execute the aforementioned cross-device generalized surgical visual segmentation method, including a data acquisition module, an intervention generation module, a network construction module, a feature extraction module, a domain invariant constraint module, a structural consistency constraint module, an instrument region and reliability estimation module, a feature fusion module, a training and update module, and an output module.

[0126] The data acquisition module is used to acquire surgical image data, as well as segmentation labels and device domain information corresponding to the image data, and to construct a training sample set.

[0127] The intervention generation module is used to apply a preset image signal processing intervention to each image data in the training sample set, adjust the imaging style of the image data, and generate corresponding intervention image data; and pair each image data with its corresponding intervention image data to construct a same-frame intervention pairing sample.

[0128] The network construction module is used to construct a multi-branch surgical visual segmentation network, which includes semantic branches, boundary detail branches, and instrument branches.

[0129] The feature extraction module is used to input the image data in the same frame intervention pairing sample and its corresponding intervention image data into the multi-branch surgical visual segmentation network constructed by the network construction module, and extract semantic features, boundary detail features and instrument features respectively.

[0130] The domain invariant constraint module is used to apply selective domain invariant constraints to a preset subset of semantic features to suppress the predictability of device domain information.

[0131] The structural consistency constraint module is used to apply multi-scale structural consistency constraints to the boundary detail features, so that the image data and the intervention image data are consistent in boundary morphology and topology.

[0132] The instrument region and reliability estimation module is used to determine the instrument region mask and the reliability index of the instrument region mask based on the extracted instrument features.

[0133] The feature fusion module is used to adaptively modulate and fuse the constrained semantic features and constrained boundary detail features based on the reliability index to generate output features.

[0134] The training update module is used to generate training segmentation results based on the output features and update the parameters of the multi-branch surgical visual segmentation network based on a preset loss function to obtain the trained multi-branch surgical visual segmentation network.

[0135] The output module is used to input the surgical image data to be processed into the trained multi-branch surgical visual segmentation network to generate the target segmentation result.

[0136] Example 3:

[0137] This embodiment provides an electronic device, including a computer-readable storage medium and a processor.

[0138] The computer-readable storage medium stores a computer program, which includes computer program code. The computer program code can be in the form of source code, object code, executable file, or some intermediate form. When the processor executes the computer program, it implements the cross-device generalized surgical visual segmentation method described in Embodiment 1.

[0139] The processor may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices. The processor is connected to the computer-readable storage medium via a bus or other communication interface.

[0140] In a preferred embodiment of this invention, the electronic device is an image processing workstation in a surgical navigation system, or an embedded processing unit integrated into a laparoscopic or endoscopic device. The device also includes input / output interfaces for connecting surgical imaging equipment, a display, and a user interaction device to achieve real-time display of segmentation results and intraoperative decision support.

[0141] The electronic device provided in this embodiment, by storing and executing the computer program of the above-mentioned cross-device generalized surgical visual segmentation method, can operate stably in multi-center, multi-brand, and multi-model imaging device environments, has real-time segmentation capabilities, and effectively solves the problem of decreased segmentation accuracy of existing devices under different imaging conditions.

[0142] Example 4:

[0143] A computer-readable storage medium for storing computer-executable instructions that, when executed, implement the cross-device generalized surgical visual segmentation method described in Embodiment 1 above.

[0144] The computer-executable instructions include program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium may be any medium capable of carrying, storing, or transmitting the computer program code, including but not limited to: USB flash drives, external hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, registers, solid-state drives (SSDs), and cloud storage space. Furthermore, the computer-readable storage medium may also include electrical carrier signals, telecommunication signals, and software distribution media for downloading and installing the computer program over a network.

[0145] In a preferred implementation of this embodiment, the computer-readable storage medium is a non-transient storage medium, that is, it does not include transient propagation signals.

[0146] The computer-readable storage medium provided in this embodiment can be conveniently distributed, deployed, and updated with the above-mentioned cross-device generalized surgical visual segmentation method, enabling laparoscopic or endoscopic systems in different medical institutions to quickly obtain cross-device generalization capabilities, reducing the technical threshold and maintenance costs of multi-center deployment.

[0147] The above are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A surgical visual segmentation method that generalizes across devices, characterized in that: Includes the following steps: S100: Acquire surgical image data, as well as segmentation labels and device domain information corresponding to the image data, and construct a training sample set; S200, apply a preset image signal processing intervention to each image data in the training sample set, adjust the imaging style of the image data, and generate corresponding intervention image data; pair each image data with its corresponding intervention image data to construct same-frame intervention paired samples; S300, construct a multi-branch surgical visual segmentation network, the multi-branch surgical visual segmentation network including semantic branches, boundary detail branches and instrument branches; S400, input the image data in the same frame intervention pairing sample and its corresponding intervention image data into the multi-branch surgical visual segmentation network, and extract semantic features, boundary detail features and instrument features respectively; S500, apply selective domain invariance constraints to a preset subset of semantic features to suppress the predictability of device domain information; and apply multi-scale structural consistency constraints to the boundary detail features to ensure that the image data and the intervention image data are consistent in boundary morphology and topology. S600: Determine the instrument region mask and the reliability index of the instrument region mask based on the extracted instrument features, and perform adaptive modulation fusion on the constrained semantic features and constrained boundary detail features based on the reliability index to generate output features. S700, generate training segmentation results based on the output features, and update the parameters of the multi-branch surgical visual segmentation network based on a preset loss function to obtain the trained multi-branch surgical visual segmentation network. S800 inputs the surgical image data to be processed into the trained multi-branch surgical visual segmentation network to generate the target segmentation result; In S500, the selective domain invariance constraint is implemented through a domain discriminator, and its optimization objective is: In the formula, To segment network parameters, For domain discriminator parameters, This refers to a predefined subset of features selected from semantic features. For device domain information, This is the probability that the domain discriminator predicts for device domain information based on the preset feature subset. For the expected operation, The main segmentation loss term, For the loss weights that are selected by the invariant constraint of the selection field, , This is the loss term for the selectivity-domain invariance constraint; In S500, the multi-scale structural consistency constraint is implemented at at least two scales, and the loss based on the multi-scale structural consistency constraint is expressed as: In the formula, For the loss term based on multi-scale structural consistency constraints, Weights for boundary topological continuity loss. For the boundary topological continuity loss term, For pairwise structural consistency loss weights, For the index of intervention types, For the expected operation of the index for all intervention types, For structural description operators, and The image data and the first Boundary detail features of interventional image data Represents image data, Indicates the first One interventional image data; In S600, the adaptive modulation fusion is represented as: In the formula, For output features, As a reliability indicator, The occlusion suppression coefficient is a monotonically varying factor that changes with reliability indicators. For the instrument area mask, This indicates element-wise operations. This refers to the fusion of semantic features after applying selective domain invariance constraints and boundary detail features after applying multi-scale structural consistency constraints. Indicates the semantic compensation coefficient. The semantic features after applying the selective domain invariance constraint; ; The occlusion suppression coefficient is positively correlated with the reliability index, and the semantic compensation coefficient is negatively correlated with the reliability index. The reliability index is a reliability distribution map corresponding to the spatial resolution of the image data; the occlusion suppression coefficient and semantic compensation coefficient are spatial distribution modulation coefficients corresponding to the spatial resolution.

2. The cross-device generalized surgical visual segmentation method according to claim 1, characterized in that: In S200, the image signal processing intervention includes one or more of the following: gamma adjustment, brightness perturbation, contrast perturbation, sharpening perturbation, denoising perturbation, color matrix perturbation, dynamic range perturbation, and quantization range perturbation.

3. The cross-device generalized surgical visual segmentation method according to claim 1, characterized in that: The output information of the device branch includes confidence level, prediction entropy, and calibration score; in S600, the reliability index of the device region mask is one or more of confidence level, prediction entropy, and calibration score.

4. The cross-device generalized surgical visual segmentation method according to claim 1, characterized in that: The preset loss function includes a main segmentation loss term, a loss term based on multi-scale structural consistency constraints, a device branch supervision loss term, and a selective domain invariance constraint loss term.

5. The cross-device generalized surgical visual segmentation method according to claim 4, characterized in that: The preset loss function Represented as: In the formula, The main segmentation loss term, Weights for structural consistency loss. For the loss term based on multi-scale structural consistency constraints, For the weight of the monitoring loss of the device branch, For the device branch supervision loss item, For the loss weights that are selected by the invariant constraint of the selection field, This is the loss term for the selection field invariance constraint.

6. A surgical visual segmentation system that can be generalized across devices, characterized in that: Used to perform the cross-device generalized surgical visual segmentation method as described in any one of claims 1-5.

7. The cross-device generalized surgical visual segmentation system according to claim 6, characterized in that: It includes a data acquisition module, an intervention generation module, a network construction module, a feature extraction module, a domain invariance constraint module, a structural consistency constraint module, an instrument region and reliability estimation module, a feature fusion module, a training and update module, and an output module; The data acquisition module is used to acquire surgical image data, as well as segmentation labels and device domain information corresponding to the image data, and to construct a training sample set. The intervention generation module is used to apply a preset image signal processing intervention to each image data in the training sample set, adjust the imaging style of the image data, and generate corresponding intervention image data; and pair each image data with its corresponding intervention image data to construct a same-frame intervention pairing sample. The network construction module is used to construct a multi-branch surgical visual segmentation network, which includes a semantic branch, a boundary detail branch, and an instrument branch. The feature extraction module is used to input the image data in the same frame intervention pairing sample and its corresponding intervention image data into the multi-branch surgical visual segmentation network constructed by the network construction module, and extract semantic features, boundary detail features and instrument features respectively. The domain invariant constraint module is used to apply selective domain invariant constraints to a preset subset of semantic features to suppress the predictability of device domain information. The structural consistency constraint module is used to apply multi-scale structural consistency constraints to the boundary detail features, so that the image data and the intervention image data are consistent in boundary morphology and topology. The instrument region and reliability estimation module is used to determine the instrument region mask and the reliability index of the instrument region mask based on the extracted instrument features. The feature fusion module is used to adaptively modulate and fuse the constrained semantic features and the constrained boundary detail features based on the reliability index to generate output features; The training update module is used to generate training segmentation results based on the output features and update the parameters of the multi-branch surgical visual segmentation network based on a preset loss function to obtain the trained multi-branch surgical visual segmentation network. The output module is used to input the surgical image data to be processed into the trained multi-branch surgical visual segmentation network to generate the target segmentation result.

8. An electronic device comprising a computer-readable storage medium and a processor, characterized in that: The computer-readable storage medium stores a computer program, and when the processor executes the computer program, it implements the cross-device generalized surgical visual segmentation method as described in any one of claims 1-5.

9. A computer-readable storage medium for storing computer-executable instructions, characterized in that: When executed, the computer-executable instructions implement the cross-device generalized surgical visual segmentation method as described in any one of claims 1-5.

Images