A sam-based unsupervised surgical instrument image segmentation method and system

Abstract

The application discloses a SAM-based unsupervised surgical instrument image segmentation method and system, which comprises the following steps: performing feature extraction on a surgical image containing a surgical instrument to generate a feature vector of the surgical image, performing dimension reduction processing on the feature vector, performing preliminary classification on the dimension-reduced feature vector through a mean clustering method, and generating a feature map after preliminary classification, optimizing the feature map through a conditional random field to generate an optimized feature map; segmenting the optimized feature map through a segmentation model SAM to generate a full segmentation result of the optimized feature map, and classifying each region in the full segmentation result through the optimized feature map to obtain a final surgical instrument image segmentation result, so that the surgical instrument image segmentation is completed.

Description

Technical Field

[0001] The present invention belongs to the technical field of unsupervised surgical instrument image segmentation, and more specifically, relates to a SAM-based unsupervised surgical instrument image segmentation method and system. Background Art

[0002] Instrument segmentation in minimally invasive surgery is fundamental to a variety of advanced computer-assisted intervention technologies, such as automated surgical skill assessment and intraoperative guidance systems. Given its importance, surgical instrument segmentation has achieved remarkable progress, ranging from early traditional methods to more recent approaches using deep learning. However, this success has primarily relied on supervised learning from large amounts of annotated data, which is prohibitively expensive and time-consuming to collect in medicine, particularly for segmentation tasks involving video data. Furthermore, the generalization ability of supervised methods is almost inevitably limited by the domain gaps between different hospitals and surgical procedures in the real world.

[0003] To address this issue, Daniil et al. attempted to leverage the robot's kinematic model and the instrument's 3D model parameters to automatically obtain labels by projecting them onto the camera's image plane. They then used a generative adversarial model to learn the mapping between the endoscope and the corresponding labels, eliminating label errors caused by imprecise kinematics. However, due to the lack of accurate annotation during the label generation process, the effects of label uncertainty persist in the generative adversarial model, which can reduce segmentation performance.

[0004] Pfeiffer et al. used synthetic datasets to obtain labels for model training. However, synthetic datasets may not fully simulate real surgical scenarios in some cases, and there are still some gaps between them and real datasets. Furthermore, the dataset is highly correlated with the model's generalization ability, which can have a certain impact on practical applications.

[0005] Sestini automatically acquires labels from existing datasets and green screen images to train a shape prior model. It then trains a surgical instrument segmentation model by combining the optical flow of surgical videos with pseudo-labels generated by the shape prior model. This approach achieves nearly the same segmentation results as supervised learning, but the preparation is cumbersome, and gradient descent, performed only on regions with high pseudo-label confidence, can compromise instrument edge detection. Summary of the Invention

[0006] To solve the above technical problems, the present invention proposes an unsupervised surgical instrument image segmentation method based on SAM, comprising:

[0007] Feature extraction is performed on a surgical image containing surgical instruments to generate a feature vector of the surgical image, dimensionality reduction is performed on the feature vector, the feature vector after dimensionality reduction is preliminarily classified by a mean clustering method, and a feature map after preliminary classification is generated, the feature map is optimized by a conditional random field to generate an optimized feature map;

[0008] The optimized feature map is segmented by a segmentation model SAM to generate a full segmentation result of the optimized feature map, and each region in the full segmentation result is classified by the optimized feature map to obtain a final surgical instrument image segmentation result, thereby completing surgical instrument image segmentation.

[0009] Further, the feature extraction on the surgical image containing surgical instruments includes:

[0010] The size of the surgical image is unified, and the unified surgical image is standardized: specifically,

[0011] The standardized parameters of the unified surgical image are:

[0012] mean={0.485, 0.456, 0.406}

[0013] std={0.229, 0.224, 0.225}

[0014] Wherein, mean is the standardized mean of each channel of the unified surgical image, and std is the standard deviation of each channel of the unified surgical image.

[0015] The DINOv2 model is used to extract features from the standardized surgical image.

[0016] Further, the dimensionality reduction of the feature vector includes:

[0017] The feature vector is converted into a matrix, where each row represents a channel and each column represents a pixel value of the channel, and the feature vector matrix is calculated by the following formula:

[0018]

[0019] Wherein, Cov is the covariance matrix of the feature vector X, X mean is the mean vector of X, V is the feature vector matrix, D is the focus matrix, the elements on the diagonal are eigenvalues, and n is the number of channels.

[0020] According to the eigenvalues, the feature vectors are sorted from large to small, and the first three feature vectors are selected, and the selected feature vectors are multiplied by the centralized input features to obtain a feature matrix after dimensionality reduction, and the formula is as follows:

[0021] X pca = V s T (X-X mean )

[0022] wherein X pca ∈R 3×20×20 is a 3-channel 20-long 20-wide feature vector after dimension reduction, V s is the first 3 selected feature vectors.

[0023] Further, the preliminary classification of the feature vectors after dimension reduction by the mean clustering method comprises: randomly selecting two initial clustering centers in a three-dimensional space as initial values, for each pixel point data in the feature vector, calculating the Euclidean distance between it and each initial clustering center, and assigning it to the cluster corresponding to the initial clustering center with the smallest Euclidean distance;

[0024] For each cluster, the average value of all pixel point data in each cluster is calculated as a new clustering center, assuming that C is a cluster containing m vectors, the update formula of the clustering center is:

[0025]

[0026] Repeat the above steps until the clustering center no longer changes, that is, the preliminary classification is completed.

[0027] Further, the feature map is optimized by a conditional random field to generate an optimized feature map, comprising: modeling each pixel point in the feature map as a node according to the conditional random field, the label of the node being the class assigned in the K-mean, and finding the best configuration of the label of the node under the given observation value by maximizing the conditional probability, so as to complete the optimization processing of the classification result, specifically:

[0028] Introducing a data potential function and a smoothing potential function, wherein the data potential function is used to represent the relationship between the observation value of each pixel point and the label, and the data potential function is:

[0029] U(x i ,y i ) = -log(ψ(x i ,y i ))

[0030] wherein x i represents the observation value of the i-th pixel point, y i represents the label of the i-th pixel point, and ψ(x i ,y i ) represents the latent relationship between the observation value of the i-th pixel point and the label of the i-th pixel point.

[0031] The smooth potential energy represents the relationship between adjacent pixel points, and is used to constrain the labels of adjacent pixel points, so that the label change is relatively small. The smooth potential energy function is:

[0032]

[0033] where y i and y j represent the label of the i-th pixel point and the label of the j-th pixel point, respectively, λ is the weight of the smooth term, p i and p j represent the position coordinates of the i-th pixel point and the position coordinates of the j-th pixel point, respectively, and ||p i -p j || represents the Euclidean distance, and σ is the standard deviation of the smooth term.

[0034] The energy function of the conditional random field model can be finally defined as:

[0035] E(Y|X)=∑U(x i ,y i )+λ∑P(y i ,y j )

[0036] The lower the energy function value, the higher the probability of the corresponding label configuration.

[0037] Further, the data potential energy function and the smooth potential energy function are optimized to make the feature map optimization more accurate, specifically:

[0038] The optimized data potential energy function is:

[0039] U(x i ,y i )=-log(ψ(x i ,y i ))+λ1R(x i ,y i )+λ2D(x i ,y i )+λ3Q(x i ,y i )

[0040] where λ1 is a regularization coefficient, λ2 is a weight coefficient of the data term, and λ3 is a weight coefficient of the prior term. R(x i ,y i ) is a regularization term, D(x i ,y i ) is a data term, which is used to model the relationship between the feature map and the attribute, and is calculated by the cross-entropy loss method. Q(x i ,y i) is a penalty term used to introduce prior knowledge or other constraints;

[0041] The optimized smooth potential energy function is:

[0042]

[0043] Wherein, K is the number of additional terms, β k is a weight coefficient, f k (y i , y j ) is a texture similarity function or an edge consistency function.

[0044] The application also provides a SAM-based unsupervised surgical instrument image segmentation system, comprising:

[0045] The feature map acquisition module is configured to extract features from the surgical image containing the surgical instrument, generate a feature vector of the surgical image, perform dimensionality reduction processing on the feature vector, preliminarily classify the dimensionally reduced feature vector by a mean clustering method, and generate a feature map after preliminary classification, and optimize the feature map by a conditional random field to generate an optimized feature map.

[0046] The segmentation module is configured to segment the optimized feature map by a segmentation model SAM to generate a full segmentation result of the optimized feature map, and classify each region in the full segmentation result by the optimized feature map to obtain a final surgical instrument image segmentation result, thereby completing the surgical instrument image segmentation.

[0047] Further, the feature extraction from the surgical image containing the surgical instrument comprises:

[0048] The size of the surgical image is unified, and the unified surgical image is standardized: specifically:

[0049] The standardized parameters of the unified surgical image are:

[0050] mean = {0.485, 0.456, 0.406}

[0051] std = {0.229, 0.224, 0.225}

[0052] Wherein, mean is the standardized mean of each channel of the unified surgical image, and std is the standard deviation of each channel of the unified surgical image.

[0053] The DINOv2 model is used to extract features from the standardized surgical image.

[0054] Further, the dimensionality reduction processing of the feature vector comprises:

[0055] The feature vector is converted into a matrix, where each row represents a channel, and each column represents a pixel value of a channel, and the feature vector matrix is calculated by the following formula:

[0056]

[0057] where Cov is the covariance matrix of the feature vector X, X mean is the mean vector of X, V is the feature vector matrix, D is the focus matrix, the elements on the diagonal are the eigenvalues, and n is the number of channels;

[0058] According to the eigenvalues, the feature vectors are sorted from large to small, and the first 3 feature vectors are selected. The selected feature vectors are multiplied by the centralized input features to obtain the reduced feature matrix, and the formula is as follows:

[0059] X pca =V s T (X-X mean )

[0060] where X pca ∈R 3×20×20 is the reduced 3-channel 20-long 20-wide feature vector, and V s is the first 3 selected feature vectors.

[0061] Further, the reduced feature vectors are preliminarily classified by the mean clustering method, including: randomly selecting two initial clustering centers in three-dimensional space as initial values, for each pixel point data in the feature vector, calculating the Euclidean distance between it and each initial clustering center, and assigning it to the cluster corresponding to the initial clustering center with the smallest Euclidean distance;

[0062] For each cluster, the average value of all pixel point data in the cluster is calculated as a new clustering center. Assuming that C is a cluster containing m vectors, the update formula of the clustering center is:

[0063]

[0064] Repeat the above steps until the clustering center no longer changes, i.e., the preliminary classification is completed.

[0065]

[0066] where y i and y j represent the label of the i-th pixel point and the label of the j-th pixel point, respectively, λ is the weight of the smoothing term, p i and p j represent the position coordinates of the i-th pixel point and the position coordinates of the j-th pixel point, respectively, and ||pi -p j || represents the Euclidean distance, and sigma is the standard deviation of the smoothing term.

[0067] Finally, the energy function of the conditional random field model can be defined as:

[0068] E(Y|X) = ∑U(x i ,y i )+λ∑P(y i ,y j )

[0069] The lower the energy function value, the higher the probability of the corresponding label configuration.

[0070] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0071] 1. The present application completes instrument segmentation in a surgical scene through an unsupervised method, without the need for manual annotation of data, thereby reducing labor and time costs.

[0072] 2. The present application uses DINOv2 to extract image features and SAM to extract image boundary information. Existing large models can effectively extract image features and have high generalization ability, and can be applied to different surgical scenes.

[0073] 3. The present application uses principal component analysis to reduce the dimensionality of the feature map, accelerating the computer operation process and making the mean clustering process converge faster.

[0074] 4. The present application considers the position information of the preliminary segmentation and the boundary information of the full segmentation of SAM, so that the segmentation effect of the unsupervised method of the present application can approach supervised learning. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 is a flowchart of the method of embodiment 1 of the present application;

[0076] Figure 2 is a structural diagram of the system of embodiment 2 of the present application;

[0077] Figure 3 is an image feature extraction process diagram of embodiment 1 of the present application;

[0078] Figure 4 is a full segmentation diagram of the image using SAM of embodiment 1 of the present application;

[0079] Figure 5 is a whole instance flowchart of surgical instrument segmentation of embodiment 1 of the present application. DETAILED DESCRIPTION

[0080] For better understanding of the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings and specific embodiments of the specification.

[0081] The method provided by the application can be implemented in a terminal environment, which can include one or more of the following components: a processor, a storage medium and a display screen. The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the method described in the following embodiments.

[0082] The processor can include one or more processing cores. The processor connects various parts in the entire terminal through various interfaces and lines, executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the storage medium, and calling data stored in the storage medium.

[0083] The storage medium can include random access memory (RAM) and read-only memory (ROM). The storage medium can be used to store instructions, programs, codes, code sets or instructions.

[0084] The display screen is used to display the user interface of each application.

[0085] All subscripts in the formula of the application are only used to distinguish parameters and have no actual meaning.

[0086] The SAM (Segment Anything Model) model. This model is different from the traditional segmentation model. The traditional segmentation model can only input the original image and output a fixed segmentation result. The SAM can input the original image and specific prompts (points, boxes, shadows, text) at the same time, then output different segmentation result images according to different prompts, and the SAM supports interactive segmentation of different prompts.

[0087] The DINOv2 model is a computer vision self-supervised model published by Meta, which can achieve SOTA level performance in tasks such as depth estimation, semantic segmentation and image similarity comparison.

[0088] In addition, those skilled in the art can understand that the structure of the terminal described above does not constitute a limitation on the terminal, and the terminal can include more or fewer components, or combine certain components, or different component arrangements. For example, the terminal also includes radio frequency circuit, input unit, sensor, audio circuit, power supply and other components, which will not be described here.

[0089] Embodiment 1

[0090] As Figure 1As shown, the embodiment of the present application provides a SAM-based unsupervised surgical instrument image segmentation method, comprising:

[0091] Step 101, feature extraction is performed on the surgical image containing the surgical instrument to generate a feature vector of the surgical image, dimension reduction is performed on the feature vector, the dimension-reduced feature vector is preliminarily classified through a mean clustering method, and a feature map after preliminary classification is generated, the feature map is optimized through a conditional random field, and an optimized feature map is generated;

[0092] Specifically, the feature extraction on the surgical image containing the surgical instrument comprises:

[0093] The size of the surgical image is unified (for example, 560x560 size), and the unified surgical image is standardized: specifically,

[0094] The standardized parameters of the unified surgical image are:

[0095] mean={0.485, 0.456, 0.406}

[0096] std={0.229, 0.224, 0.225}

[0097] Wherein, mean is the standardized mean of each channel of the unified surgical image, and std is the standard deviation of each channel of the unified surgical image.

[0098] The DINOv2 model is used to extract features from the standardized surgical image, as shown in Figure 3 I∈R 3 ×560×560 is a 3-channel RGB input picture, X∈R 1024×20×20 is a 1024-channel DINOv2 feature vector.

[0099] Specifically, the dimension reduction processing on the feature vector comprises:

[0100] The feature vector is converted into a matrix (for example, a matrix with a size of 1024x400), wherein each row represents a channel, and each column represents a pixel value of a channel, and the feature vector matrix is calculated through the following formula:

[0101]

[0102] Wherein, Cov is the covariance matrix of the feature vector X, X mean is the mean vector of X, V is the feature vector matrix, D is the focusing matrix, the elements on the diagonal line are the eigenvalues, and n is the number of channels.

[0103] According to the eigenvalues, the eigenvectors are sorted from large to small, the first three eigenvectors are selected, the selected eigenvectors are multiplied with the centralized input features, and a feature matrix after dimension reduction is obtained, and the formula is as follows:

[0104] X pca =V s T (X-X mean )

[0105] Wherein, X pca ∈R 3×20×20 is a 3-channel 20-long 20-wide eigenvector after dimension reduction, and V s is the first three selected eigenvectors.

[0106] Specifically, the mean clustering method is used to preliminarily classify the eigenvectors after dimension reduction, including: randomly selecting two initial clustering centers in three-dimensional space as initial values, calculating the Euclidean distance between each pixel point data in the eigenvector and each initial clustering center, and distributing it to the cluster corresponding to the initial clustering center with the smallest Euclidean distance.

[0107] For each cluster, the average value of all pixel point data in each cluster is calculated, which is used as a new clustering center. Assuming that C is a cluster containing m vectors, the update formula of the clustering center is:

[0108]

[0109] Repeat the above steps until the clustering center no longer changes, that is, the preliminary classification is completed.

[0110] Specifically, the feature map is optimized by a conditional random field to generate an optimized feature map, including: according to the conditional random field, each pixel point in the feature map is modeled as a node, and the label of the node is the class assigned in the K-mean. The best configuration of the label of the node under the given observation value is found by maximizing the conditional probability, so as to complete the optimization processing of the classification result, and the specific process is as follows:

[0111] Data potential function and smoothing potential function are introduced, wherein the data potential function is used to represent the relationship between the observation value of each pixel point and the label, and the data potential function is as follows:

[0112] U(x i ,y i )=-log(ψ(x i ,y i ))

[0113] Wherein, x i represents the observation value of the i-th pixel point, y i represents the label of the i-th pixel point, and ψ(xi ,y i ) represents the latent relationship between the observation value of the i-th pixel point and the label of the i-th pixel point;

[0114] The smooth potential energy represents the relationship between adjacent pixel points, and is used to constrain the labels of adjacent pixel points, so that the label change is relatively small. The smooth potential energy function is:

[0115]

[0116] where y i and y j represent the label of the i-th pixel point and the label of the j-th pixel point, respectively, λ is the weight of the smooth term, p i and p j represent the position coordinates of the i-th pixel point and the position coordinates of the j-th pixel point, respectively, and ||p i -p j || represents the Euclidean distance, and σ is the standard deviation of the smooth term.

[0117] The energy function of the conditional random field model can be finally defined as:

[0118] E(Y|X)=∑U(x i ,y i )+λ∑P(y i ,y j )

[0119] The lower the energy function value, the higher the probability of the corresponding label configuration.

[0120] Specifically, the data potential energy function and the smooth potential energy function are optimized to make the feature map optimization more accurate, specifically:

[0121] The optimized data potential energy function is:

[0122] U(x i ,y i )=-log(ψ(x i ,y i ))+λ1R(x i ,y i )+λ2D(x i ,y i )+λ3Q(x i ,y i )

[0123] where λ1 is a regularization coefficient, λ2 is a weight coefficient of the data term, and λ3 is a weight coefficient of the prior term. R(x i ,y i ) is a regularization term, D(x i ,y i) is a data item for modeling the relationship between the feature map and the attribute, and the data item is calculated by the method of cross-entropy loss, Q(x i ,y i ) is a penalty term for introducing prior knowledge or other constraints;

[0124] The optimized smooth potential energy function is:

[0125]

[0126] Wherein, K is the number of additional terms, β k is a weight coefficient, f k (y i , y j ) is a texture similarity function or an edge consistency function.

[0127] Step 102, segment the optimized feature map by the segmentation model SAM to generate the full segmentation result of the optimized feature map, and classify each region in the full segmentation result by the optimized feature map to obtain the final surgical instrument image segmentation result, so as to complete the surgical instrument image segmentation, as shown in Figure 4 .

[0128] Specifically, as shown in Figure 5 , it further includes: based on the conditional random field segmentation result of the approximate position of the surgical instrument obtained in step 101, classifying each region in the full segmentation result with the object accurate boundary obtained in step 102 to obtain the final instrument segmentation result. The region classification in the full segmentation result is determined according to the category with the largest pixel ratio in each region. The expression of this process is as follows:

[0129]

[0130] Wherein, Y r is the category to which the full segmentation result belongs in the region r, sum(r,y=0) is the number of pixel points belonging to category 0 in the region r in the conditional random field segmentation result, sum(r,y=1) is the number of pixel points belonging to category 1 in the region r, and y is the category. The classification processing is performed on each region in the full segmentation result, that is, the final segmentation result is obtained.

[0131] Embodiment 2

[0132] As shown in Figure 2 , the embodiment of the application further provides a SAM-based unsupervised surgical instrument image segmentation system, which comprises:

[0133] The acquisition feature map module is configured to perform feature extraction on the surgical image containing the surgical instrument, generate a feature vector of the surgical image, perform dimension reduction processing on the feature vector, perform preliminary classification on the dimension-reduced feature vector through a mean clustering method, and generate a preliminary classified feature map, and perform optimization on the feature map through a conditional random field to generate an optimized feature map.

[0134] Specifically, the feature extraction on the surgical image containing the surgical instrument includes:

[0135] The size of the surgical image is unified, and the unified surgical image is standardized: specifically,

[0136] The standardized parameters of the unified surgical image are:

[0137] mean = {0.485, 0.456, 0.406}

[0138] std = {0.229, 0.224, 0.225}

[0139] wherein mean is the standardized mean of each channel of the unified surgical image, and std is the standard deviation of each channel of the unified surgical image.

[0140] The DINOv2 model is used to extract features from the standardized surgical image.

[0141] Specifically, the dimension reduction processing on the feature vector includes:

[0142] The feature vector is converted into a matrix, wherein each row represents a channel, and each column represents a pixel value of the channel, and the feature vector matrix is calculated by the following formula:

[0143]

[0144] wherein Cov is the covariance matrix of the feature vector X, X mean is the mean vector of X, V is the feature vector matrix, D is the focus matrix, the elements on the diagonal are eigenvalues, and n is the number of channels.

[0145] According to the eigenvalues, the feature vectors are sorted from large to small, and the first three feature vectors are selected, and the selected feature vectors are multiplied by the centralized input features to obtain a dimension-reduced feature matrix, and the formula is as follows:

[0146] X pca = V s T (X-X mean )

[0147] wherein X pca ∈R 3×20×20V is a 3-channel 20-long 20-wide feature vector after dimension reduction s V is the first three selected feature vectors.

[0148] Specifically, the preliminary classification of the feature vector after dimension reduction by the mean clustering method includes: randomly selecting two initial clustering centers as initial values in a three-dimensional space, for each pixel point data in the feature vector, calculating the Euclidean distance between it and each initial clustering center, and assigning it to the cluster corresponding to the initial clustering center with the smallest Euclidean distance;

[0149] For each cluster, calculate the average value of all pixel point data in each cluster, and take it as a new clustering center. Assuming that C is a cluster containing m vectors, the update formula of the clustering center is:

[0150]

[0151] Repeat the above steps until the clustering center no longer changes, that is, the preliminary classification is completed.

[0152] Specifically, the optimization of the feature map by the conditional random field includes: modeling each pixel point in the feature map as a node according to the conditional random field, the label of the node being the class assigned in the K-mean, and finding the best configuration of the label of the node under the given observation value by maximizing the conditional probability, so as to complete the optimization processing of the classification result, specifically:

[0153] Introduce a data potential function and a smoothing potential function, wherein the data potential function is used to represent the relationship between the observation value of each pixel point and the label, and the data potential function is:

[0154] U(x i ,y i )=-log(ψ(x i ,y i ))

[0155] Where x i represents the observation value of the i-th pixel point, y j represents the label of the i-th pixel point, and ψ(x i ,y j ) represents the latent relationship between the observation value of the i-th pixel point and the label of the i-th pixel point.

[0156] The smoothing potential represents the relationship between adjacent pixel points, and is used to constrain the labels of adjacent pixel points so that the label changes are small. The smoothing potential function is:

[0157]

[0158] Where y i and yj respectively represent the label of the i-th pixel point and the label of the j-th pixel point, λ is a weight of a smoothing term, p i and p j respectively represent the position coordinates of the i-th pixel point and the position coordinates of the j-th pixel point, ||p i -p j || represents a Euclidean distance, and σ is a standard deviation of the smoothing term.

[0159] Finally, an energy function of the conditional random field model can be defined as:

[0160] E(Y|X)=∑U(x i ,y i )+λ∑P(y i ,y j )

[0161] The lower the energy function value is, the higher the probability of the corresponding label configuration is.

[0162] Specifically, the data potential function and the smoothing potential function are optimized to make the feature map optimization more accurate, specifically:

[0163] The optimized data potential function is:

[0164] U(x i ,y i )=-log(ψ(x i ,y i ))+λ1R(x i ,y i )+λ2D(x i ,y i )+λ3Q(x i ,y i )

[0165] Wherein, λ1 is a regularization coefficient, λ2 is a weight coefficient of a data term, λ3 is a weight coefficient of a priori term, R(x i ,y i ) is a regularization term, D(x i ,y i ) is a data term, used to model the relationship between the feature map and the attribute, and the data term is calculated by a cross-entropy loss method, Q(x i ,y i ) is a penalty term, used to introduce priori knowledge or other constraint conditions.

[0166] The optimized smoothing potential function is:

[0167]

[0168] Wherein, K is the number of additional terms, β kis a weight coefficient, f k (y i , y j ) is a texture similarity function or an edge consistency function.

[0169] The segmentation module is configured to segment the optimized feature map by using a segmentation model (SAM), to generate a full segmentation result of the optimized feature map, and to classify each region in the full segmentation result by using the optimized feature map, so as to obtain a final surgical instrument image segmentation result, thereby completing the surgical instrument image segmentation.

[0170] Embodiment 3

[0171] The embodiment of the present application also provides a storage medium storing a plurality of instructions for implementing the SAM-based unsupervised surgical instrument image segmentation method.

[0172] Optionally, in the embodiment, the storage medium can be located in any one of computer terminals in a computer terminal group in a computer network, or in any one of mobile terminals in a mobile terminal group.

[0173] Optionally, in the embodiment, the storage medium is configured to store program codes for performing the following steps: step 101, performing feature extraction on a surgical image containing a surgical instrument to generate a feature vector of the surgical image, performing dimension reduction processing on the feature vector, performing preliminary classification on the dimension-reduced feature vector by using a mean clustering method, and generating a feature map after preliminary classification, and performing optimization on the feature map by using a conditional random field to generate an optimized feature map;

[0174] Specifically, the feature extraction on the surgical image containing the surgical instrument includes:

[0175] The size of the surgical image is unified, and the unified surgical image is standardized: specifically,

[0176] The standardized parameters of the unified surgical image are:

[0177] mean={0.485, 0.456, 0.406}

[0178] std={0.229, 0.224, 0.225}

[0179] wherein mean is the standardized mean of each channel of the unified surgical image, and std is the standard deviation of each channel of the unified surgical image.

[0180] The DINOv2 model is used to perform feature extraction on the standardized surgical image.

[0181] Specifically, the dimension reduction processing on the feature vector comprises:

[0182] The feature vector is converted into a matrix, each row of which represents a channel, and each column represents a pixel value of the channel, and the feature vector matrix is calculated by the following formula:

[0183]

[0184] wherein Cov is the covariance matrix of the feature vector X, X mean is the mean vector of X, V is the feature vector matrix, D is the focus matrix, the elements on the diagonal are eigenvalues, and n is the number of channels;

[0185] According to the eigenvalues, the feature vectors are sorted from large to small, the first three feature vectors are selected, and the selected feature vectors are multiplied by the centralized input features to obtain a feature matrix after dimension reduction, and the formula is as follows:

[0186] X pca =V s T (X-X mean )

[0187] wherein X pca ∈R 3×20×20 is a feature vector of 3 channels, 20 long and 20 wide after dimension reduction, and V s is the first three selected feature vectors.

[0188] Specifically, the preliminary classification of the feature vector after dimension reduction by the mean clustering method comprises: randomly selecting two initial clustering centers in a three-dimensional space as initial values, calculating the Euclidean distance between each pixel point data in the feature vector and each initial clustering center, and distributing it to the cluster corresponding to the initial clustering center with the smallest Euclidean distance;

[0189] For each cluster, the average value of all pixel point data in each cluster is calculated as a new clustering center. Assuming that C is a cluster containing m vectors, the update formula of the clustering center is:

[0190]

[0191] Repeat the above steps until the clustering center no longer changes, i.e., the preliminary classification is completed.

[0192] Specifically, the feature map is optimized by a conditional random field to generate an optimized feature map, including: modeling each pixel point in the feature map as a node according to the conditional random field, the label of the node being a class assigned in K-mean, and finding the best configuration of the label of the node under the given observation value by maximizing the conditional probability, so as to complete the optimization processing of the classification result, specifically:

[0193] A data potential function and a smoothing potential function are introduced, wherein the data potential function is used to represent the relationship between the observation value of each pixel point and the label, and the data potential function is:

[0194] U(x i ,y i )=-log(ψ(x i ,y i ))

[0195] Wherein, x i represents the observation value of the i-th pixel point, y i represents the label of the i-th pixel point, and ψ(x i ,y i ) represents the potential relationship between the observation value of the i-th pixel point and the label of the i-th pixel point.

[0196] The smoothing potential represents the relationship between adjacent pixel points, and is used to constrain the labels of adjacent pixel points so that the label changes are relatively small, and the smoothing potential function is:

[0197]

[0198] Wherein y i and y j represent the label of the i-th pixel point and the label of the j-th pixel point respectively, λ is the weight of the smoothing term, p i and p j represent the position coordinates of the i-th pixel point and the position coordinates of the j-th pixel point respectively, ||p i -p j || represents the Euclidean distance, and σ is the standard deviation of the smoothing term.

[0199] Finally, the energy function of the conditional random field model can be defined as:

[0200] E(Y|X)=ΣU(x i ,y i )+λΣP(y i ,y j )

[0201] The lower the energy function value, the higher the probability of the corresponding label configuration.

[0202] Specifically, the data potential function and the smooth potential function are optimized to make the feature map optimization more accurate, specifically:

[0203] The optimized data potential function is:

[0204] U(x i ,y i )=-log(ψ(x i ,y i ))+λ1R(x i ,y i )+λ2D(x i ,y i )+λ3Q(x i ,y i )

[0205] Wherein, λ1 is a regularization coefficient, λ2 is a weight coefficient of a data item, λ3 is a weight coefficient of a priori item, R(x i ,y i ) is a regularization term, D(x i ,y i ) is a data item, used to model the relationship between the feature map and the attribute, and the data item is calculated by the method of cross-entropy loss, Q(x i ,y i ) is a penalty term, used to introduce prior knowledge or other constraint conditions;

[0206] The optimized smooth potential function is:

[0207]

[0208] Wherein, K is the number of additional terms, β k is a weight coefficient, f k (y i , y j ) is a texture similarity function or an edge consistency function.

[0209] Step 102, the optimized feature map is segmented by the segmentation model SAM to generate a full segmentation result of the optimized feature map, and each region in the full segmentation result is classified by the optimized feature map to obtain a final surgical instrument image segmentation result, so as to complete the surgical instrument image segmentation.

[0210] Embodiment 4

[0211] The embodiment of the application also provides an electronic device comprising a processor and a storage medium connected with the processor, wherein the storage medium stores a plurality of instructions, the instructions can be loaded and executed by the processor, so that the processor can execute the SAM-based unsupervised surgical instrument image segmentation method.

[0212] In particular, the electronic device of the embodiment can be a computer terminal, which can include one or more processors and a storage medium.

[0213] The storage medium can be used to store software programs and modules, such as the unsupervised surgical instrument image segmentation method based on SAM in the embodiment of the present application, corresponding program instructions / modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the storage medium, that is, implements the above-mentioned unsupervised surgical instrument image segmentation method based on SAM. The storage medium can include a high-speed random storage medium, and can also include a non-volatile storage medium, such as one or more magnetic storage systems, flash memories, or other non-volatile solid-state storage media. In some examples, the storage medium can further include storage media remotely arranged with respect to the processor, which can be connected to the terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0214] The processor can call the information and application programs stored in the storage medium through the transmission system to perform the following steps: step 101, feature extraction is performed on the surgical image containing the surgical instrument to generate a feature vector of the surgical image, dimensionality reduction is performed on the feature vector, the feature vector after dimensionality reduction is preliminarily classified by a mean clustering method, and a feature map after preliminary classification is generated, the feature map is optimized by a conditional random field, and an optimized feature map is generated;

[0215] In particular, the feature extraction on the surgical image containing the surgical instrument includes:

[0216] The size of the surgical image is unified, and the unified surgical image is standardized: specifically:

[0217] The standardized parameters of the unified surgical image are:

[0218] mean = {0.485, 0.456, 0.406}

[0219] std = {0.229, 0.224, 0.225}

[0220] Wherein, mean is the standardized mean of each channel of the unified surgical image, and std is the standard deviation of each channel of the unified surgical image;

[0221] The DINOv2 model is used to extract features from the standardized surgical image.

[0222] In particular, the dimensionality reduction of the feature vector includes:

[0223] The feature vector is converted into a matrix, where each row represents a channel, and each column represents a pixel value of a channel, and the feature vector matrix is calculated by the following formula:

[0224]

[0225] where Cov is the covariance matrix of the feature vector X, X mean is the mean vector of X, V is the feature vector matrix, D is the focus matrix, the elements on the diagonal are the eigenvalues, and n is the number of channels;

[0226] According to the eigenvalues, the feature vectors are sorted from large to small, and the first three feature vectors are selected, and the selected feature vectors are multiplied by the centralized input features to obtain the reduced feature matrix, and the formula is as follows:

[0227] X pca =V s T (X-X mean )

[0228] where X pca ∈R 3×20×20 is the reduced 3-channel 20-long 20-wide feature vector, and V s is the first three selected feature vectors.

[0229] Specifically, the mean clustering method is used to preliminarily classify the reduced feature vectors, including randomly selecting two initial clustering centers as initial values in a three-dimensional space, calculating the Euclidean distance between each pixel data in the feature vector and each initial clustering center, and assigning it to the cluster corresponding to the initial clustering center with the smallest Euclidean distance.

[0230] For each cluster, the average value of all pixel data in each cluster is calculated as a new clustering center. Assuming that C is a cluster containing m vectors, the update formula of the clustering center is:

[0231]

[0232] Repeat the above steps until the clustering center no longer changes, i.e., the preliminary classification is completed.

[0233] Specifically, the feature map is optimized by a conditional random field to generate an optimized feature map, including modeling each pixel point in the feature map as a node according to the conditional random field, the label of the node being the class assigned in K-mean, and finding the best configuration of the label of the node under the given observation value by maximizing the conditional probability, so as to complete the optimization processing of the classification result, specifically:

[0234] A data potential function and a smooth potential function are introduced, wherein the data potential function is used to represent a relationship between an observation value of each pixel point and a label, and the data potential function is:

[0235] U(x i ,y i )=-log(ψ(x i ,y i ))

[0236] wherein x i represents an observation value of an i-th pixel point, y i represents a label of the i-th pixel point, and ψ(x i ,y i ) represents a potential relationship between the observation value of the i-th pixel point and the label of the i-th pixel point.

[0237] The smooth potential represents a relationship between adjacent pixel points, and is used to constrain the labels of adjacent pixel points so that the labels change relatively small, and the smooth potential function is:

[0238]

[0239] wherein y i and y j represent a label of an i-th pixel point and a label of a j-th pixel point respectively, λ is a weight of a smooth term, p i and p j represent a position coordinate of the i-th pixel point and a position coordinate of the j-th pixel point respectively, ||p i -p j || represents an Euclidean distance, and σ is a standard deviation of the smooth term.

[0240] An energy function of the conditional random field model can be finally defined as:

[0241] E(Y|X)=∑U(x i ,y i )+λ∑P(y i ,y j )

[0242] The lower the energy function value is, the higher the probability of the corresponding label configuration is.

[0243] Specifically, the data potential function and the smooth potential function are optimized so that the feature map optimization is more accurate, and specifically:

[0244] The optimized data potential function is:

[0245] U(x i ,y i )=-log(ψ(x i ,y i ))+λ1R(xi ,y i )+λ2D(x i ,y i )+λ3Q(x i ,y i )

[0246] wherein, λ1 is a regularization coefficient, λ2 is a weight coefficient of a data term, λ3 is a weight coefficient of a prior term, R(x i ,y i ) is a regularization term, D(x i ,y i ) is a data term, used for modeling the relationship between a feature map and an attribute, the data term is calculated by a cross-entropy loss method, Q(x i ,y i ) is a penalty term, used for introducing prior knowledge or other constraint conditions;

[0247] The optimized smooth potential energy function is:

[0248]

[0249] wherein, K is the number of additional terms, β k is a weight coefficient, f k (y i , y j ) is a texture similarity function or an edge consistency function.

[0250] In step 102, the optimized feature map is segmented by a segmentation model SAM to generate a full segmentation result of the optimized feature map, and each region in the full segmentation result is classified by the optimized feature map to obtain a final surgical instrument image segmentation result, so that the surgical instrument image segmentation is completed.

[0251] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0252] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0253] In several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Among them, the above-mentioned system embodiments are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division mode, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection between the units or modules through some interfaces, and can be electrical or other forms.

[0254] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0255] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0256] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or all or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a read-only storage medium (ROM, Read-Only Memory), a random access storage medium (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various storage medium that can store program codes.

[0257] Obviously, the above embodiments are only examples for clear illustration, and are not limitations on the embodiments. For ordinary skilled persons in the art, other different forms of changes or variations can be made on the basis of the above description. Here, all the embodiments need not and cannot be exhausted. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. An unsupervised surgical instrument image segmentation method based on SAM, characterized in that: include: Extracting features from a surgical image containing surgical instruments to generate a feature vector of the surgical image, performing dimensionality reduction on the feature vector, performing preliminary classification on the reduced feature vector using a mean clustering method, and generating a feature map after preliminary classification; optimizing the feature map using a conditional random field to generate an optimized feature map; The feature map is optimized by a conditional random field to generate an optimized feature map, which includes: modeling each pixel in the feature map as a node according to the conditional random field, with the node label being the category assigned in the K-means, and finding the optimal configuration of the node label under a given observation value by maximizing the conditional probability, thereby completing the optimization processing of the classification result, specifically: Data potential energy function and smooth potential energy function are introduced, where the data potential energy function is used to represent the relationship between the observation value and label of each pixel; Smooth potential energy represents the relationship between adjacent pixels and is used to constrain the labels of adjacent pixels so that the label changes are relatively small; The data potential energy function and the smooth potential energy function are optimized to make the feature map optimization more accurate, specifically: The optimized data potential energy function is: U(x i ,y i )=-log(ψ(x i ,y i ))+λ1R(x i ,y i )+λ2D(x i ,y i )+λ3Q(x i ,y i ), Among them, x i is the observation value of the i-th pixel, y i is the label of the i-th pixel, is the regularization coefficient, is the weight coefficient of the data item, is the weight coefficient of the prior term, R(x i ,y i ) is the regularization term, D(x i ,y i ) is a data item used to model the relationship between feature maps and attributes. The data item is calculated by the cross entropy loss method. Q(x i ,y i ) is a penalty term used to introduce prior knowledge or other constraints, ψ(x i ,y i ) is the potential relationship between the observation value of the i-th pixel and the label of the i-th pixel; The optimized smooth potential energy function is: ， Among them, y j For the The label of each pixel, p i and p j are the position coordinates of the i-th pixel and the The position coordinates of the pixel points, ||p i -p j || is the Euclidean distance, is the standard deviation of the smoothing term, is the number of additional items, is the weight coefficient, is a texture similarity function or an edge consistency function; The optimized feature map is segmented through the segmentation model SAM to generate a full segmentation result of the optimized feature map, and each area in the full segmentation result is classified through the optimized feature map to obtain the final surgical instrument image segmentation result, thereby completing the surgical instrument image segmentation.

2. The unsupervised surgical instrument image segmentation method based on SAM according to claim 1, characterized in that: Feature extraction of surgical images containing surgical instruments includes: Unify the size of surgical images and perform standardization on the unified surgical images: Specifically: Unified surgical image standardization parameters: ， in, is the standardized mean of each channel of the unified surgical image, is the standard deviation of each channel of the unified surgical image; The DINOv2 model is used to extract features from the standardized surgical images.

3. The unsupervised surgical instrument image segmentation method based on SAM according to claim 1, characterized in that: Performing dimensionality reduction processing on the feature vector includes: The eigenvector is converted into a matrix, where each row represents a channel and each column represents the pixel value of a channel, and the eigenvector matrix is ​​calculated by the following formula: ， in, is the eigenvector The covariance matrix, X mean for The mean vector of is the eigenvector matrix, is the focus matrix, the elements on the diagonal are the eigenvalues, is the number of channels; According to the eigenvalue, the eigenvectors are sorted from large to small, the first three eigenvectors are selected, and the selected eigenvectors are multiplied by the centralized input features to obtain the reduced dimension feature matrix. The formula is as follows: X pca =V s T (X-X mean ), Among them, X pca ∈R 3×20×20 is the 3-channel 20-length 20-width feature vector after dimensionality reduction, V s are the first three eigenvectors selected.

4. The unsupervised surgical instrument image segmentation method based on SAM according to claim 1, characterized in that: The mean clustering method is used to perform preliminary classification of the feature vector after dimensionality reduction, including: randomly selecting two initial cluster centers in three-dimensional space as initial values, calculating the Euclidean distance between each pixel point in the feature vector and each initial cluster center, and assigning it to the cluster corresponding to the initial cluster center with the smallest Euclidean distance; For each cluster, calculate the average value of all pixel data in each cluster and use it as the new cluster center. Assuming C is a cluster containing m vectors, the update formula of the cluster center is: ， Repeat the above steps until the cluster center no longer changes, and the preliminary classification is completed.

5. The unsupervised surgical instrument image segmentation method based on SAM according to claim 1, characterized in that: The feature map is optimized by a conditional random field to generate an optimized feature map, which includes: modeling each pixel in the feature map as a node according to the conditional random field, with the node label being the category assigned in the K-means, and finding the optimal configuration of the node label under a given observation value by maximizing the conditional probability, thereby completing the optimization processing of the classification result, specifically: The data potential energy function and smooth potential energy function are introduced, where the data potential energy function is used to represent the relationship between the observation value and label of each pixel point. The data potential energy function is: U(x i ,y i )=-log(ψ(x i ,y i )), Among them, x i Indicates the The observed value of the pixel, y i Indicates the The label of the pixel, ψ(x i ,y i ) indicates the The observation value of the pixel and the The potential relationship between the labels of pixels; Smooth potential energy represents the relationship between adjacent pixels and is used to constrain the labels of adjacent pixels so that the label changes are relatively small. The smooth potential energy function is: ， where y i and y j Respectively represent The label of the pixel and The labels of pixels, is the weight of the smoothing term, p i and p j Respectively represent The position coordinates of the pixel points and the The position coordinates of the pixel points, ||p i -p j || represents the Euclidean distance, is the standard deviation of the smoothing term; Finally, the energy function of the conditional random field model can be defined as: E(Y|X)=∑U(x i ,and i )+λ∑P(y i ,and j ), The lower the energy function value, the higher the probability of the corresponding label configuration.

6. An unsupervised surgical instrument image segmentation system based on SAM, characterized in that: include: A feature map acquisition module is used to extract features from a surgical image containing surgical instruments, generate a feature vector of the surgical image, perform dimensionality reduction on the feature vector, perform preliminary classification on the reduced feature vector using a mean clustering method, generate a feature map after preliminary classification, and optimize the feature map using a conditional random field to generate an optimized feature map; The feature map is optimized by a conditional random field to generate an optimized feature map, which includes: modeling each pixel in the feature map as a node according to the conditional random field, with the node label being the category assigned in the K-means, and finding the optimal configuration of the node label under a given observation value by maximizing the conditional probability, thereby completing the optimization processing of the classification result, specifically: Data potential energy function and smooth potential energy function are introduced, where the data potential energy function is used to represent the relationship between the observation value and label of each pixel; Smooth potential energy represents the relationship between adjacent pixels and is used to constrain the labels of adjacent pixels so that the label changes are relatively small; The data potential energy function and the smooth potential energy function are optimized to make the feature map optimization more accurate, specifically: The optimized data potential energy function is: ， Among them, x i For the The observed value of the pixel, y i For the The labels of pixels, is the regularization coefficient, is the weight coefficient of the data item, is the weight coefficient of the prior term, R(x i ,y i ) is the regularization term, is a data item used to model the relationship between feature maps and attributes. The data item is calculated by the cross entropy loss method. Q(x i ,y i ) is a penalty term used to introduce prior knowledge or other constraints, ψ(x i ,y i ) is the The observation value of the pixel and the The potential relationship between the labels of pixels; The optimized smooth potential energy function is: ， Among them, y j For the The label of each pixel, p i and p j Respectively The position coordinates of the pixel points and the The position coordinates of the pixel points, ||p i -p j || is the Euclidean distance, is the standard deviation of the smoothing term, K is the number of additional terms, is the weight coefficient, is a texture similarity function or an edge consistency function; The segmentation module is used to segment the optimized feature map through the segmentation model SAM, generate a full segmentation result of the optimized feature map, and classify each area in the full segmentation result through the optimized feature map to obtain the final surgical instrument image segmentation result, thereby completing the surgical instrument image segmentation.

7. The unsupervised surgical instrument image segmentation system based on SAM according to claim 6, characterized in that: Feature extraction of surgical images containing surgical instruments includes: Unify the size of surgical images and perform standardization on the unified surgical images: Specifically: Unified surgical image standardization parameters: ， in, is the standardized mean of each channel of the unified surgical image, is the standard deviation of each channel of the unified surgical image; The DINOv2 model is used to extract features from the standardized surgical images.

8. The unsupervised surgical instrument image segmentation system based on SAM as claimed in claim 6, characterized in that: Performing dimensionality reduction processing on the feature vector includes: The eigenvector is converted into a matrix, where each row represents a channel and each column represents the pixel value of a channel, and the eigenvector matrix is ​​calculated by the following formula: ， in, is the eigenvector The covariance matrix, X mean for The mean vector of is the eigenvector matrix, is the focus matrix, the elements on the diagonal are the eigenvalues, and n is the number of channels; According to the eigenvalue, the eigenvectors are sorted from large to small, the first three eigenvectors are selected, and the selected eigenvectors are multiplied by the centralized input features to obtain the reduced dimension feature matrix. The formula is as follows: X pca =V s T (X-X mean ), Among them, X pca ∈R 3×20×20 is the 3-channel 20-length 20-width feature vector after dimensionality reduction, V s are the first three eigenvectors selected.

9. The unsupervised surgical instrument image segmentation system based on SAM according to claim 6, characterized in that: The mean clustering method is used to perform preliminary classification of the feature vector after dimensionality reduction, including: randomly selecting two initial cluster centers in three-dimensional space as initial values, calculating the Euclidean distance between each pixel point in the feature vector and each initial cluster center, and assigning it to the cluster corresponding to the initial cluster center with the smallest Euclidean distance; For each cluster, calculate the average value of all pixel data in each cluster and use it as the new cluster center. Assuming C is a cluster containing m vectors, the update formula of the cluster center is: ， Repeat the above steps until the cluster center no longer changes, and the preliminary classification is completed.

Images