Surgical instrument, procedure, and tissue intelligence identification method based on video understanding

Abstract

The application provides a surgical instrument, operation and tissue intelligent identification method based on video understanding, and relates to the technical field of minimally invasive surgical behavior identification.The application can simultaneously identify instruments, operations and tissues based on multi-task learning.The extraction of operation and tissue feature maps is guided by instrument feature maps, the identification stability is improved based on the characteristics of high instrument identification accuracy and strong stability.Meanwhile, the priori knowledge of surgery is fully utilized, the correlation between instruments, operations and tissues is explicitly modeled on a semantic level through a graph convolution network, the correlation is further embedded into an image feature extraction network and a semantic feature extraction network through a spatial semantic attention alignment module, finally, the visual recognition result and the correlation recognition result are fused, and the accuracy of identification is improved.Finally, the triple feature is distinguished through metric learning, and the identification result is optimized, and the misidentification probability when multiple triplets appear simultaneously in the surgical endoscope video is significantly reduced.

Description

Technical Field

[0001] This invention relates to the field of minimally invasive surgical behavior recognition technology, specifically to a video understanding-based intelligent recognition method for surgical instruments, operations, and tissues. Background Technology

[0002] Minimally invasive surgery typically involves making small incisions on the patient's body surface and performing the procedure guided by imaging equipment such as endoscopes. Currently, minimally invasive surgery primarily utilizes endoscopes, displaying endoscopic video on a monitor for the doctor to observe. Minimally invasive surgical behavior recognition refers to identifying which instruments are performing which operations on which tissues within the endoscopic video, thereby recognizing fine-grained behaviors during the surgery.

[0003] Existing surgical behavior recognition methods typically identify instruments, procedures, and targets independently, and then fuse the recognition results to obtain the behavior recognition result.

[0004] However, existing technologies fail to consider the relationships between instruments, operations, and targets, resulting in deficiencies in recognition performance. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a video-understanding-based intelligent recognition method for surgical instruments, operations, and tissues, which solves the problem of low recognition performance in existing technologies.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A video-understanding-based intelligent recognition method for surgical instruments, manipulations, and tissues, comprising:

[0010] Obtain video frame feature maps and corresponding semantic features from endoscopic videos;

[0011] The corresponding instrument visual recognition probability, operation visual recognition probability, and tissue visual recognition probability are obtained based on video frame feature maps; and the instrument association recognition probability, operation association recognition probability, and tissue association recognition probability are obtained based on semantic features.

[0012] The device recognition probability, operation recognition probability, and tissue recognition probability are obtained by integrating the visual recognition probabilities and the corresponding associated recognition probabilities; and the triplet label classification probability is obtained based on the device recognition probability, operation recognition probability, and tissue recognition probability.

[0013] Furthermore, the acquisition of video frame feature maps and corresponding semantic features of the endoscopic video includes:

[0014] The endoscopic video is preprocessed to obtain video frames that meet the preset resolution;

[0015] By using video frames as input to the feature extraction network, we obtain video frame feature maps and corresponding semantic features.

[0016] The video frame feature maps are extracted using a convolutional network.

[0017] The semantic features are extracted using a graph convolutional network.

[0018] Furthermore, both the convolutional network and the graph convolutional network contain several sub-networks;

[0019] Furthermore, the feature maps and semantic features output by each sub-network of the convolutional network are input into the spatial semantic attention alignment module, and the feature maps and corresponding semantic features output by the spatial semantic attention alignment module are used as the input to the next sub-network.

[0020] Furthermore, the spatial semantic attention alignment module outputs feature maps and corresponding semantic features, including:

[0021] Membership matrices are obtained through convolutional and fully connected layers. Spatial and channel attention are then obtained by performing SOFTMAX and 2D average pooling AVGPOOL operations on the membership matrices. Finally, spatial attention is fused into the feature map through residual connections, and channel attention is fused into the semantic features.

[0022] Furthermore, the step of obtaining the corresponding instrument visual recognition probability, operation visual recognition probability, and tissue visual recognition probability based on the video frame feature map includes:

[0023] Extract device feature maps from video frame feature maps, and then extract device classification features from them;

[0024] The video frame feature map is stitched together with the instrument feature map, and then the operation feature map and tissue feature map are extracted from them, and then the operation classification feature and tissue classification feature are extracted from them.

[0025] Based on the instrument classification features, operation classification features, and tissue classification features, the corresponding instrument visual recognition probability, operation visual recognition probability, and tissue visual recognition probability are obtained.

[0026] Furthermore, based on semantic features, the probabilities of device association recognition, operation association recognition, and tissue association recognition are obtained, including:

[0027] Obtain instrument semantic features, operational semantic features, and tissue semantic features from semantic features;

[0028] Based on the semantic features of the medical device, the semantic features of the operation, the semantic features of the organization, and the corresponding classification features, the corresponding probability of medical device association recognition, the probability of operation association recognition, and the probability of organization association recognition are obtained.

[0029] Furthermore, the method also includes:

[0030] The device feature map, motion feature map, and tissue feature map are concatenated along the channel dimension. The concatenated feature map is then input into a triplet convolutional network to extract triplet feature maps. Triplet features are obtained through two-dimensional average pooling. The triplet features are then input into a triplet fully connected classifier to obtain triplet adjustment scores. The final triplet recognition probability can be calculated based on the triplet adjustment scores and the probability of triplet labels. Metric learning is used for training.

[0031] Furthermore, the loss functions for instrument, operation, and tissue identification, as well as triplet label identification, all employ binary cross-entropy loss.

[0032] Furthermore, the method for obtaining the co-occurrence frequency matrix of the input graph convolutional network is as follows: arrange the labels in the order of instrument, operation, and organization, obtain the co-occurrence frequency matrix of each label in the training set, and normalize it.

[0033] Furthermore, the instrument feature map, operation feature map, and tissue feature map are all extracted using convolutional networks.

[0034] (III) Beneficial Effects

[0035] This invention provides a video-understanding-based intelligent recognition method for surgical instruments, procedures, and tissues. Compared with existing technologies, it offers the following advantages:

[0036] 1. This invention is based on multi-task learning and can simultaneously identify instruments, operations, and tissues. It utilizes instrument feature maps to guide the extraction of operation and tissue feature maps, and improves the stability of identification based on the high accuracy and stability of instrument recognition.

[0037] 2. This invention makes full use of prior surgical knowledge and explicitly models the relationships between instruments, operations, and tissues at the semantic level through graph convolutional networks. These relationships are further embedded into image feature extraction networks and semantic feature extraction networks through a spatial semantic attention alignment module, ultimately fusing visual recognition results with association recognition results to improve recognition accuracy.

[0038] 3. This invention uses metric learning to distinguish triplet features and optimize the recognition results, which significantly reduces the probability of misidentification when multiple triplets appear simultaneously in surgical endoscopic videos. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of a convolutional graph network with spatial semantic attention alignment according to an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the spatial semantic attention alignment module according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of multi-task feature extraction based on device guidance according to an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram illustrating the optimization of the recognition results in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] This application provides a video-understanding-based intelligent recognition method for surgical instruments, operations, and tissues, which solves the problem of low recognition performance in existing technologies.

[0047] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0048] The existing technical defects are as follows:

[0049] 1. Poor stability in instrument, operation, and tissue identification.

[0050] In endoscopic surgery videos, instruments are prominently displayed, and existing technologies can quickly and reliably identify instrument information. However, procedures and tissues are not prominently displayed, and current technologies struggle to reliably identify them. Furthermore, existing technologies identify instruments, procedures, and tissues independently, without using instruments as guiding information, resulting in poor identification stability.

[0051] 2. Difficulty in coupling analysis of instruments, procedures, and tissues.

[0052] Surgical procedures are standardized and conform to certain clinical rules, which manifest as explicit semantic relationships between surgical instruments, procedures, and tissues. Existing technologies, based on feature association and fusion, only implicitly model these relationships and are affected by the amount of training data and model convergence. They fail to treat these relationships as prior knowledge at the semantic level, leading to limitations in recognition performance. Furthermore, existing solutions mostly follow a process of first extracting image features and then performing semantic classification. The integration of image feature extraction and semantic classification is not high, resulting in insufficient discriminative power of the extracted features for specific semantics. This is particularly true in the recognition of endoscopic video instruments, procedures, and tissues, where intra-class differences are significant while inter-class differences are small, making it even more difficult for the feature extraction process of the aforementioned solutions to capture specific semantics.

[0053] 3. Prone to misidentification in situations involving multiple instruments, multiple operations, and multiple organizations.

[0054] When multiple instruments perform various operations on different tissues simultaneously in an endoscopic surgery video, the large combination space of instruments, operations, and tissues can easily lead to incorrect triplet combinations. Existing technologies do not differentiate between different triplets, which can easily result in incorrect triplet combination identification.

[0055] Regarding the aforementioned defects:

[0056] 1. This invention utilizes instrument information to guide operation and tissue identification, and improves the stability of identification based on the high accuracy and strong stability of instrument identification.

[0057] 2. This invention makes full use of prior surgical knowledge, explicitly models the relationship between instruments, operations, and tissues at the semantic level, and embeds this semantic relationship into the feature extraction network. Finally, it integrates the vision-based recognition results and the relationship-based recognition results to improve the accuracy of recognition.

[0058] 3. Metric learning is used to distinguish triplet features and optimize the recognition results, which significantly reduces the probability of misidentification when multiple triplets appear simultaneously in surgical endoscopic videos.

[0059] Example 1:

[0060] This invention provides a video-understanding-based intelligent recognition method for surgical instruments, operations, and tissues. This method is executed by a computer and includes:

[0061] Obtain video frame feature maps and corresponding semantic features from endoscopic videos;

[0062] The corresponding instrument visual recognition probability, operation visual recognition probability, and tissue visual recognition probability are obtained based on video frame feature maps; and the instrument association recognition probability, operation association recognition probability, and tissue association recognition probability are obtained based on semantic features.

[0063] The device recognition probability, operation recognition probability, and tissue recognition probability are obtained by integrating the visual recognition probabilities and the corresponding associated recognition probabilities; and the triplet label classification probability is obtained based on the device recognition probability, operation recognition probability, and tissue recognition probability.

[0064] The beneficial effects of this embodiment:

[0065] This invention is based on multi-task learning and can simultaneously identify instruments, operations, and tissues. It utilizes instrument feature maps to guide the extraction of operation and tissue feature maps, and improves the stability of identification based on the high accuracy and stability of instrument recognition.

[0066] like Figure 1 As shown below, the specific implementation process of this embodiment will be described in detail:

[0067] S1. Obtain video frame feature maps of the endoscopic video. d and corresponding semantic features l d .

[0068] In this embodiment, endoscopic video refers to video captured by an endoscope, and the video image contains at least one of the three types of targets: instruments, operations, and tissues.

[0069] In practical implementation, the methods for obtaining video frame feature maps and semantic features include the following steps:

[0070] S101, Extracting video frame feature maps m d Previously, the endoscopic video needed to be preprocessed to obtain video frame x to meet the input requirements of the subsequent model. The preprocessing steps specifically included:

[0071] One frame is extracted from each second of the endoscopic video as a video frame, denoted as . Where H is the height of the video frame, W is the width of the video frame, and 3 represents the RGB three channels with pixel values ​​ranging from 0 to 255. The video frame resolution is adjusted to a fixed H′×W′, and then the pixel values ​​are normalized as follows:

[0072]

[0073] mean and std are the mean and variance of pixel values, respectively. They need to be determined comprehensively based on the characteristics of the video and the model type. Typically, mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225].

[0074] S102. Use the preprocessed video frame x as input to the feature extraction network to obtain the video frame feature map m. d and corresponding semantic features l d ;

[0075] In practical implementation, the video frame feature map m d Convolutional network architectures such as AlexNet, VGG, GoogLeNet, ResNet, and DenseNet can be used for extraction; these networks can be decomposed into d sub-network sequences, i.e., CNN0 = [LAYER0, LAYER1, ..., LAYER...]. j ,…,LAYER d-1 Each subnetwork is used to further extract feature maps m. j ′:

[0076]

[0077] Among them, c j For feature map m j The number of channels, h j and w j Feature map m j The height and width of the first layer are the preprocessed frame image, i.e., m0 = x.

[0078] semantic features l d A graph convolutional network containing d sub-network sequences can be used for extraction, i.e., GCN = [GCONV0, GCONV1, ..., GCONV1]. j ,…,GCONV d-1 Each layer can extract semantic features containing relationships. j ′:

[0079]

[0080] Where A′ represents the normalized co-occurrence frequency matrix, and the method for obtaining it is as follows: arrange the labels in the order of instrument, operation, and tissue, obtain the co-occurrence frequency matrix A of each label in the training set, and normalize it, which can be expressed as:

[0081]

[0082]

[0083]

[0084] Among them, a ij This represents the co-occurrence frequency of the i-th label and the j-th label.

[0085] These semantic features represent the vectorized representation of each label. Graph convolutional networks aggregate and transform the information in each semantic feature with the information of its associated semantics, thereby capturing the interrelationships between semantics and learning higher-level semantic features.

[0086] Existing related technical solutions in recognition tasks mostly follow a process of first extracting image features and then performing semantic classification. The integration of image feature extraction and semantic classification processes is not high, resulting in insufficient discriminative power of the extracted features for specific semantics. Especially in the recognition of endoscopic video instruments, operations, and tissues, endoscopic videos exhibit significant intra-class differences but relatively small inter-class differences, making it even more difficult for the feature extraction processes of the aforementioned solutions to capture specific semantics. Therefore, this embodiment adopts a scheme for parallel extraction of image features and semantic features. Furthermore, to enable semantics to be applied to image feature extraction earlier, this technology integrates the image feature extraction and semantic feature extraction processes, thereby enriching the semantic information in the feature map and facilitating recognition in downstream multi-task tasks.

[0087] To leverage the relationship between image and semantic features, a common approach is to use attention to dynamically focus on important features. However, this approach typically requires learning through supervised signals, which is challenging in complex endoscopic video scenarios with difficulties in multi-semantic recognition, leading to insufficient feature fusion. Considering that when end-effectors in endoscopic videos perform surgical operations on tissues, they usually exhibit significant discriminative power within a specific active region, while other regions have lower importance, this embodiment employs an active region mining approach to obtain feature attention. This aligns image and semantic features spatially and uses the alignment results in the form of attention to guide the feature extraction process. To this end, this embodiment designs a Spatial Semantic Attention Alignment (SA) module, which mines salient semantics from feature maps using spatial attention and salient semantic features using channel attention. The Spatial Semantic Attention Alignment module is further integrated into each sub-network of the image convolutional network and the graph convolutional network via residual connections, forming a Spatial Semantic Attention Aligned Convolutional Graph Network (SA-GCNN).

[0088] The feature extraction network in this embodiment can be as follows: Figure 2 The spatial semantic attention aligned convolutional graph network (SA-GCNN) shown here has a spatial semantic attention alignment module SA placed after each subnetwork of the image convolutional network CNN0 and the graph convolutional network GCN, i.e.:

[0089] (mj+1 ,l j+1 ) = SA(LAYER j (m j ),GCONV j (l j )),

[0090] j = 0, 1, ..., d-1

[0091] And such as Figure 3 As shown, in each spatial semantic attention alignment module, the membership matrix aff is obtained through the convolutional layer CONV and the fully connected layer FC. This membership matrix represents the response of each spatial location on the feature map to a specific semantic, and can be represented as follows:

[0092]

[0093] Membership matrices can measure feature importance, but due to their high dimensionality and sparsity, they are difficult to directly incorporate into image and semantic features. Therefore, this embodiment decomposes the membership matrix into spatial and semantic feature importance, specifically by obtaining spatial attention (att) through SOFTMAX and 2D average pooling (AVGPOOL) operations. s and channel attention att c , can be represented as:

[0094]

[0095] in, Let σ(x) represent the matrix of the i-th channel of aff, where σ(x) = 1 / (1+e) -x () represents the sigmoid function, and λ and γ are the scaling factors in spatial attention and channel attention, respectively, which can be greater than 1 depending on the feature map size. Based on the discovery that surgical instruments, manipulations, and tissues share the same active region, this embodiment uses SOFTMAX to highlight spatial locations with high responses in the membership matrix to mine the active region, and uses two-dimensional average pooling and the sigmoid function to calculate the overall response degree of the semantics corresponding to each channel to measure the importance of semantic features. λ and γ are introduced to alleviate the sparsity problem of the membership matrix.

[0096] Then, the attention is fused into the feature map and semantic features through residual connections to obtain the feature map and semantic features output by the spatial semantic attention alignment module, which can be represented as:

[0097] m j+1 =αm j +att s ·m j

[0098] l j+1 =αlj +att c ·l j

[0099] Where α is the fusion coefficient, representing the amount of information retained in the previous layer feature map and semantic features, and can usually be a value between 0.9 and 1.0.

[0100] The aforementioned spatial semantic attention alignment module SA can be represented as:

[0101] (m j+1 ,l j+1 0 = SA(m′) j ,l′ j )

[0102] Finally, the SA-GCNN network, composed of d layers, outputs the video frame feature map m. d and semantic features l d It can be represented as:

[0103] (m d ,l d =SA-GCNN(x,l0,A′)

[0104] Where x represents the preprocessed video frame;

[0105] A set of learnable semantic features representing a GCN network;

[0106] A′ represents the normalized co-occurrence frequency matrix.

[0107] Feature maps m obtained through SA-GCNN network d It adaptively focuses on salient features in the image and has richer semantic information. At the same time, the semantic features also adaptively focus on the categories of instruments, operations and tissues appearing in the image, which helps in recognition in downstream multi-tasks.

[0108] S2, based on video frame feature map m d Obtain the corresponding instrument visual recognition probability Operational visual recognition probability and organizational visual recognition probability And based on semantic features l d Obtain the probability of device association recognition Operation association recognition probability Organizational association identification probability

[0109] In practical implementation, the probability of visual recognition of the device is obtained. Operational visual recognition probability and organizational visual recognition probability like Figure 4 As shown, the specific steps include S201 to S204:

[0110] S201, From video frame feature map m d Extract instrument feature maps from them.

[0111] In practical implementation, a convolutional neural network (CNN) is used. i From image feature map m d The extracted instrument feature map can be represented as:

[0112]

[0113] Where c′ is the feature map m i The number of channels.

[0114] S202, Transfer video frame feature map m d Instrument feature diagram m i The data is then stitched together, and operational and organizational feature maps are extracted from it.

[0115] In practical implementation, in order to use the instrument feature map to guide the operation and tissue feature extraction, the image feature map and the instrument feature map are concatenated in the channel dimension, which can be represented as:

[0116]

[0117] After obtaining the stitched feature map m i+ Then, the operation of the convolutional network (CNN) is utilized. v Organizing Convolutional Networks (CNNs) t Extracting the operational feature map and the tissue feature map separately can be represented as follows:

[0118]

[0119]

[0120] S203. Extract instrument classification features, operation classification features, and tissue classification features from the instrument feature map, operation feature map, and tissue feature map.

[0121] In practical implementation, the instrument classification features f are output through three two-dimensional average pooling AVGPOOLs. i Operational classification features f v and organizational classification characteristics f t , can be represented as:

[0122]

[0123]

[0124]

[0125] S204. Based on the instrument classification features, operation classification features, and tissue classification features, obtain the corresponding instrument visual recognition probability, operation visual recognition probability, and tissue visual recognition probability.

[0126] In practical implementation, three fully connected classifiers are used: the instrument fully connected classifier FC. i Operation of fully connected classifiers (FC) v Fully connected classifier (FC) t Input the instrument classification features f respectively i Operational classification features f v and organizational classification characteristics f t Obtain the probability of visual recognition of the device. Operational visual recognition probability and organizational visual recognition probability It can be represented as:

[0127]

[0128]

[0129]

[0130] Where σ(x)=1 / (1+e -x ) represents the sigmoid function, which maps the result to the interval (0,1) to obtain the recognition probability.

[0131] When training the network, the training set contains several endoscopic images. and corresponding behavior triple tags That is, there are a total of k p A predefined category The image represents the occurrence of triples of type i. Each type represents a triple consisting of instrument, operation, and tissue, such as "grasper, traction, gallbladder". Each frame of the endoscopic video may contain 0, 1, or more such triples. Clearly, the triple labels can be decomposed, let y p = <y i ,y v ,y t >, among which Represents the medical device label. Represents the operation label. Represents organizational tags, k i k v k t These represent the number of predefined instrument categories, operation categories, and tissue categories, respectively.

[0132] Through the above steps, this invention, based on multi-task learning, can simultaneously identify instruments, operations, and tissues. Steps S201–S204 utilize instrument feature maps to guide the extraction of operation and tissue feature maps, improving the stability of the identification process due to the high accuracy and stability of instrument recognition.

[0133] Furthermore, in specific implementation, based on semantic features l d Obtaining the device association identification probability, operation association identification probability, and tissue association identification probability specifically includes steps S205 to S206:

[0134] S205, via the fully connected layer FC l From semantic features d Obtain the semantic features w of the device i Operational semantic features w v Organizational semantic features w t , can be represented as:

[0135]

[0136] Among them, l′ i Represents the i-th semantic feature;

[0137] And let As a semantic feature of medical devices;

[0138] make As an operational semantic feature;

[0139] make As an organizational semantic feature;

[0140] S206, Based on instrument semantic features w i Operational semantic features w v Organizational semantic features w t and the corresponding classification features f i f v and f t Obtain the corresponding device association recognition probability Operation association recognition probability Organizational association identification probability It can be represented as:

[0141]

[0142]

[0143]

[0144] S3. Integrate the visual recognition probabilities and the corresponding associated recognition probabilities to obtain the device recognition probability, operation recognition probability and tissue recognition probability; and obtain the triplet label classification probability based on the device recognition probability, operation recognition probability and tissue recognition probability, specifically including steps S301 to 302.

[0145] S301. Fuse each visual recognition probability with the corresponding associated recognition probability to obtain the instrument recognition probability, operation recognition probability and tissue recognition probability;

[0146]

[0147]

[0148]

[0149] Furthermore, the loss function for instrument, manipulation, and tissue identification employs binary cross-entropy loss, which measures the difference between the identification probability and the true label, as shown in the following formula:

[0150]

[0151]

[0152]

[0153] Among them, L i L v and L t The identification losses are respectively for instruments, operations, and tissues.

[0154] S302. Based on the predefined triplet categories, let the label of the triplet of category a be... The probability distribution transformation of the triplet labels of class a is as follows:

[0155]

[0156] The loss function for triplet label recognition also uses binary cross-entropy loss, with a recognition loss L. p The calculation is as follows:

[0157]

[0158] Thus, the mapping probabilities of various triplet labels can be obtained. This embodiment of the invention integrates visual recognition results and association recognition results, further improving the accuracy of recognition. However, the inventors further discovered that the behavioral triplet label probabilities obtained through the above steps may produce mapping errors. Based on metric learning, triplet features of the same triplet category are closer together, while triplet features of different triplet categories are farther apart, enabling the differentiation of surgical triplet features. Therefore, if... Figure 5 As shown, this embodiment may further include the following steps to solve the problem:

[0159] S401, Transfer the instrument feature map m i Action feature map m v With organizational feature map m t Channel-dimensional splicing can be represented as:

[0160]

[0161] S402, then m ivt Input into a triplet convolutional network (CNN) p Extracting triplet feature maps m p Then, the triplet feature f is obtained through two-dimensional average pooling operation. p ;

[0162] S403, using a triplet fully connected classifier FC p The adjusted scores of the triples are obtained. It can be represented as:

[0163]

[0164] S404. The final triplet recognition probability can be calculated based on the triplet adjustment score and the probability of the triplet label, which can be specifically expressed as:

[0165]

[0166] Where, σ -1 (x) = ln(x / (1-x)) is the inverse function of sigmoid.

[0167] And the loss function based on metric learning is:

[0168]

[0169] The loss function samples the triplet features of the i-th, j-th, and k-th frames of the video, respectively, where f i p and The triples are of the same category, f i p and The categories of triples are different.

[0170] At this point, the optimized triplet label classification probability can be obtained. By employing metric learning to distinguish triplet features and optimizing the recognition results, the false recognition probability when multiple triples appear simultaneously in surgical endoscopy videos is significantly reduced. The final recognition result can then be output as needed, for example:

[0171] The triplet is sorted according to the optimized recognition probability. The triplet pairs are sorted from highest to lowest probability, and the top n pairs with a probability greater than a threshold τ are output as the final result. The value of n is determined based on the maximum number of triplet pairs appearing simultaneously in the surgical video, and is usually no more than 3; the value of τ is determined comprehensively based on factors such as surgical type, number of triplet categories, and task difficulty, and is usually a value between 0.2 and 0.5. The output information includes the surgical triplet pairs of the current frame and the corresponding recognition confidence.

[0172] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0173] 1. This invention is based on multi-task learning and can simultaneously identify instruments, operations, and tissues. It utilizes instrument feature maps to guide the extraction of operation and tissue feature maps, and improves the stability of identification based on the high accuracy and stability of instrument recognition.

[0174] 2. This invention makes full use of prior surgical knowledge and explicitly models the relationships between instruments, operations, and tissues at the semantic level through graph convolutional networks. These relationships are further embedded into image feature extraction networks and semantic feature extraction networks through a spatial semantic attention alignment module, ultimately fusing visual recognition results with association recognition results to improve recognition accuracy.

[0175] 3. This invention uses metric learning to distinguish triplet features and optimize the recognition results, which significantly reduces the probability of misidentification when multiple triplets appear simultaneously in surgical endoscopic videos.

[0176] It should be noted that, through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the prior art, can be embodied in the form of software products. These computer software products can be stored in computer-readable storage media, such as ROM / RAM, magnetic disks, optical disks, etc., and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain portions of the embodiments. In this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0177] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent recognition of surgical instruments, operations, and tissues based on video understanding, characterized in that, The method includes: The video frame feature maps and corresponding semantic features of the endoscopic video are obtained. The video frames are used as input to a feature extraction network to obtain the video frame feature maps and corresponding semantic features. The video frame feature maps are extracted using a convolutional network, and the semantic features are extracted using a graph convolutional network. The corresponding instrument visual recognition probability, operation visual recognition probability, and tissue visual recognition probability are obtained based on video frame feature maps; and the instrument association recognition probability, operation association recognition probability, and tissue association recognition probability are obtained based on semantic features. The device recognition probability, operation recognition probability, and tissue recognition probability are obtained by fusing the various visual recognition probabilities and their corresponding associated recognition probabilities; and the triplet label classification probability is obtained based on the device recognition probability, operation recognition probability, and tissue recognition probability. Both the convolutional network and the graph convolutional network contain several sub-networks; and the feature map output by each sub-network of the convolutional network and the semantic features output by each sub-network of the graph convolutional network are input into the spatial semantic attention alignment module, and the feature map and corresponding semantic features output by the spatial semantic attention alignment module are used as the input of the next sub-network. The instrument feature map, operation feature map, and tissue feature map are concatenated along the channel dimension. The concatenated feature map is then input into a triplet convolutional network to extract triplet feature maps. Triplet features are obtained through two-dimensional average pooling. The triplet features are then input into a triplet fully connected classifier to obtain triplet adjustment scores. The final triplet recognition probability can be calculated based on the triplet adjustment scores and the probability of triplet labels. Metric learning is used for training.

2. The video-understanding-based intelligent recognition method for surgical instruments, operations, and tissues as described in claim 1, characterized in that, The acquisition of video frame feature maps and corresponding semantic features of endoscopic videos includes: The endoscopic video is preprocessed to obtain video frames that meet the preset resolution.

3. The video-understanding-based intelligent recognition method for surgical instruments, operations, and tissues as described in claim 2, characterized in that, The spatial semantic attention alignment module outputs feature maps and corresponding semantic features, including: The membership matrix is ​​obtained through convolutional layers and fully connected layers, and then the membership matrix is ​​processed by... SOFTMAX Operations and 2D average pooling AVGPOOL The operation acquires spatial attention and channel attention; finally, spatial attention is fused into the feature map through residual connections, and channel attention is fused into the semantic features.

4. The video-understanding-based intelligent recognition method for surgical instruments, operations, and tissues as described in claim 1, characterized in that, The process of obtaining the corresponding instrument visual recognition probability, operation visual recognition probability, and tissue visual recognition probability based on video frame feature maps includes: Extract device feature maps from video frame feature maps, and then extract device classification features from them; The video frame feature map is stitched together with the instrument feature map, and then the operation feature map and tissue feature map are extracted from them, and then the operation classification feature and tissue classification feature are extracted from them. Based on the instrument classification features, operation classification features, and tissue classification features, the corresponding instrument visual recognition probability, operation visual recognition probability, and tissue visual recognition probability are obtained.

5. The video-understanding-based intelligent recognition method for surgical instruments, operations, and tissues as described in claim 1, characterized in that, Based on semantic features, the probabilities of device association recognition, operation association recognition, and tissue association recognition are obtained, including: Obtain instrument semantic features, operational semantic features, and tissue semantic features from semantic features; Based on the semantic features of the medical device, the semantic features of the operation, the semantic features of the organization, and the corresponding classification features, the corresponding probability of medical device association recognition, the probability of operation association recognition, and the probability of organization association recognition are obtained.

6. The video-understanding-based intelligent recognition method for surgical instruments, operations, and tissues as described in claim 1, characterized in that, The loss function for instrument, operation, and tissue identification, as well as triplet tag identification, all adopts the binary cross-entropy loss.

7. The video-understanding-based intelligent recognition method for surgical instruments, operations, and tissues as described in claim 2, characterized in that, The method for obtaining the co-occurrence frequency matrix of the input graph convolutional network is as follows: arrange the labels in the order of instrument, operation, and organization, obtain the co-occurrence frequency matrix of each label in the training set, and normalize it.

8. The video-understanding-based intelligent recognition method for surgical instruments, operations, and tissues as described in claim 1, characterized in that, The instrument feature map, operation feature map, and tissue feature map are all extracted using convolutional networks.

Images