A method and device for joint identification of surgical instruments, behaviors and target tissues

Abstract

The present invention relates to the field of medical image processing, and specifically to a method and device for joint identification of surgical instruments, behaviors, and target tissues. The method and device first utilize a category-aligned channel attention mechanism to perform feature category alignment and decoupling on the surgical instruments, behaviors, and target tissue subtasks in the scene; then introduce a long short-term memory network to perform spatiotemporal feature fusion on the action information of the surgical instruments, behaviors, and target tissue subtasks in the scene after feature category alignment and decoupling; and then identify the surgical instruments, behaviors, and target tissue subtasks after spatiotemporal feature fusion through a fully connected layer. The present invention achieves a more comprehensive spatial feature description by extracting local diversity fine-grained features in the surgical scene, and achieves accurate identification of multiple instruments and multiple targets in surgical operations through category decoupling, and comprehensively realizes automatic real-time analysis of the key content of the precise and specific surgical scene.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a method and device for jointly identifying surgical instruments, behaviors and target tissues. Background Art

[0002] The joint identification of surgical instruments, behaviors and target tissues is the key to analyzing surgical scenes. The precise operation of surgical instruments guarantees the safety and effectiveness of surgery. As the most prominent target in the video footage of the guided surgery, accurate instrument identification is the primary task in scene perception and the basis for judging surgical actions and target tissues. Surgical behavior recognition, on the basis of instrument identification, comprehensively considers the target tissues and the movement of surgical instruments involved in the instrument movement to accurately judge the specific surgical operation currently being performed. Workflow recognition is a global perception of the surgical process at the stage level based on specific instruments and surgical behaviors. Through the joint identification of instruments, behaviors and target tissues, surgeons can be provided with sufficient surgical status analysis and surgical decision support during the operation, while realizing the estimation of the remaining time of the operation, providing assistance for the coordination of personnel in and between operating rooms, and effectively improving the safety and efficiency of laparoscopic minimally invasive surgery. After surgery, accurate analysis of surgical video content also greatly facilitates the corresponding surgical records and teaching. Therefore, the proposed high-precision joint identification of surgical instruments, behaviors and target tissues is the basis and key to realizing computer-assisted intervention in minimally invasive surgery.

[0003] In order to solve the problem of surgical scene perception, predecessors have mainly carried out a series of single-task and multi-task joint recognition work on workflow and surgical instruments in laparoscopic cholecystectomy. Early methods used artificially selected features such as intensity, gradient, shape, color and tissue texture based on a single image to realize the recognition of surgical workflow and instruments. Considering the correlation between frames, some scholars used time series models represented by hidden Markov models to process surgical videos over a continuous period of time. With the widespread application of deep learning methods in natural scenes, Twinanda et al. first introduced the deep convolutional network EndoNet for deep visual feature extraction of surgical scenes, while continuing the hidden Markov model to extract inter-frame related information, and used two independent networks to realize the classification and recognition of workflow and surgical instruments respectively. In view of the limitation of EndoNet in processing spatiotemporal features independently, Jin et al. used the characteristics of long short-term memory network as an effective time series model, combined with the end-to-end network constructed by deep convolutional network to extract sufficient spatiotemporal fusion features for the first time to realize workflow recognition. Alshirbaji et al. migrated this method to the instrument recognition task and also achieved recognition accuracy that surpassed previous methods.

[0004] Observing the strong correlation between different tasks in the surgical scene, Jin et al. proposed a multi-task surgical instrument and workflow joint recognition network based on a joint loss function. The instrument recognition and workflow recognition branches share the spatial features on the backbone network, and the long short-term memory network is connected to the workflow task branch to integrate the action information in the time dimension. Finally, the weighted loss function is used to construct a joint loss function for multi-task network training. In order to more enrich and specifically analyze the key content in the surgical scene, Nwoye et al. constructed three key contents of instruments, actions, and target tissues to describe the interaction between instruments and tissues in the surgical scene, and used a 3D mapping interaction space function to realize multi-task joint learning.

[0005] Most of the previous work on workflow and instrument recognition around the problem of surgical scene perception uses a general deep convolutional network for visual feature extraction and uses a fully connected layer to realize the recognition of the corresponding class. These methods achieve a coarse-grained description of the spatial features at the current moment by extracting global features of category fusion. They do not pay attention to the rich fine-grained features of the surgical scene in laparoscopy due to the characteristics of highly similar and overlapping target tissue textures and the existence of only local details such as the tip of the instrument difference. At the same time, they also do not pay attention to the multi-label and multi-target problems when multiple instruments appear simultaneously in the surgical scene. In addition, the existing research on surgical scene perception mainly focuses on the two aspects of workflow and instrument tasks, but lacks recognition tasks that describe surgical actions more specifically. The collaborative recognition method between multiple tasks only uses a simple weighted average of the loss function, which fails to fully utilize the correlation between different surgical tasks. Summary of the invention

[0006] The embodiments of the present invention provide a method and apparatus for joint identification of surgical instruments, behaviors, and target tissues, so as to at least solve the technical problem that the prior art lacks identification tasks for describing surgical actions.

[0007] According to an embodiment of the present invention, a method for joint identification of surgical instruments, behaviors and target tissues is provided, comprising the following steps:

[0008] The category-aligned channel attention mechanism is used to decouple the surgical instruments, behaviors, and target tissue subtasks in the scene by aligning their features into categories.

[0009] A long short-term memory network is introduced to perform spatiotemporal feature fusion on the action information of surgical instruments, behaviors, and target tissue subtasks in the scene after feature category alignment and decoupling.

[0010] The surgical instruments, behaviors, and target tissue subtasks are identified after the spatiotemporal features are fused through a fully connected layer.

[0011] Furthermore, the category-aligned channel attention mechanism is used to align and decouple the surgical instruments, behaviors, and target tissue subtasks in the scene, including:

[0012] The multi-label mutual channel loss based on channel attention is applied to the deep convolutional network to extract spatial features of surgical instruments, behaviors and target tissue sub-tasks in the scene.

[0013] Furthermore, the multi-label mutual channel loss based on channel attention is applied to the deep convolutional network to extract spatial features of surgical instruments, behaviors and target tissue subtasks in the scene, including:

[0014] A deep residual network is used as the backbone module to initially extract deep features, and then a global pooling operation is used to obtain a multi-dimensional feature vector to construct subtask branches;

[0015] Based on the total number of categories of each task, the corresponding global features are divided into category-aligned feature groups.

[0016] Furthermore, a deep residual network is used as the backbone module to initially extract deep features, and then a global pooling operation is used to obtain a multi-dimensional feature vector to construct subtask branches including:

[0017] First, a fifty-layer deep residual network consisting of four residual modules is used as the backbone module to initially extract deep features, and then a global pooling operation is used to obtain a 2048-dimensional feature vector as the output of the backbone module;

[0018] A 1×1 convolution operation is used to transform the extracted 2048-dimensional feature vector to the number of channels suitable for each task branch.

[0019] Furthermore, based on the total number of categories of each task, the corresponding global features are divided into category-aligned feature groups including:

[0020] Laparoscopic cholecystectomy involves 15 types of target tissues. A 1×1 convolution operation is used to obtain a 2040-dimensional global feature F, which is divided into 15 groups of features:

[0021] F={F0,F1,…,F 14};

[0022] Each group F i Contains ξ channels, which are used to extract the diverse local fine-grained features corresponding to the i-th target tissue in the surgical scene;

[0023] The multi-label mutual channel loss consists of a discriminative module and a diversity module, which act on the 15 groups of features F and each group of features F in a single task respectively. i internal;

[0024] For the i-th group of features F i , the discriminative module first generates a 0-1 diagonal matrix M iThe Mask operation in deep learning is performed on the ξ channels in the group, and then the maximum pooling operation across channels is performed on the features in the group after the Mask operation to obtain the final response of the current image to the i-th category. The specific discriminative module is expressed as:

[0025]

[0026] Where W and H represent the width and height of the feature map, F i,j,k Represents the kth element position on the jth channel in the i-th group of features;

[0027] The final response of the input image to each category Dis(F0) to Dis(F n-1 ) is obtained, and the multi-label discriminative loss function is obtained through Softmax operation:

[0028]

[0029]

[0030] where y i represents the true label of the current image for the i-th category, and n represents the total number of categories of this subtask;

[0031] The diversity module is used in each set of features F i An element-by-element Softmax operation is performed internally, and then a cross-channel average pooling operation is performed on each feature map in the group:

[0032]

[0033] After the average response on each graph is calculated, the diversity loss can be calculated as follows:

[0034]

[0035] The complete multi-label mutual channel loss is obtained by weighted sum of the diversity module and the discriminative module:

[0036] L MC (F) = λ1L dis +λ2L div ;

[0037] The corresponding weights are adjusted and set according to the needs of specific tasks.

[0038] Furthermore, the long short-term memory network is introduced to perform spatiotemporal feature fusion on the action information of surgical instruments, behaviors, and target tissue subtasks in the scene after feature category alignment and decoupling, including:

[0039] After the fine-grained visual feature extraction module of each task, a single-layer long short-term memory network is used to extract motion features within a period of input to obtain 512-dimensional spatiotemporal fusion features and finally realize the recognition of the corresponding task through a fully connected layer.

[0040] Furthermore, at the visual feature level, a skip link method is used to achieve cascaded effective visual feature transfer, where the overall loss function of the long short-term memory network is composed of the mutual channel loss of the visual feature level and the standard cross entropy loss weighted by the spatiotemporal fusion features to obtain the classification results.

[0041] According to another embodiment of the present invention, there is provided a device for joint identification of surgical instruments, behaviors and target tissues, comprising:

[0042] A category-aligned fine-grained visual feature extraction module that uses a category-aligned channel attention mechanism to decouple the surgical instrument, behavior, and target tissue subtasks in the scene from feature category alignment;

[0043] The spatiotemporal feature fusion module is used to introduce a long short-term memory network to perform spatiotemporal feature fusion on the action information of surgical instruments, behaviors, and target tissue subtasks in the scene after feature category alignment and decoupling;

[0044] A multi-task cascade module is used to identify surgical instruments, behaviors, and target tissue subtasks after fusion of spatiotemporal features through a fully connected layer.

[0045] A storage medium stores a program file capable of implementing any of the above-mentioned methods for joint identification of surgical instruments, behaviors and target tissues.

[0046] A processor is used to run a program, wherein when the program is run, any of the above-mentioned methods for joint identification of surgical instruments, behaviors and target tissues is executed.

[0047] The method and device for joint identification of surgical instruments, behaviors and target tissues in the embodiment of the present invention first utilizes the category-aligned channel attention mechanism to perform feature category alignment and decoupling of the surgical instruments, behaviors and target tissue subtasks in the scene; then introduces the long short-term memory network to perform spatiotemporal feature fusion of the action information of the surgical instruments, behaviors and target tissue subtasks in the scene after feature category alignment and decoupling; and then identifies the surgical instruments, behaviors and target tissue subtasks after spatiotemporal feature fusion through the fully connected layer. The present invention achieves a more comprehensive spatial feature description by extracting fine-grained features of local diversity in the surgical scene, and achieves accurate identification of multiple instruments and multiple targets in surgical operations through category decoupling, and comprehensively realizes automatic real-time analysis of the key content of the precise and specific surgical scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0049] Figure 1 A flowchart of the method for joint identification of surgical instruments, behaviors and target tissues of the present invention;

[0050] Figure 2 A multi-task learning framework diagram of the method for joint identification of surgical instruments, behaviors and target tissues of the present invention;

[0051] Figure 3 A schematic diagram showing the working principle of the diversity loss module of the method for joint identification of surgical instruments, behaviors and target tissues of the present invention;

[0052] Figure 4 Module diagram of the apparatus for joint identification of surgical instruments, behaviors and target tissues of the present invention. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] Surgical scene perception is an important task for modern intelligent operating rooms to develop information integration and intelligence under the conditions of sophisticated hardware equipment and rich real-time sensor signals. In endoscope-guided computer-assisted minimally invasive surgery, by understanding and processing key information in the current surgical field of view, the surgical scene perception system can monitor the entire surgical process in real time and provide specific auxiliary information to surgeons at any time. In minimally invasive surgery represented by laparoscopic cholecystectomy, the tiny incisions on the body surface reduce the burden of surgery on patients, but the limitations of the endoscopic imaging field of view have caused certain difficulties in guiding the operation of the surgery. Specifically, the viewing range of the endoscope lens limits the doctor's surgical field of view, the smoke in the cavity and the reflection of the screen also block the doctor's field of view, and the high similarity and overlap of the texture of the target tissue under the limited viewing angle also make it difficult for the doctor to judge the current surgical environment, making the surgical risks difficult to predict. Therefore, in order to improve surgical safety while retaining the advantages of minimally invasive surgery, the key technologies in the development of modern operating room scene perception systems are to identify and analyze the key contents of the surgical scene based on the real-time surgical video signals obtained by the intraoperative endoscope, and to provide surgeons with real-time surgical monitoring and scene analysis to provide auxiliary intervention.

[0056] Example 1

[0057] According to an embodiment of the present invention, a method for joint identification of surgical instruments, behaviors and target tissues is provided. Figure 1 , including the following steps:

[0058] S100: Use the category-aligned channel attention mechanism to align and decouple the surgical instruments, behaviors, and target tissue subtasks in the scene;

[0059] S200: Introducing a long short-term memory network to perform spatiotemporal feature fusion on the action information of surgical instruments, behaviors, and target tissue subtasks in the scene after feature category alignment and decoupling;

[0060] S300: Identify surgical instruments, behaviors, and target tissue subtasks after fusion of spatiotemporal features through a fully connected layer.

[0061] The method for joint identification of surgical instruments, behaviors and target tissues in the embodiment of the present invention first uses the category-aligned channel attention mechanism to align and decouple the surgical instruments, behaviors and target tissue subtasks in the scene; then introduces a long short-term memory network to perform spatiotemporal feature fusion on the action information of the surgical instruments, behaviors and target tissue subtasks in the scene after feature category alignment and decoupling; and then identifies the surgical instruments, behaviors and target tissue subtasks after spatiotemporal feature fusion through a fully connected layer. The present invention achieves a more comprehensive spatial feature description by extracting local diversity fine-grained features in the surgical scene, and achieves accurate identification of multiple instruments and multiple targets in surgical operations through category decoupling, and comprehensively realizes automatic real-time analysis of the key content of the precise and specific surgical scene.

[0062] Among them, the channel attention mechanism of category alignment is used to decouple the surgical instruments, behaviors, and target tissue subtasks in the scene by aligning the feature categories, including:

[0063] The multi-label mutual channel loss based on channel attention is applied to the deep convolutional network to extract spatial features of surgical instruments, behaviors and target tissue sub-tasks in the scene.

[0064] Among them, the spatial features extracted from the surgical instruments, behaviors, and target tissue subtasks in the scene using multi-label mutual channel loss based on channel attention on the deep convolutional network include:

[0065] A deep residual network is used as the backbone module to initially extract deep features, and then a global pooling operation is used to obtain a multi-dimensional feature vector to construct subtask branches;

[0066] Based on the total number of categories of each task, the corresponding global features are divided into category-aligned feature groups.

[0067] Among them, the deep residual network is used as the backbone module to initially extract deep features, and then the global pooling operation is used to obtain multi-dimensional feature vectors to construct subtask branches including:

[0068] First, a fifty-layer deep residual network consisting of four residual modules is used as the backbone module to initially extract deep features, and then a global pooling operation is used to obtain a 2048-dimensional feature vector as the output of the backbone module;

[0069] A 1×1 convolution operation is used to transform the extracted 2048-dimensional feature vector to the number of channels suitable for each task branch.

[0070] Among them, based on the total number of categories of each task, the corresponding global features are divided into category-aligned feature groups including:

[0071] Laparoscopic cholecystectomy involves 15 types of target tissues. A 1×1 convolution operation is used to obtain a 2040-dimensional global feature F, which is divided into 15 groups of features:

[0072] F={F0,F1,…,F 14};

[0073] Each group F i Contains ξ channels, which are used to extract the diverse local fine-grained features corresponding to the i-th target tissue in the surgical scene;

[0074] The multi-label mutual channel loss consists of a discriminative module and a diversity module, which act on the 15 groups of features F and each group of features F in a single task respectively. i internal;

[0075] For the i-th group of features F i , the discriminative module first generates a 0-1 diagonal matrix M i The Mask operation in deep learning is performed on the ξ channels in the group, and then the maximum pooling operation across channels is performed on the features in the group after the Mask operation to obtain the final response of the current image to the i-th category. The specific discriminative module is expressed as:

[0076]

[0077] Where W and H represent the width and height of the feature map, F i,j,k Represents the kth element position on the jth channel in the i-th group of features;

[0078] The final response of the input image to each category Dis(F0) to Dis(F n-1 ) is obtained, and the multi-label discriminative loss function is obtained through Softmax operation:

[0079]

[0080]

[0081] where y i represents the true label of the current image for the i-th category, and n represents the total number of categories of this subtask;

[0082] The diversity module is used in each set of features F i An element-by-element Softmax operation is performed internally, and then a cross-channel average pooling operation is performed on each feature map in the group:

[0083]

[0084] When the average response on each graph is calculated, the diversity loss can be calculated as follows:

[0085]

[0086] The complete multi-label mutual channel loss is obtained by weighted sum of the diversity module and the discriminative module:

[0087] L MC (F) = λ1L dis +λ2L div ;

[0088] The corresponding weights are adjusted and set according to the needs of specific tasks.

[0089] Among them, the long short-term memory network is introduced to perform spatiotemporal feature fusion on the action information of surgical instruments, behaviors, and target tissue subtasks in the scene after feature category alignment and decoupling, including:

[0090] After the fine-grained visual feature extraction module of each task, a single-layer long short-term memory network is used to extract motion features within a period of input to obtain 512-dimensional spatiotemporal fusion features and finally realize the recognition of the corresponding task through a fully connected layer.

[0091] Among them, the skip link method is used at the visual feature level to realize the cascaded effective visual feature transmission, and the overall loss function of the long short-term memory network is composed of the mutual channel loss of the visual feature level and the standard cross entropy loss weighted by the spatiotemporal fusion features to obtain the classification results.

[0092] The following is a detailed description of the method for joint identification of surgical instruments, behaviors and target tissues of the present invention with reference to specific embodiments:

[0093] The joint recognition of surgical instruments, behaviors, and target tissues is a key technology in computer-assisted minimally invasive surgery. However, under the limited viewing angle of the laparoscope, fine-grained characteristics such as the texture similarity of the target tissue, the similar structure of the instrument tip, and repeated non-specific behavioral movements during the operation stage all make it difficult to accurately identify these key surgical contents. In view of the existing shortcomings of previous methods, the purpose of the present invention is to provide a more accurate and specific surgical scene analysis method by utilizing the joint recognition of surgical instruments, target tissues, and execution action subtasks. By extracting the local diversity of fine-grained features in the surgical scene, a more sufficient spatial feature description is achieved, and accurate recognition of multiple instruments and multiple targets in surgical operations is achieved through category decoupling.

[0094] The present invention proposes a method for joint recognition of surgical instruments, behaviors, and target tissues based on multi-label mutual channel loss, which is mainly used for scene and action recognition during and after computer-assisted minimally invasive surgery represented by laparoscopic cholecystectomy, and is committed to solving the problem of solving the key global and local fused visual features and action features under long-term dependencies in surgical videos through fine-grained classification and multi-task learning models. The present invention uses a category-aligned channel attention mechanism to achieve visual feature decoupling, introduces a long short-term memory network to extract temporal features of action information in the scene, and realizes multi-task joint recognition in a cascade manner. Experimental verification shows that the present invention has achieved good recognition results that are superior to previous methods in both single tasks and joint tasks.

[0095] In the fine-grained visual feature extraction module with category alignment shared by multiple task branches, the present invention applies a multi-label mutual channel loss function to achieve the decoupling of feature category alignment on the channel, so as to achieve the purpose of focusing on multiple local details in the surgical scene. In the three sub-task branches, the long short-term memory network module is connected to the tasks involving actions in continuous time to achieve the purpose of integrating spatiotemporal features. Finally, in multi-task joint recognition, the long short-term memory network module uses the cascade transmission of spatial features to strengthen the relationship between the presentation of the instrument and the interaction of tissue behavior, and comprehensively realizes the automatic real-time analysis of the key content of the precise and specific surgical scene.

[0096] The multi-task learning framework of the present invention is as follows Figure 2 As shown in the figure, the main structures include three parts: category-aligned fine-grained visual feature extraction module, spatiotemporal feature fusion module and multi-task cascade module.

[0097] 1. Category-aligned fine-grained visual feature extraction module

[0098] For the recognition of subtasks such as instruments, behaviors, and target tissues, the general method is to use deep convolutional networks to extract global visual features of category fusion. In order to fully analyze the local detailed visual features in the surgical scene to achieve accurate recognition in multi-label and multi-entity situations, this module introduces a multi-label mutual channel loss based on channel attention to act on the spatial features extracted by the deep convolutional network.

[0099] Specifically, since multiple subtask branches share some visual features in the surgical scene, a fifty-layer deep residual network consisting of four residual modules is first used as the backbone module to initially extract deep features, and then a global pooling operation is used to obtain a 2048-dimensional feature vector as the output of the backbone module. In order to facilitate the application and calculation of multi-label mutual channel loss in different task branches, a 1×1 convolution operation is used to transform the extracted 2048-dimensional feature vector to the number of channels suitable for each task branch. In laparoscopic cholecystectomy, there are 6 types of surgical instruments, 10 types of actions, and 15 types of target tissues. Therefore, for the three task branches, 1×1 convolution is used to transform them into global visual features of 2040, 2000, and 2040 channels, respectively, and each group contains local features of 340, 200, and 136 channels.

[0100] 2. The working mode and composition principle of multi-label mutual channel loss

[0101] After constructing the subtask branches, the corresponding global features can be divided into category-aligned feature groups based on the total number of categories of each task. Taking the target tissue recognition task branch as an example, laparoscopic cholecystectomy involves 15 categories of target tissues. The 2040-dimensional global feature F is obtained by using a 1×1 convolution operation, so it is divided into 15 groups of features:

[0102] F={F0,F1,…,F 14};

[0103] Each group F i It contains ξ channels, which are used to extract the diverse local fine-grained features corresponding to the i-th target tissue in the surgical scene. The multi-label mutual channel loss consists of a discriminative module and a diversity module, which act on the 15 groups of features F and each group of features F in a single task. i internal.

[0104] The discriminative module is used to guide different groups of features to learn the features related to the corresponding categories and distinguish them. i , the discriminative module first generates a 0-1 diagonal matrix M i Mask operation is performed on ξ channels in the group, and then the maximum pooling operation across channels is performed on the masked group features, so as to retain the maximum response of each position in the feature map to the category, and the final response of the current image to the i-th category is obtained through global average pooling. The specific discriminative module is expressed as:

[0105]

[0106] Where W and H represent the width and height of the feature map, F i,j,k Represents the kth element position on the jth channel in the i-th set of features.

[0107] The final response of the input image to each category Dis(F0) to Dis(F n-1 ) is obtained, and the multi-label discriminative loss function can be obtained through Softmax operation:

[0108]

[0109]

[0110] where y i Represents the true label of the current image for the i-th category, and n represents the total number of categories in this subtask.

[0111] The diversity module is used in each set of features F i An element-by-element Softmax operation is performed internally, and then a cross-channel average pooling operation is performed on each feature map in the group:

[0112]

[0113] After the average response on each graph is calculated, the diversity loss can be calculated as follows:

[0114]

[0115] It is worth noting that in multi-label multi-entity surgical scenarios where multiple instruments appear simultaneously, the working principle of the diversity loss module is as follows Figure 3 shown. Figure 3 In the figure, the left side is the schematic diagram of the multi-label diversity module, and the right side is the schematic diagram of the single-label diversity module.

[0116] The complete multi-label mutual channel loss is obtained by weighted sum of the diversity module and the discriminative module:

[0117] L MC (F) = λ1L dis +λ2L div ;

[0118] The corresponding weights are adjusted and set according to the needs of specific tasks.

[0119] 3. Spatiotemporal feature fusion, multi-task cascade and overall loss function

[0120] In order to capture the motion information contained between consecutive frames, a single-layer long short-term memory network is used to extract motion features within a period of input after the fine-grained visual feature extraction module of each task, and a 512-dimensional spatiotemporal fusion feature is obtained, and finally the corresponding task recognition is realized through a fully connected layer. In addition, considering that surgical instruments are the most significant features and prerequisites for surgical actions, and the surgical operation rules of instruments and actions acting on target tissues, a skip link method is used at the visual feature level to achieve cascaded effective visual feature transmission. The overall loss function of the long short-term memory network consists of the mutual channel loss at the visual feature level and the standard cross entropy loss weighted by the spatiotemporal fusion features to obtain the classification results.

[0121] The innovative technical points of the method of the present invention are at least:

[0122] 1. Improved design of diversity module in the mutual channel loss function in multi-label and multi-entity situations;

[0123] 2. Improved design of the discriminative module in the mutual channel loss function in the case of multi-label and multi-entity;

[0124] 3. Cascaded spatial feature transfer structure for joint identification of surgical instruments, behaviors and target tissues;

[0125] 4. Design an improved multi-label mutual channel loss for extracting fine-grained spatial features in laparoscopic scenarios.

[0126] The beneficial effects of the method of the present invention are at least:

[0127] The multi-label mutual channel loss function with improved design of the present invention can fully extract local features distributed in different areas of the field of view in laparoscopic surgery scenarios; the loss design in the multi-label case can better cope with the application situation in which multiple instruments and multiple entities perform surgical operations simultaneously in the surgical scenario; the decoupling mechanism of category alignment increases the visibility and interpretability of the model; the cascaded joint recognition of instruments, behaviors and target organizations better utilizes the correlation between multiple tasks, improves the recognition accuracy of single tasks and multiple tasks, and thus provides more specific and accurate instructions for real-time assistance in surgery.

[0128] The multi-task learning method for joint identification of surgical instruments, behaviors and target tissues based on multi-label mutual channel loss proposed in the present invention has been experimented on the public datasets CholecT40 and HeiCholec, and has achieved effective improvement over the above-mentioned predecessors in both single tasks and multi-task combination. Verification on multiple datasets also shows the robustness of the model, which can meet the needs of auxiliary analysis of instruments, behaviors and target tissues in laparoscopic surgery scenarios. Experimental verification shows that the multi-label mutual channel loss function proposed in the present invention can effectively realize the decoupling and category alignment of local fine-grained features of the image, and the long short-term memory network module effectively extracts the action information contained in the continuous time based on the decoupled feature sequence. The cascaded multi-task joint recognition structure makes full use of the prior relationship between instruments, behaviors and target tissues, so that the joint recognition network proposed in the present invention is significantly improved compared with the existing methods.

[0129] Example 2

[0130] According to another embodiment of the present invention, a device for joint identification of surgical instruments, behaviors and target tissues is provided. Figure 4 ,include:

[0131] A category-aligned fine-grained visual feature extraction module 100, for utilizing a category-aligned channel attention mechanism to perform feature category alignment and decoupling on surgical instruments, behaviors, and target tissue subtasks in a scene;

[0132] The spatiotemporal feature fusion module 200 is used to introduce a long short-term memory network to perform spatiotemporal feature fusion on the action information of surgical instruments, behaviors and target tissue subtasks in the scene after feature category alignment and decoupling;

[0133] The multi-task cascade module 300 is used to identify the surgical instrument, behavior and target tissue sub-tasks after the spatiotemporal feature fusion through the fully connected layer.

[0134] The device for joint identification of surgical instruments, behaviors and target tissues in the embodiment of the present invention first uses the channel attention mechanism of category alignment to decouple the surgical instruments, behaviors and target tissue subtasks in the scene by aligning the feature categories; then introduces the long short-term memory network to perform spatiotemporal feature fusion on the action information of the surgical instruments, behaviors and target tissue subtasks in the scene after the feature category alignment decoupling; and then identifies the surgical instruments, behaviors and target tissue subtasks after the spatiotemporal feature fusion through the fully connected layer. The present invention achieves a more comprehensive spatial feature description by extracting the local diversity fine-grained features in the surgical scene, and achieves accurate identification of multiple instruments and multiple targets in surgical operations through category decoupling, and comprehensively realizes the automatic real-time analysis of the key content of the precise and specific surgical scene.

[0135] The following is a detailed description of the surgical instrument, behavior and target tissue joint identification device of the present invention with specific embodiments:

[0136] The joint recognition of surgical instruments, behaviors, and target tissues is a key technology in computer-assisted minimally invasive surgery. However, under the limited viewing angle of the laparoscope, fine-grained characteristics such as the texture similarity of the target tissue, the similar structure of the instrument tip, and repeated non-specific behavioral movements during the operation stage all make it difficult to accurately identify these key surgical contents. In view of the existing shortcomings of previous methods, the purpose of the present invention is to provide a more accurate and specific surgical scene analysis device by utilizing the joint recognition of surgical instruments, target tissues, and execution action subtasks. By extracting the local diversity of fine-grained features in the surgical scene, a more sufficient spatial feature description is achieved, and accurate recognition of multiple instruments and multiple targets in surgical operations is achieved through category decoupling.

[0137] The present invention proposes a device for joint identification of surgical instruments, behaviors and target tissues based on multi-label mutual channel loss, which is mainly used for scene and action recognition during and after computer-assisted minimally invasive surgery represented by laparoscopic cholecystectomy, and is committed to solving the problem of solving the key global and local fused visual features and action features under long-term dependencies in surgical videos through fine-grained classification and multi-task learning models. The present invention uses a category-aligned channel attention mechanism to achieve visual feature decoupling, introduces a long short-term memory network to extract temporal features of action information in the scene, and realizes multi-task joint recognition in a cascade manner. Experimental verification shows that the present invention has achieved good recognition results that are superior to previous methods in both single tasks and joint tasks.

[0138] In the fine-grained visual feature extraction module 100 with category alignment shared by multiple task branches, the present invention applies a multi-label mutual channel loss function to achieve feature category alignment decoupling on the channel, so as to achieve the purpose of focusing on multiple local details in the surgical scene. In the three sub-task branches, the long short-term memory network module is connected to the tasks involving actions in continuous time to achieve the purpose of integrating spatiotemporal features. Finally, in multi-task joint recognition, the long short-term memory network module uses the cascade transmission of spatial features to strengthen the relationship between the presentation of the instrument and the interaction of tissue behavior, and comprehensively realizes the automatic real-time analysis of the key content of the precise and specific surgical scene.

[0139] The multi-task learning framework of the present invention is as follows Figure 2 As shown, the structure mainly includes three parts: a category-aligned fine-grained visual feature extraction module 100, a spatiotemporal feature fusion module 200, and a multi-task cascade module 300.

[0140] 1. Category-aligned fine-grained visual feature extraction module 100

[0141] For the recognition of subtasks such as instruments, behaviors, and target tissues, the general method is to use deep convolutional networks to extract global visual features of category fusion. In order to fully analyze the local detailed visual features in the surgical scene to achieve accurate recognition in multi-label and multi-entity situations, this module introduces a multi-label mutual channel loss based on channel attention to act on the spatial features extracted by the deep convolutional network.

[0142] Specifically, since multiple subtask branches share some visual features in the surgical scene, a fifty-layer deep residual network consisting of four residual modules is first used as the backbone module to initially extract deep features, and then a global pooling operation is used to obtain a 2048-dimensional feature vector as the output of the backbone module. In order to facilitate the application and calculation of multi-label mutual channel loss in different task branches, a 1×1 convolution operation is used to transform the extracted 2048-dimensional feature vector to the number of channels suitable for each task branch. In laparoscopic cholecystectomy, there are 6 types of surgical instruments, 10 types of actions, and 15 types of target tissues. Therefore, for the three task branches, 1×1 convolution is used to transform them into global visual features of 2040, 2000, and 2040 channels, respectively, and each group contains local features of 340, 200, and 136 channels.

[0143] 2. The working mode and composition principle of multi-label mutual channel loss

[0144] After constructing the subtask branches, the corresponding global features can be divided into category-aligned feature groups based on the total number of categories of each task. Taking the target tissue recognition task branch as an example, laparoscopic cholecystectomy involves 15 categories of target tissues. The 2040-dimensional global feature F is obtained by using a 1×1 convolution operation, so it is divided into 15 groups of features:

[0145] F={F0,F1,…,F 14};

[0146] Each group F i It contains ξ channels, which are used to extract the diverse local fine-grained features corresponding to the i-th target tissue in the surgical scene. The multi-label mutual channel loss consists of a discriminative module and a diversity module, which act on the 15 groups of features F and each group of features F in a single task. i internal.

[0147] The discriminative module is used to guide different groups of features to learn the features related to the corresponding categories and distinguish them. i , the discriminative module first generates a 0-1 diagonal matrix M iMask operation is performed on ξ channels in the group, and then the maximum pooling operation across channels is performed on the masked group features, so as to retain the maximum response of each position in the feature map to the category, and the final response of the current image to the i-th category is obtained through global average pooling. The specific discriminative module is expressed as:

[0148]

[0149] Where W and H represent the width and height of the feature map, F i,j,k Represents the kth element position on the jth channel in the i-th set of features.

[0150] The final response of the input image to each category Dis(F0) to Dis(F n-1 ) is obtained, and the multi-label discriminative loss function can be obtained through Softmax operation:

[0151]

[0152]

[0153] where y i represents the true label of the current image for the i-th category, and n represents the total number of categories of this subtask.

[0154] The diversity module is used in each set of features F i An element-by-element Softmax operation is performed internally, and then a cross-channel average pooling operation is performed on each feature map in the group:

[0155]

[0156] When the average response on each graph is calculated, the diversity loss can be calculated as follows:

[0157]

[0158] It is worth noting that in multi-label multi-entity surgical scenarios where multiple instruments appear simultaneously, the working principle of the diversity loss module is as follows Figure 3 shown. Figure 3 In the figure, the left side is the schematic diagram of the multi-label diversity module, and the right side is the schematic diagram of the single-label diversity module.

[0159] The complete multi-label mutual channel loss is obtained by weighted sum of the diversity module and the discriminative module:

[0160] L MC (F) = λ1L dis +λ2L div ;

[0161] The corresponding weights are adjusted and set according to the needs of specific tasks.

[0162] 3. Spatiotemporal feature fusion, multi-task cascade and overall loss function

[0163] In order to capture the motion information contained between consecutive frames, a single-layer long short-term memory network is used to extract motion features within a period of input after the fine-grained visual feature extraction module of each task, and a 512-dimensional spatiotemporal fusion feature is obtained, and finally the corresponding task recognition is realized through a fully connected layer. In addition, considering that surgical instruments are the most significant features and prerequisites for surgical actions, and the surgical operation rules of instruments and actions acting on target tissues, a skip link method is used at the visual feature level to achieve cascaded effective visual feature transmission. The overall loss function of the long short-term memory network consists of the mutual channel loss at the visual feature level and the standard cross entropy loss weighted by the spatiotemporal fusion features to obtain the classification results.

[0164] The innovative technical points of the device of the present invention are at least:

[0165] 1. Improved design of diversity module in the mutual channel loss function in multi-label and multi-entity situations;

[0166] 2. Improved design of the discriminative module in the mutual channel loss function in the case of multi-label and multi-entity;

[0167] 3. Cascaded spatial feature transfer structure for joint identification of surgical instruments, behaviors and target tissues;

[0168] 4. Design an improved multi-label mutual channel loss for extracting fine-grained spatial features in laparoscopic scenarios.

[0169] The beneficial effects of the device of the present invention are at least:

[0170] The multi-label mutual channel loss function with improved design of the present invention can fully extract local features distributed in different areas of the field of view in laparoscopic surgery scenarios; the loss design in the multi-label case can better cope with the application situation in which multiple instruments and multiple entities perform surgical operations simultaneously in the surgical scenario; the decoupling mechanism of category alignment increases the visibility and interpretability of the model; the cascaded joint recognition of instruments, behaviors and target organizations better utilizes the correlation between multiple tasks, improves the recognition accuracy of single tasks and multiple tasks, and thus provides more specific and accurate instructions for real-time assistance in surgery.

[0171] The multi-task learning device for joint identification of surgical instruments, behaviors and target tissues based on multi-label mutual channel loss proposed in the present invention has been experimented on the public datasets CholecT40 and HeiCholec, and has achieved effective improvement over the above-mentioned predecessors in both single tasks and multi-task combination. Verification on multiple datasets also shows the robustness of the model, which can meet the needs of auxiliary analysis of instruments, behaviors and target tissues in laparoscopic surgery scenarios. Experimental verification shows that the multi-label mutual channel loss function proposed in the present invention can effectively realize the decoupling and category alignment of local fine-grained features of the image, and the long short-term memory network module effectively extracts the action information contained in the continuous time based on the decoupled feature sequence. The cascaded multi-task joint recognition structure makes full use of the prior relationship between instruments, behaviors and target tissues, so that the joint recognition network proposed in the present invention is significantly improved compared with the existing methods.

[0172] Example 3

[0173] A storage medium stores a program file capable of implementing any of the above-mentioned methods for joint identification of surgical instruments, behaviors and target tissues.

[0174] Example 4

[0175] A processor is used to run a program, wherein when the program is run, any of the above-mentioned methods for joint identification of surgical instruments, behaviors and target tissues is executed.

[0176] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0177] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the system embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0180] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0182] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for joint identification of surgical instruments, behaviors and target tissues, characterized in that: The following steps are involved: The category-aligned channel attention mechanism is used to decouple the surgical instruments, behaviors, and target tissue subtasks in the scene by aligning their features into categories. Specifically include: The multi-label mutual channel loss based on channel attention is used to act on the deep convolutional network to extract spatial features of surgical instruments, behaviors and target tissue subtasks in the scene; the deep residual network is used as the backbone module to initially extract deep features, and then the multi-dimensional feature vector is obtained by global pooling operation to construct subtask branches; based on the total number of categories of each task, the corresponding global features are divided into category-aligned feature groups; The long short-term memory network is introduced to perform spatiotemporal feature fusion on the action information of surgical instruments, behaviors, and target tissue subtasks in the scene after feature category alignment and decoupling; specifically, it includes: After the fine-grained visual feature extraction module of each task, a single-layer long short-term memory network is used to extract motion features within a period of time input to obtain 512-dimensional spatiotemporal fusion features; The surgical instruments, behaviors, and target tissue subtasks are identified after the spatiotemporal features are fused through a fully connected layer.

2. The method for joint identification of surgical instruments, behaviors and target tissues according to claim 1, characterized in that: The deep residual network is used as the backbone module to initially extract deep features, and then a global pooling operation is used to obtain a multi-dimensional feature vector to construct subtask branches, including: First, a fifty-layer deep residual network consisting of four residual modules is used as the backbone module to initially extract deep features, and then a global pooling operation is used to obtain a 2048-dimensional feature vector as the output of the backbone module; A 1×1 convolution operation is used to transform the extracted 2048-dimensional feature vector to the number of channels suitable for each task branch.

3. The method for joint identification of surgical instruments, behaviors and target tissues according to claim 1, characterized in that: The method of dividing the corresponding global features into category-aligned feature groups based on the total number of categories of each task includes: Laparoscopic cholecystectomy involves 15 types of target tissues. A 1×1 convolution operation is used to obtain a 2040-dimensional global feature F, which is divided into 15 groups of features: F={F0,F1,…,F 14 }; Each group F i Contains ξ channels, which are used to extract the diverse local fine-grained features corresponding to the i-th target tissue in the surgical scene; The multi-label mutual channel loss consists of a discriminative module and a diversity module, which act on the 15 groups of features F and each group of features F in a single task respectively. i internal; For the i-th group of features F i , the discriminative module first generates a 0-1 diagonal matrix M i The Mask operation in deep learning is performed on the ξ channels in the group, and then the maximum pooling operation across channels is performed on the features in the group after the Mask operation to obtain the final response of the current image to the i-th category. The specific discriminative module is expressed as: Where W and H represent the width and height of the feature map, F i,j,k Represents the kth element position on the jth channel in the i-th group of features; The final response of the input image to each category Dis(F0) to Dis(F n-1 ) is obtained, and the multi-label discriminative loss function is obtained through Softmax operation: Where yi represents the true label of the current image for the i-th category, and n represents the total number of categories of this subtask; The diversity module is used in each set of features F i An element-by-element Softmax operation is performed internally, and then a cross-channel average pooling operation is performed on each feature map in the group: When the average response on each graph is calculated, the diversity loss can be calculated as follows: The complete multi-label mutual channel loss is obtained by weighted sum of the diversity module and the discriminative module: L MC (F)=λ1L dis +λ2L div ; The corresponding weights are adjusted and set according to the needs of specific tasks.

4. The method for joint identification of surgical instruments, behaviors and target tissues according to claim 1, characterized in that: At the visual feature level, the skip link method is used to achieve cascaded effective visual feature transmission, where the overall loss function of the long short-term memory network is composed of the mutual channel loss of the visual feature level and the standard cross entropy loss weighted by the spatiotemporal fusion features to obtain the classification results.

5. The method for joint identification of surgical instruments, behaviors and target tissues according to any one of claims 1 to 4, characterized in that: Also included is a device for joint identification of surgical instruments, behaviors and target tissues, the device comprising: A category-aligned fine-grained visual feature extraction module that uses a category-aligned channel attention mechanism to decouple the surgical instrument, behavior, and target tissue subtasks in the scene from feature category alignment; The spatiotemporal feature fusion module is used to introduce a long short-term memory network to perform spatiotemporal feature fusion on the action information of surgical instruments, behaviors, and target tissue subtasks in the scene after feature category alignment and decoupling; A multi-task cascade module is used to identify surgical instruments, behaviors, and target tissue subtasks after fusion of spatiotemporal features through a fully connected layer.

6. A storage medium, characterized in that: The storage medium stores a program file capable of implementing the method for joint identification of surgical instruments, behaviors and target tissues as described in any one of claims 1 to 4.

7. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes the method for joint identification of surgical instruments, behaviors and target tissues as described in any one of claims 1 to 4.

Images