A method, system, device and medium for locating and tracking bleeding points in minimally invasive surgery

Abstract

A method, system, device and medium for locating and tracking bleeding points in minimally invasive surgery, relating to the field of image processing technology, including obtaining a surgical video frame sequence, performing preprocessing and preliminary encoding, and generating an initial spatiotemporal feature map; using a neural network to extract spatiotemporal features of different resolutions in the initial spatiotemporal feature map to form a multi-scale feature map; constructing a BLAZEN model, based on the multi-scale feature map, using a multi-scale feature fusion method, a cross-attention mechanism, a feature alignment method and a feature fusion method to generate an enhanced feature map for accurately locating potential bleeding points; constructing an HSN model, assigning different weights to multiple scale features in the enhanced feature map and fusing and calculating them to obtain a high-quality feature map; using the high-quality feature map to locate and track bleeding points; and being able to locate and track bleeding points during surgery.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, system, device and medium for locating and tracking bleeding points in minimally invasive surgery. Background Art

[0002] During medical surgery, timely identification of the location of bleeding points and taking effective hemostasis measures are crucial to ensuring surgical safety and improving patient prognosis; however, due to the complexity of surgical scenes and the rapid occurrence of bleeding events, the bleeding points are usually quickly covered by blood, making it difficult to continuously identify the exact location of the bleeding points.

[0003] Current technology mainly uses an endoscope to film the surgical process, and then uses color features to determine whether there is blood in a single picture taken, or uses a target positioning network to find the blood-covered area.

[0004] However, the existing technology can only detect the blood-covered area in the image, but cannot accurately locate the position of the bleeding point, and is not clinically practical; and the existing technology can only realize the recognition of a single image, and cannot continuously and accurately track the bleeding point based on time series information.

[0005] A Chinese patent with publication number CN113421231B discloses a bleeding point detection method, device and system, which outputs a binary mask detection image through a target network model, determines the bleeding point position inside the target object based on the bleeding area in the binary mask detection image, and accurately assists in locating the bleeding point in real time, reminding the doctor to find the bleeding point as soon as possible and treat the bleeding point. By inputting the optical flow detection image into the target network model, the optical flow information (i.e., the motion information of the bleeding point) can be input into the target network model, so that the motion information can be used to guide the target network model to make a more accurate judgment, thereby improving the accuracy of bleeding point position detection; however, the position of the bleeding point may change with the progress of the operation, and this method cannot track and judge the bleeding point;

[0006] Therefore, we propose a method that can locate and track the bleeding point. Summary of the invention

[0007] The purpose of the present invention is to provide a method, system, device and medium for locating and tracking bleeding points in minimally invasive surgery.

[0008] The present invention is achieved through the following technical solutions:

[0009] A method for locating and tracking bleeding points in minimally invasive surgery includes:

[0010] Obtain surgical video frame sequences, perform preprocessing and preliminary encoding, and generate initial spatiotemporal feature maps;

[0011] A neural network is used to extract spatiotemporal features of different resolutions from the initial spatiotemporal feature map to form a multi-scale feature map;

[0012] Construct the BLAZEN model, based on multi-scale feature maps, and use multi-scale feature fusion, cross-attention mechanism, feature alignment and feature fusion methods to generate enhanced feature maps for accurately locating potential bleeding points;

[0013] Construct an HSN model, assign different weights to multiple scale features in the enhanced feature map and fuse them to obtain a high-quality feature map;

[0014] Use high-quality feature maps to locate and track bleeding points.

[0015] Furthermore, the multi-scale feature fusion method includes:

[0016] Each feature in the multi-scale feature map is normalized;

[0017] 3D convolution is used to dynamically upsample low-resolution features, and the cross-attention mechanism is used to fuse low-resolution features into high-resolution features.

[0018] Furthermore, the cross attention mechanism includes:

[0019] Semantic features are used to guide and strengthen the expression of bleeding-related features;

[0020] Generate an attention weight map to highlight potential bleeding areas.

[0021] Furthermore, the feature alignment method is specifically as follows: a linear layer is used to process the dynamically upsampled features and the attention weight map respectively.

[0022] Furthermore, the feature fusion method is specifically: using 3D convolution to fuse the aligned dynamic up-sampled features and the attention weight map to obtain an enhanced feature map.

[0023] Furthermore, the specific steps of the HSN model include:

[0024] Perform preliminary convolution processing on the feature maps of each scale in the enhanced feature map to unify the number of channels of the feature maps of each scale;

[0025] Adaptively calculate the weight of each scale feature map;

[0026] According to the weight of each scale feature map, the features of all scales are weighted summed.

[0027] Furthermore, the method for locating and tracking the bleeding point includes:

[0028] Based on the high-quality feature map, a classifier is used to determine whether there is a bleeding point in the current scene;

[0029] Determine whether there is a bleeding point in each frame of the high-quality feature map;

[0030] Determine the coordinates of the bleeding point, and use the information between consecutive frames to predict the position of the bleeding point in each bleeding frame;

[0031] Connect multiple locations to complete the tracking of bleeding points.

[0032] A bleeding point positioning and tracking system for minimally invasive surgery, comprising an acquisition module, a feature extraction module, a BLAZEN module, an HSN module and an output module;

[0033] The acquisition module is used to acquire the frame sequence of the surgical video, and perform preprocessing and preliminary encoding to generate an initial spatiotemporal feature map;

[0034] The feature extraction module uses a neural network to extract spatiotemporal features of different resolutions in the initial spatiotemporal feature map to form a multi-scale feature map;

[0035] The BLAZEN module has a BLAZEN model, which generates an enhanced feature map based on a multi-scale feature map, using a multi-scale feature fusion, a cross-attention mechanism, a feature alignment and a feature fusion method, for accurately locating potential bleeding points;

[0036] The HSN module constructs an HSN model, which assigns different weights to multiple scale features in the enhanced feature map and fuses them for calculation to obtain a high-quality feature map;

[0037] The output module uses high-quality feature maps to locate and track bleeding points.

[0038] A computer-readable storage medium stores a computer program, characterized in that when the program is executed by a processor, the above-mentioned bleeding point positioning and tracking method for minimally invasive surgery is implemented.

[0039] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable, wherein the processor implements the above-mentioned method for locating and tracking bleeding points in minimally invasive surgery when executing the program.

[0040] The technical solution of the present invention has at least the following advantages and beneficial effects:

[0041] The present invention discloses a method, system, device and medium for locating and tracking bleeding points in minimally invasive surgery. The scale feature fusion mechanism of the BLAZEN model is used to improve the system's adaptability to complex surgical scenes. The high-quality feature map generated by the HSN model helps to improve the accuracy of bleeding point positioning.

[0042] By utilizing timing information, bleeding points can be tracked in real time, continuously, and with high precision; ultimately, by accurately locating bleeding points during minimally invasive surgery, it has higher clinical practicality than existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0044] Figure 2 It is a schematic diagram of the BLAZEN model process of the present invention;

[0045] Figure 3 It is a schematic diagram of the HSN model process of the present invention;

[0046] Figure 4 It is a schematic diagram of the system flow of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0048] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0050] In the description of the present invention, it should be noted that if the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the product of the application is usually placed when used. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0051] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0052] Example 1

[0053] As attached Figure 1 A method for locating and tracking bleeding points in minimally invasive surgery includes:

[0054] Obtain surgical video frame sequences, perform preprocessing and preliminary encoding, and generate initial spatiotemporal feature maps;

[0055] The surgical video is a video stream from the laparoscope, usually a 30fps RGB video.

[0056] The video preprocessing process is as follows:

[0057] a. Resolution unification: adjust the input video to a uniform resolution, such as 640x480 pixels, and use center cropping and scaling to obtain 224x224 pixel input;

[0058] b. Frame sampling: 10 frames are sampled per second to balance computational efficiency and temporal resolution, and finally form the initial spatiotemporal feature map.

[0059] A neural network is used to extract spatiotemporal features of different resolutions from the initial spatiotemporal feature map to form a multi-scale feature map;

[0060] The specific process is: use a neural network (such as SwinTransformer3D) as the backbone network to extract the spatiotemporal features of the video sequence:

[0061] The input uses 32 consecutive frames of preprocessed images to form a 32x224x224x3 tensor;

[0062] Then perform 3D feature encoding to encode the input into 3D features to form 32x196x128 features;

[0063] After that, feature extraction is performed at different stages, i.e., through four SwinTransformer3D modules, more advanced spatiotemporal features are gradually extracted;

[0064] Finally, multi-scale feature maps are generated as output, including feature maps with resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32.

[0065] Construct the BLAZEN model, based on multi-scale feature maps, and use multi-scale feature fusion, cross-attention mechanism, feature alignment and feature fusion methods to generate enhanced feature maps for accurately locating potential bleeding points;

[0066] Construct an HSN model, assign different weights to multiple scale features in the enhanced feature map and fuse them to obtain a high-quality feature map;

[0067] Use high-quality feature maps to locate and track bleeding points.

[0068] Example 2

[0069] As attached Figure 2 As shown, in the BLAZEN model, the multi-scale feature fusion method includes:

[0070] Each feature in the multi-scale feature map is normalized;

[0071] 3D convolution is used to dynamically upsample low-resolution features, and the cross-attention mechanism is used to merge low-resolution features into high-resolution features.

[0072] The specific steps of normalization are: use the Layer Norm layer to normalize feature maps of different resolutions, and for the input feature map x, the calculation formula of Layer Norm is as follows:

[0073] ;

[0074] in:

[0075] is the mean of the feature map in the channel dimension: ;

[0076] is the variance of the feature map in the channel dimension: ;

[0077] is a small constant (such as 1e-5) for numerical stability;

[0078] γ and β are learnable scaling and translation parameters, and C is the number of channels;

[0079] Layer Norm normalizes the feature vector at each position independently, making the feature distribution more stable and facilitating network training.

[0080] The specific steps for dynamically upsampling low-resolution features are: Use 3D convolution, i.e. Conv3D Module (e.g. Conv3D->BN3D->GELU) to dynamically upsample the low-resolution feature map. First, the input low-resolution feature map is given: , where B, T, , H and W are the batch size, number of frames, number of input channels, height and width of the feature map, respectively. B, T, , H and W are the batch size, number of frames, number of output channels, height and width of the feature map respectively, and then perform the following operations:

[0081] a.3D convolution operation:

[0082] ;

[0083] .

[0084] b. Normalization:

[0085] ;

[0086] ;

[0087] .

[0088] c.GELU activation:

[0089] ;

[0090] ;

[0091] .

[0092] d. Dynamic upsampling:

[0093] ;

[0094] is the trilinear interpolation kernel function:

[0095] ;

[0096] , , is the upsampling scale factor for the temporal and spatial dimensions.

[0097] e. Output upsampled feature map: .

[0098] The cross attention mechanism includes:

[0099] Semantic features are used to guide and strengthen the expression of bleeding-related features;

[0100] Generate an attention weight map to highlight potential bleeding areas;

[0101] Specifically, the cross attention module is used to fuse the low-resolution feature map into the high-resolution feature map. The specific fusion method is:

[0102] a. Input high-resolution feature map and low-resolution feature maps ;

[0103] b. Query, Key, Value mapping, i.e. Q, K, V; Query, Key, Value mapping corresponds to query, key, and value mapping respectively, which is the semantic feature in the cross-attention mechanism:

[0104] ;

[0105] ;

[0106] .

[0107] c. Calculate attention weights:

[0108] ;

[0109] ;

[0110] d. Feature fusion:

[0111] ;

[0112] e. Multi-head attention mechanism:

[0113] ;

[0114] .

[0115] f. Residual connection and normalization:

[0116] .

[0117] g. Output the fused feature map: , i.e., the attention weight map;

[0118] in , , , is a learnable linear change matrix, , , is the learnable bias matrix, is the dimension of Key, n is the number of attention heads, It is a channel concatenation function that concatenates the output results of multiple attention heads together in the channel C dimension.

[0119] The feature alignment method is specifically as follows: a linear layer, namely Linear, is used to process the dynamically upsampled features and the attention weight map.

[0120] Specifically, two linear layers are used to align the upsampled feature map and the attention weight map. The alignment operation is to use two linear layers to make a projection change on the two parts of the feature map to achieve dynamic scaling, so that the high-resolution features in the multi-scale feature map can be focused on potential bleeding points.

[0121] a. Input features are: .

[0122] b. Projection changes: ;

[0123] Where W is the linear change matrix, b is the bias matrix, and the output .

[0124] The feature fusion method is specifically: using 3D convolution, that is, Conv3D Module, to fuse the aligned dynamic up-sampled features and the attention weight map to obtain an enhanced feature map.

[0125] Specifically, the upsampled feature map and the attention weight map are concatenated together, and the final feature map can be fused and output by using 3D convolution, i.e. Conv3D Module (e.g. Conv3D->BN3D->GELU);

[0126] The concatenation operation is performed in the channel dimension, that is, the input , , output , the fusion operation is as follows:

[0127] ;

[0128] in =C, =2C, after fusion That is, the feature map is enhanced to provide high-quality bleeding-related information for subsequent processing.

[0129] In addition, in the process of generating the attention weight map, there is a role of dynamically scaling the multi-scale feature map through dynamic upsampling and cross-attention mechanism, thereby focusing the high-resolution features on the potential bleeding area and improving the positioning accuracy;

[0130] In addition, in the calculation process of the above-mentioned 3D convolution, namely Conv3D Module, the information of adjacent frames is used to enhance the temporal consistency of each spatiotemporal feature in the multi-scale feature map; thereby, the features of continuous frames capture the temporal dynamics of bleeding.

[0131] Example 3

[0132] As attached Figure 3 The specific steps of the HSN model shown include:

[0133] Receive the enhanced feature map, perform preliminary convolution processing on the feature maps of each scale in the enhanced feature map, and unify the number of channels of the feature maps of each scale;

[0134] Adaptively calculate the weight of each scale feature map;

[0135] According to the weight of each scale feature map, the features of all scales are weighted summed;

[0136] Specifically, the first received enhanced feature map contains 4 feature maps of different scales, which are spliced ​​together, where the splicing operation is performed in the channel dimension. , , , Output ;

[0137] Then, 3D convolution, namely Conv3D Module (e.g. Conv3D->BN3D->GELU) and Layer Norm layer are used to normalize the concatenated feature map, and the specific details of the 3D convolution and normalization operations are consistent with the operations in the BLAZEN model.

[0138] Then, the first 3D convolution, i.e. Conv3D Module (e.g. Conv3D->BN3D->GELU), Tanh module, the second 3D convolution, i.e. Conv3D Module (e.g. Conv3D->BN3D->GELU) and Softmax module are used to extract the importance of feature map weights at different levels, i.e. importance feature extraction. The specific extraction method is as follows:

[0139] a. Input ;

[0140] b. The first 3D convolution:

[0141] ;

[0142] c.Tanh activation function:

[0143] ;

[0144] d. The second 3D convolution:

[0145] ;

[0146] e.Softmax weight calculation:

[0147] ;

[0148] For each spatial position (t,h,w), ;

[0149] in , is the linear change matrix, ;

[0150] Finally, the feature maps are weighted summed according to the calculated weights, i.e. feature fusion. The calculation process is:

[0151] .

[0152] Example 4

[0153] The method for locating and tracking the bleeding point includes:

[0154] Based on the high-quality feature map, a classifier is used to determine whether there is a bleeding point in the current scene;

[0155] Determine whether there is a bleeding point in each frame of the high-quality feature map;

[0156] Determine the coordinates of the bleeding point, and use the information between consecutive frames to predict the position of the bleeding point in each bleeding frame;

[0157] Connect multiple locations to complete the tracking of bleeding points.

[0158] Bleeding classification: Use global average pooling and a one-layer fully connected network to output three types of probabilities, including no bleeding, hemorrhage, and clinically relevant bleeding;

[0159] Bleeding frame classification: Apply global average pooling to the feature map generated by HSN to obtain the feature map of each frame, and use a one-layer fully connected network to output whether each frame has bleeding;

[0160] Bleeding point positioning and tracking: Expand the feature map of each frame on the feature map generated by HSN, use a layer of fully connected network to output the coordinates of the bleeding point of each frame, and generate and output the motion trajectory of the bleeding point in the video sequence.

[0161] Example 5

[0162] As attached Figure 4 A bleeding point positioning and tracking system for minimally invasive surgery is shown, comprising an acquisition module, a feature extraction module, a BLAZEN module, an HSN module and an output module;

[0163] The acquisition module is used to acquire the frame sequence of the surgical video, and perform preprocessing and preliminary encoding to generate an initial spatiotemporal feature map;

[0164] The feature extraction module uses a neural network to extract spatiotemporal features of different resolutions in the initial spatiotemporal feature map to form a multi-scale feature map, and the feature extraction module is composed of multiple components, each feature extraction module is used to extract spatiotemporal features of a resolution, and finally form a multi-scale feature map;

[0165] The BLAZEN module has a BLAZEN model, which generates an enhanced feature map based on a multi-scale feature map, using a multi-scale feature fusion, a cross-attention mechanism, a feature alignment and a feature fusion method, so as to accurately locate potential bleeding points. The BLAZEN module also has multiple modules, which analyze and process the spatiotemporal features of each resolution respectively, and the final processing results are fused into an enhanced feature map;

[0166] The HSN module constructs an HSN model, which assigns different weights to multiple scale features in the enhanced feature map and fuses them for calculation to obtain a high-quality feature map;

[0167] The output module uses high-quality feature maps to locate and track bleeding points, and the output module can also distinguish the types of bleeding points.

[0168] Example 6

[0169] A computer-readable storage medium stores a computer program, characterized in that when the program is executed by a processor, the above-mentioned bleeding point positioning and tracking method for minimally invasive surgery is implemented.

[0170] Example 7

[0171] An electronic device comprises a memory, a processor and a computer program stored and executable on the memory, wherein the processor implements the above-mentioned method for locating and tracking bleeding points in minimally invasive surgery when executing the program.

[0172] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for locating and tracking bleeding points in minimally invasive surgery, characterized in that: include: Obtain surgical video frame sequences, perform preprocessing and preliminary encoding, and generate initial spatiotemporal feature maps; A neural network is used to extract spatiotemporal features of different resolutions from the initial spatiotemporal feature map to form a multi-scale feature map; Construct the BLAZEN model, based on the multi-scale feature map, and use the multi-scale feature fusion method, cross-attention mechanism, feature alignment method and feature fusion method to generate enhanced feature maps for accurately locating potential bleeding points; The multi-scale feature fusion method comprises: Each feature in the multi-scale feature map is normalized; 3D convolution is used to dynamically upsample low-resolution features, and the cross-attention mechanism is used to merge low-resolution features into high-resolution features; The cross attention mechanism includes: Semantic features are used to guide and strengthen the expression of bleeding-related features; Generate an attention weight map to highlight potential bleeding areas; The feature alignment method is specifically as follows: a linear layer is used to process the dynamically upsampled features and the attention weight map respectively; The feature fusion method specifically comprises: using 3D convolution to fuse the aligned dynamic up-sampled features and the attention weight map to obtain an enhanced feature map; Construct an HSN model, assign different weights to multiple scale features in the enhanced feature map and fuse them to obtain a high-quality feature map; The specific operation process of the HSN model includes: Perform preliminary convolution processing on the feature maps of each scale in the enhanced feature map to unify the number of channels of the feature maps of each scale; Adaptively calculate the weight of each scale feature map; According to the weight of each scale feature map, the features of all scales are weighted summed; Use high-quality feature maps to locate and track bleeding points; The process of locating and tracking the bleeding point includes: Based on the high-quality feature map, a classifier is used to determine whether there is a bleeding point in the current scene; Determine whether there is a bleeding point in each frame of the high-quality feature map; Determine the coordinates of the bleeding point, and use the information between consecutive frames to predict the position of the bleeding point in each bleeding frame; Connect multiple predicted positions to complete the tracking of bleeding points.

2. A bleeding point location tracking system for minimally invasive surgery using the bleeding point location tracking method for minimally invasive surgery as claimed in claim 1, comprising an acquisition module, a feature extraction module, a BLAZEN module, an HSN module and an output module; The acquisition module is used to acquire the frame sequence of the surgical video, and perform preprocessing and preliminary encoding to generate an initial spatiotemporal feature map; The feature extraction module uses a neural network to extract spatiotemporal features of different resolutions in the initial spatiotemporal feature map to form a multi-scale feature map; The BLAZEN module has a BLAZEN model, which generates an enhanced feature map based on a multi-scale feature map, using a multi-scale feature fusion, a cross-attention mechanism, a feature alignment and a feature fusion method, for accurately locating potential bleeding points; The HSN module constructs an HSN model, which assigns different weights to multiple scale features in the enhanced feature map and fuses them for calculation to obtain a high-quality feature map; The output module uses high-quality feature maps to locate and track bleeding points.

3. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable, characterized in that: When the processor executes the program, the bleeding point positioning and tracking method for minimally invasive surgery as described in claim 1 is implemented.

4. A computer-readable storage medium storing a computer program, characterized in that: When the program is executed by a processor, the bleeding point positioning and tracking method for minimally invasive surgery as claimed in claim 1 is implemented.

Images