Segmentation method, device and medium for medical video inconspicuous lesion area

Abstract

The present invention provides a method, device and medium for segmenting inconspicuous lesion areas in medical videos, including first collecting gastrointestinal video data, ensuring the high quality of the data through preprocessing steps, and using memory alignment network technology to capture the spatial feature correspondence between adjacent video frames and the current frame, thereby enhancing the feature representation of the current frame. The enhanced features are sent to the main decoder, which effectively integrates high-level semantic features with low-level detail features through a constructed multi-level feature fusion network, and deeply mines the spatial and structural information of inconspicuous lesions to generate a preliminary segmentation map. A progressive segmentation optimization network is applied to further refine the segmentation results to improve the accuracy of segmentation. The model trained through supervised learning can output an accurate predicted segmentation map of inconspicuous lesions in each image sequence. The present invention effectively solves the problem that inconspicuous lesions of the gastrointestinal tract are difficult to detect, and significantly improves the accuracy of segmentation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a segmentation method, device and storage medium for an inconspicuous lesion area in a medical video. Background Art

[0002] In modern endoscopic examinations, accurate segmentation technology for inconspicuous lesion areas can significantly reduce misdiagnosis and missed diagnosis, improve the early detection rate of colorectal cancer, thereby achieving more accurate treatment and reducing mortality. However, existing segmentation technology still has certain limitations when dealing with inconspicuous lesion areas, resulting in low segmentation accuracy. Therefore, the field of medical research and technology development is actively exploring more accurate methods for segmenting lesion areas in gastrointestinal video data, in order to provide patients with safer, more effective and accurate detection methods.

[0003] With the rapid development of artificial intelligence technologies such as deep learning in the field of medical imaging, initial progress has been made in the segmentation of inconspicuous lesion areas in gastrointestinal video data, and it has shown great potential. However, these technologies still face many challenges in clinical applications, such as the low contrast between inconspicuous lesion areas and surrounding tissues, and the difficulty of segmentation in single imaging.

[0004] To accurately segment the inconspicuous lesion areas in gastrointestinal video data, it is necessary to consider factors such as the morphological changes, growth patterns, and image noise of the lesions. These factors are crucial for accurately analyzing the nature of lesions and diagnosing related diseases. Although existing deep learning models have made some progress in fully automatic segmentation, their effectiveness depends largely on the training of a large amount of labeled data. These models have not yet fully utilized deep learning and other machine learning techniques to deeply analyze the morphological and contextual features of inconspicuous lesion areas, resulting in the need to improve the segmentation effect. Therefore, future research should focus on improving data quality, optimizing algorithms, and in-depth exploration of the imaging characteristics of inconspicuous lesion areas.

[0005] In summary, the precise segmentation method for unclear lesion areas in gastrointestinal video data has far-reaching clinical significance. It can not only improve the detection efficiency of early colorectal cancer, but also promote the development of medical artificial intelligence technology. Although the current technology faces accuracy challenges and has not yet been widely used in clinical practice, its potential and development prospects are worthy of further exploration and investment. Summary of the invention

[0006] The present invention proposes a method, device and storage medium for segmenting inconspicuous lesion areas in medical videos, which can solve at least one of the technical problems in the background technology. Specifically, it can solve the technical problem that the prior art does not fully utilize deep learning and other machine learning technologies to analyze the micro-temporal and spatial characteristics of inconspicuous lesion areas and the hidden boundaries, resulting in inaccurate segmentation.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for segmenting an inconspicuous lesion area in a medical video, wherein a computer device performs the following steps:

[0009] S1. Collect the initial gastrointestinal tract unclear lesion image dataset, including images and segmentation labels, screen the data, divide the dataset into training set and test set, and process the training data;

[0010] S2, build a memory alignment network to extract the spatial information of the current frame and the historical frames, and enhance the features of the current frame through the attention mechanism, and then pass this information to the main encoder;

[0011] S3, build the main encoder to fully extract the enhanced image features, and use the multi-level feature fusion network to further explore the high-level features in a cascade manner to obtain accurate location information;

[0012] S4, introduce a progressive segmentation optimization network, use the initial segmentation results to guide the low-level features to further optimize and improve the segmentation details, and obtain the segmentation model;

[0013] S5, using the collected training samples to train the segmentation model, and using the test samples to test and select the optimal network parameters to form the final inconspicuous lesion segmentation model;

[0014] S6. When applied, the image sequence containing the inconspicuous lesion area is input into the above-mentioned final inconspicuous lesion area segmentation model, and the inconspicuous lesion area segmentation image is output after calculation.

[0015] Furthermore, the step S1 comprises:

[0016] S11. Collect image datasets of gastrointestinal tract areas with subtle lesions;

[0017] S12. Filter out the datasets of unclear lesion areas through the basic algorithm, and retain the datasets containing lesions with segmentation accuracy less than 50%;

[0018] S13, adjusting the image resolution to 256×256 by bilinear interpolation method, and dividing the data set into training data and test data;

[0019] S14, perform data augmentation by randomly flipping the images and their labels in the training dataset horizontally, rotating them 90 degrees clockwise, and rotating them 180 degrees clockwise;

[0020] S15. Using the image sequence of the inconspicuous lesion area in the above-mentioned data-enhanced training data set as input, and using the output of the main decoder as the final inconspicuous lesion segmentation result of the model.

[0021] Furthermore, the step S2 comprises:

[0022] S21, constructing a memory alignment network: the input image sequence of the inconspicuous lesion area shares the memory alignment network to obtain an enhanced feature set;

[0023] S22, the memory alignment network consists of a ResNet50 encoder with initialized pre-trained weights, a cross-frame non-local attention module, and a memory enhancement module;

[0024] S23, initialize the ResNet50 encoder with pre-trained weights and input the current frame and adjacent frames , extracting the feature representation of each image separately. ResNet50 starts with a 7x7 convolutional layer with a stride of 2 and 64 filters, followed by a batch normalization (BN) layer and a ReLU activation layer. This is followed by a 3x3 max pooling layer with a stride of 2. The initial image size is 224×224, and after this stage the size becomes 56×56.

[0025] The main body of the network is divided into four stages, each of which contains a different number of residual blocks: Stage 1: Contains 3 residual blocks, each of which has two convolutional layers. The first is a 1x1 convolutional layer with 64 filters for dimensionality reduction; the second is a 3x3 convolutional layer with 64 filters. There is also a BN layer and a ReLU activation layer between these two convolutional layers. The output of each residual block is dimensionally matched through a 1x1 convolutional layer (with 256 filters) and then added to the input. After processing in this stage, the image output is still 56×56, but the number of channels increases from 64 to 256. Stage 2: Contains 4 residual blocks, each of which has three convolutional layers. The first 1x1 convolutional layer has 128 filters, the second 3x3 convolutional layer also has 128 filters, and the third 1x1 convolutional layer has 256 filters. The first convolutional layer is followed by a BN layer and a ReLU activation layer, and the second convolutional layer is followed by a BN layer and a ReLU activation layer. After stage 2, the number of image channels becomes 512 and the size becomes 28×28. Stage 3: Contains 6 residual blocks, each of which also has three convolutional layers. The first 1x1 convolutional layer has 256 filters, the second 3x3 convolutional layer also has 256 filters, and the third 1x1 convolutional layer has 512 filters. BN layers and ReLU activation layers are also interspersed between these convolutional layers. At this point, the number of image channels becomes 1024 and the size becomes 14×14. Stage 4: Contains 3 residual blocks, each of which has three convolutional layers. The first 1x1 convolutional layer has 512 filters, the second 3x3 convolutional layer also has 512 filters, and the third 1x1 convolutional layer has 1024 filters. There are also BN layers and ReLU activation layers between these convolutional layers. At the beginning of each stage, to reduce the size of the feature map, the first 1x1 convolution layer in the first residual block has a stride of 2, and an additional 1x1 convolution layer (with a stride of 2) is used to process the input to match the dimensions. The final image has 2048 channels and a size of 7×7.

[0026] S24. In the cross-frame non-local attention module, the central feature is used As a query tensor , adjacent frame features { } as the key tensor and value tensor For each pixel in the query tensor, non-local attention is only performed on its spatial neighbors in the temporal neighbor frames. The K time slices of are expanded into 9×9 patches in the spatial domain. Therefore, for each pixel There is a query vector , the vector corresponding to key becomes , the corresponding matrix Becomes 1×81. Each row of is different; each It only represents the spatial neighborhood of a specific pixel in the current frame. , and the final correlation matrix Γ is obtained. For each row in Γ, only the largest entries are selected: , .in is the attention score, yes The column coordinates in . and They form one-hot attention results respectively and . After this operation, , which is transformed into As the output of the cross-frame non-local attention module.

[0027] S25, the memory enhancement module contains a global memory block As the parameters of the network, non-local attention is used to query the current features in the global memory block M. . . Similar to the cross-frame non-local attention module, , which is transformed into As the output of the memory enhancement module.

[0028] S26, will and After a convolution operation, the sum is performed , and then sent to the main encoder.

[0029] Furthermore, the step S3 comprises:

[0030] S31, the main decoder consists of a multi-level feature fusion network;

[0031] S32, the multi-level feature fusion network uses three highest-level features To perform adjacent fusion, specifically, first, the three layers of the highest level features extracted by the backbone network , perform a 3x3 convolution operation to reduce the number of channels to 32. Then use average pooling (GAP) and point convolution (PWC) to convolution the features. Realize channel modulation and achieve channel semantic alignment Then, high-level features are used to enhance low-level features across layers while ensuring semantic consistency within the layer. Perform two - fold upsampling and then multiply with the feature to obtain P. Similarly, for the feature perform two - fold upsampling and then multiply with the feature to obtain Q. Then, perform two - fold upsampling on P and multiply with Q to further obtain R. Then, perform four - fold upsampling on the feature and concatenate it with the obtained R. Then, pass through a 3×3 convolution and a Sigmoid function to obtain the initial segmentation map, and input it into the progressive optimization network for optimizing the segmentation result. This process can be expressed as:

[0032]

[0033] where represents element - wise multiplication, represents a 3×3 convolution block, represents two - fold upsampling.

[0034] Furthermore, the step S4 includes:

[0035] S41. Introduce a progressive segmentation optimization network to capture the details of non - obvious lesions in the low - level feature map and further optimize the segmentation result;

[0036] S42. The progressive segmentation optimization network consists of a foreground - guiding block, a background - guiding block, and a multi - scale context - awareness block. Using the segmentation result of the high - level feature as prior knowledge, it respectively performs foreground - guiding and background - guiding on the low - level features;

[0037] S43. First, take the high - level prediction as the foreground - guiding map and obtain the background - guiding map through a reverse operation , and adopt a progressive iterative method to respectively guide the foreground and the background;

[0038] S44. First, expand the number of channels of the current feature { }, i ∈ {1, 2, 3} through a 1×1 convolution, and divide it into G groups along the channel dimension. Then, interpolate the foreground - guiding map (background - guiding map ) among the segmentation features to obtain the feature . The first group g 1 is further divided into three feature sets through a convolutional layer . Among them, is used for iteration with the next group, and the other two are used for channel modulation. For the j - th group (1 < j < G), the feature is combined with the Connect along the channel dimension, followed by the convolution layer and segmentation operation, and also divide the feature group into three feature sets. It should be noted that The output of the group contains only the and After iteration, the feature Concatenated and converted into a feature modulation vector through global average pooling (GAP) and point-wise convolution (PWC) layers , used to weight the features , add weighted features to the input features Then the foreground guidance features are obtained through the convolution layer (Background guide features ).

[0039] S45, the multi-scale context-aware block consists of four branches. In each branch, 1×1 convolution is first used to reduce the channel. In order to improve the inference efficiency without reducing the representation ability, and Convolutional layers consisting of convolutions are used to capture multi-scale contextual information. This is followed by a 3×3 dilated convolution with a dilation rate of Output , i∈{1,2,3} will be sent to the next branch for further processing in a larger receptive field. , i∈{1,2,3,4} are set to 1,3,5,7 respectively, , i∈{1, 2, 3, 4} is set to 1, 2, 4, 8 respectively. The output of each branch is fused through a 1×1 convolution to obtain the final false positive feature (False negative feature ), and then correct the current features to obtain the corrected features, and further obtain a more accurate segmentation map.

[0040] Furthermore, the step S5 comprises:

[0041] S51. Build a fully supervised neural network segmentation network for the inconspicuous lesion area, use the Adam optimizer, and set the initial learning rate to , reduced by 0.5 every 20 epochs; the input is a sequence of images of areas with inconspicuous lesions. After passing through the memory alignment network, the spatiotemporal information in the sequence is fully acquired, and the acquired spatiotemporal order is passed to the main decoder to output the segmentation result Mask.

[0042] S52. During the training process, a 5-fold cross validation method will be used to evaluate the performance of the model. Assuming that the training set D contains labeled data, the training set D is evenly divided into 5 subsets, one of which is used as a validation set each time, and the remaining 4 subsets are used for training.

[0043] S53. A combination of the cross entropy loss function and the Dice loss function is used to guide the network training process to better achieve convergence effect.

[0044] The loss function L of the neural network for the image of the inconspicuous lesion area is set as:

[0045]

[0046]

[0047] is the cross entropy loss, which is used to measure the difference between the predicted value and the true value. is the true value, is the predicted value;

[0048]

[0049] Dice loss is a loss function commonly used in image segmentation tasks to measure the similarity between two samples; is the predicted probability of the ith pixel in the predicted segmentation map, is the true label of the i-th pixel in the true segmentation map;

[0050] S54, training the model through the loss function, adjusting the parameters through back propagation, and selecting the model parameters with the highest accuracy in the test set for saving;

[0051] S55. Select evaluation indicators including Dice coefficient, IOU index, Accuracy, Recall index, Precision index, Specificity coefficient, S-Measure index, and MAE index to evaluate model performance.

[0052] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0053] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0054] It can be seen from the above technical solutions that the segmentation method for inconspicuous lesion areas in medical videos of the present invention, by cleverly constructing a spatiotemporal network to obtain valuable information in historical frames, mines key information in inconspicuous lesion sequence images, and effectively improves the model's ability to identify and segment inconspicuous lesion areas. Before inputting into the spatiotemporal network, effective image preprocessing technology is adopted, including data enhancement of image flipping, which not only optimizes data quality, but also increases data volume, laying the foundation for high-precision segmentation. In the memory alignment network, the introduction of non-local attention mechanism and memory enhancement module fully obtains the spatiotemporal information between inconspicuous lesion sequence images, laying the foundation for further feature fusion, and then fully fuses the spatiotemporal features through a multi-level feature fusion network. Finally, a progressive segmentation optimization network is used for the underlying features to obtain more detailed information, further improving the accuracy of segmentation. The segmentation model of inconspicuous lesion areas has been rigorously trained and tested, and cross-validation and performance comparison with traditional methods have shown excellent segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of the steps of a segmentation method according to an embodiment of the present invention;

[0056] Figure 2 It is a structural schematic diagram of a segmentation model according to an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of a memory alignment network structure according to an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of the structure of a main decoder according to an embodiment of the present invention;

[0059] Figure 5 Schematic diagram of a progressive segmentation optimization network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0061] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a schematic diagram of a method for video segmentation of a gastrointestinal tract region with inconspicuous lesions based on spatiotemporal continuity analysis, and the specific steps are as follows:

[0062] Step 1: Collect the initial gastrointestinal lesion area image dataset, including images and segmentation labels, screen the data, divide the dataset into training set and test set, and process the training data.

[0063] The specific process is:

[0064] Step 1.1: Collect the lesion area dataset images;

[0065] Step 1.2: Filter out the datasets of inconspicuous lesion areas through the basic algorithm, and retain the datasets containing inconspicuous lesion areas with segmentation accuracy less than 50%;

[0066] Step 1.3: Adjust the image resolution to 256×256 by bilinear interpolation method, and divide the dataset into training data and test data;

[0067] Step 1.4: Perform data augmentation by randomly flipping the images and their labels in the training dataset horizontally, rotating them 90 degrees clockwise, and rotating them 180 degrees clockwise;

[0068] Step 1.5: Use the image sequence of the inconspicuous lesion area in the above data-enhanced training data set as input, and use the output of the main decoder as the final inconspicuous lesion segmentation result of the model.

[0069] Step 2: Construct a memory alignment network to extract the spatial information of the current frame and the historical frames, enhance the features of the current frame through the attention mechanism, and then pass this information to the main encoder. The implementation process is as follows Figure 3 shown.

[0070] The specific process is:

[0071] Step 2.1: Construct a memory alignment network: The input image sequence of inconspicuous lesions shares the memory alignment network to obtain an enhanced feature set;

[0072] Step 2.2: The memory alignment network consists of a ResNet50 encoder with initialized pre-trained weights, a cross-frame non-local attention module, and a memory enhancement module;

[0073] Step 2.3: Initialize the ResNet50 encoder with pre-trained weights and input the current frame and adjacent frames , extracting the feature representation of each image separately. ResNet50 starts with a 7x7 convolutional layer with a stride of 2 and 64 filters, followed by a batch normalization (BN) layer and a ReLU activation layer. This is followed by a 3x3 max pooling layer with a stride of 2. The initial image size is 224×224, and after this stage the size becomes 56×56.

[0074] The main body of the network is divided into four stages, each of which contains a different number of residual blocks: Stage 1: Contains 3 residual blocks, each of which has two convolutional layers. The first is a 1x1 convolutional layer with 64 filters for dimensionality reduction; the second is a 3x3 convolutional layer with 64 filters. There is also a BN layer and a ReLU activation layer between these two convolutional layers. The output of each residual block is dimensionally matched through a 1x1 convolutional layer (with 256 filters) and then added to the input. After processing in this stage, the image output is still 56×56, but the number of channels increases from 64 to 256. Stage 2: Contains 4 residual blocks, each of which has three convolutional layers. The first 1x1 convolutional layer has 128 filters, the second 3x3 convolutional layer also has 128 filters, and the third 1x1 convolutional layer has 256 filters. The first convolutional layer is followed by a BN layer and a ReLU activation layer, and the second convolutional layer is followed by a BN layer and a ReLU activation layer. After stage 2, the number of image channels becomes 512 and the size becomes 28×28. Stage 3: Contains 6 residual blocks, each of which also has three convolutional layers. The first 1x1 convolutional layer has 256 filters, the second 3x3 convolutional layer also has 256 filters, and the third 1x1 convolutional layer has 512 filters. BN layers and ReLU activation layers are also interspersed between these convolutional layers. At this point, the number of image channels becomes 1024 and the size becomes 14×14. Stage 4: Contains 3 residual blocks, each of which has three convolutional layers. The first 1x1 convolutional layer has 512 filters, the second 3x3 convolutional layer also has 512 filters, and the third 1x1 convolutional layer has 1024 filters. There are also BN layers and ReLU activation layers between these convolutional layers. At the beginning of each stage, to reduce the size of the feature map, the first 1x1 convolution layer in the first residual block has a stride of 2, and an additional 1x1 convolution layer (with a stride of 2) is used to process the input to match the dimensions. The final image has 2048 channels and a size of 7×7.

[0075] Step 2.4: In the cross-frame non-local attention module, use the center feature As a query tensor , adjacent frame features { } as the key tensor and value tensor For each pixel in the query tensor, non-local attention is only performed on its spatial neighbors in the temporal neighbor frames. The K time slices of are expanded into 9×9 patches in the spatial domain. Therefore, for each pixel There is a query vector , the vector corresponding to key becomes , the corresponding matrix Becomes 1×81. Each row of is different; each It only represents the spatial neighborhood of a specific pixel in the current frame. , and the final correlation matrix Γ is obtained. For each row in Γ, only the largest entries are selected: , .in is the attention score, yes The column coordinates in . and They form one-hot attention results respectively. and . After this operation, , which is transformed into As the output of the cross-frame non-local attention module.

[0076] Step 2.5: The memory enhancement module contains a global memory block As the parameters of the network, non-local attention is used to query the current features in the global memory block M. . . Similar to the cross-frame non-local attention module, , which is transformed into As the output of the memory enhancement module.

[0077] Step 2.6: and After a convolution operation, the sum is performed , and then sent to the main encoder.

[0078] Step 3: Build the main decoder to fully extract the enhanced image features, and use the multi-level feature fusion network to further explore the high-level features in a cascade manner to obtain accurate location information. Figure 4 shown.

[0079] The specific process is:

[0080] Step 3.1: The main decoder consists of a multi-level feature fusion network;

[0081] Step 3.2: The multi-level feature fusion network uses the three highest level features To perform adjacent fusion, specifically, first, the three layers of the highest level features extracted by the backbone network , perform 3x3 convolution operations to reduce the number of channels to 32. Then use average pooling (GAP) and point convolution (PWC) to convolution the features. Realize channel modulation and achieve channel semantic alignment Then, high-level features are used to enhance low-level features across layers while ensuring semantic consistency within the layer. Upsample twice and then compare with the feature Multiply them to get P, and do the same for the features Upsample twice and then compare with the feature Multiply them to get Q, then upsample P by two times and multiply it with Q to get R, and then add the feature Perform four times upsampling and then concatenate it with the obtained R, then go through a 3×3 convolution and a Sigmoid function to get the initial segmentation map, which is passed into the progressive optimization network to optimize the segmentation result. The process can be expressed as:

[0082]

[0083] in represents element-wise multiplication, represents a 3×3 convolutional block, Indicates two times upsampling.

[0084] Step 4: Optimize the network through progressive segmentation to filter out interference information in low-level features and further capture the details of the lesion on the feature map. The implementation process is as follows Figure 5 shown.

[0085] S41. A progressive segmentation optimization network is introduced to capture the details of the lesions in the low-level feature map and further optimize the segmentation results.

[0086] Step 4.2: The progressive segmentation optimization network consists of a foreground guidance block, a background guidance block, and a multi-scale context perception block. The segmentation results of high-level features are used as prior knowledge to guide the foreground and background of low-level features respectively.

[0087] Step 4.3: First use the high-level prediction as a foreground guide map After the reverse operation, the background guide map is obtained ,A progressive iterative method is adopted to guide the foreground and background respectively;

[0088] Step 4.4, first use 1×1 convolution to transform the current feature { }, i∈{1,2,3}, and divide it into G groups along the channel dimension. Then, the foreground guidance map is constructed between the segmentation features. (Background Guidance Map ) Perform interpolation to obtain features , the first group g 1 Further divide into three feature sets through the convolutional layer . Among them, Is used for iteration with the next group, and the other two are used for channel modulation. For the j-th group (1 < j < G), the feature Is concatenated with that of the previous group Along the channel dimension, followed by a convolutional layer and a splitting operation, and this feature group is also divided into three feature sets. It should be noted that The output of the group only contains those for channel modulation And . After iteration, the feature Is connected and converted into a feature modulation vector through the global average pooling (GAP) and pointwise convolution (PWC) layers , which is used to weight the feature . The weighted feature is added to the input feature , and then the foreground-guided feature is obtained through the convolutional layer (background-guided feature ).

[0089] Step 4.5: The multi-scale context-aware block consists of four branches. In each branch, first use a 1×1 convolution to reduce the channels. To improve the inference efficiency without reducing the representation ability, a convolutional layer composed of And Convolutions is used to capture multi-scale context information. Subsequently, a 3×3 dilated convolution with a dilation rate of . The output , i ∈ {1, 2, 3} will be sent to the next branch for further processing in a larger receptive field. Set , i ∈ {1, 2, 3, 4} to 1, 3, 5, 7 respectively, , i ∈ {1, 2, 3, 4} to 1, 2, 4, 8 respectively; the output of each branch is fused through a 1×1 convolution to obtain the final false positive feature (false negative feature ), and then the current feature is corrected.

[0090] Step 5: Use the collected training samples to train the above segmentation model including the memory alignment network, encoder module, and fusion attention module, and use the test samples for testing to select the optimal network parameters to form the final non-obvious lesion segmentation model;

[0091] Step 5.1: Build a fully supervised non-obvious lesion neural network segmentation network, use the Adam optimizer, and set the initial learning rate to , set 200 rounds for training, and the learning rate is reduced by 10% every 20 rounds; the input is an image sequence of the inconspicuous lesion area. After the memory alignment network, the spatiotemporal information in the sequence is fully acquired, and the acquired spatiotemporal information is passed to the main decoder to output the initial segmentation result P, and then the segmentation result is further optimized using the progressive segmentation optimization network. The Mask output by the network represents the segmentation map of the inconspicuous lesion area.

[0092] Step 5.2: During the training process, the 5-fold cross-validation method will be used to evaluate the performance of the model. Assume that the training set D contains labeled data. Then the training set D is evenly divided into 5 subsets, one of which is used as the validation set each time, and the remaining 4 subsets are used for training.

[0093] Step 5.3: Use a combination of cross entropy loss function and Dice loss function to guide the network training process to better achieve convergence effect.

[0094] The loss function L of the neural network for the image of the inconspicuous lesion area is set as:

[0095]

[0096]

[0097] is the cross entropy loss, which is used to measure the difference between the predicted value and the true value; is the true value, is the predicted value;

[0098]

[0099] Dice loss is a loss function commonly used in image segmentation tasks to measure the similarity between two samples. is the predicted probability of the ith pixel in the predicted segmentation map, is the true label of the ith pixel in the true segmentation map. Step 5.4: Train the model through the loss function, adjust the parameters through back propagation, and select the model parameters with the highest accuracy in the test set for saving;

[0100] Step 5.5, select evaluation indicators including Dice coefficient, IOU index, Accuracy, Recall index, Precision index, Specificity coefficient, S-Measure index, and MAE index to evaluate the model performance.

[0101] Step 6: When applied, the gastrointestinal image sequence containing inconspicuous lesions is input into the above-mentioned final inconspicuous lesion region segmentation model, and the inconspicuous lesion region segmentation image is output after calculation.

[0102] This model has been fully trained and tested to verify the model, and the recognition accuracy of 11 diseases in 8 parts has reached more than 80%.

[0103] In summary, an embodiment of the present invention discloses a video segmentation method for inconspicuous lesion areas based on spatiotemporal continuity analysis. The method includes collecting a lesion sequence data set and performing data processing. Then, a memory alignment network is used to extract the temporal features and spatial features in the input image sequence respectively, and the extracted features are passed to the main encoder respectively. Secondly, the main encoder is used to effectively fuse the temporal features and spatial features to complete the feature extraction of the input image sequence, realize accurate capture of the representation of inconspicuous lesion areas, and generate an initial segmentation map through multi-level feature fusion. Finally, combined with the detail information compensated by the progressive segmentation optimization network, the segmentation map of the inconspicuous lesion area of ​​each image sequence is output.

[0104] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0105] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0106] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the segmentation methods for inconspicuous lesion areas in medical videos in the above-mentioned embodiments.

[0107] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above methods.

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0109] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0110] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0111] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A segmentation method for inconspicuous lesion areas in medical videos, characterized in that: The following steps are included: S1. Collect an initial gastrointestinal lesion image dataset, wherein the dataset includes images and their corresponding segmentation labels; screen the collected data, divide them into a training set and a test set, and preprocess the training set data; S2. Build a memory alignment network to extract the spatial information of the current frame and the historical frames, enhance the features of the current frame through the attention mechanism, and pass the enhanced feature information to the main encoder for processing; The memory alignment network consists of a ResNet50 encoder with initialized pre-trained weights, a cross-frame non-local attention module, and a memory enhancement module; S3, build the main encoder to fully extract the enhanced image features, and use the multi-level feature fusion network to further explore the high-level features in a cascade manner to obtain accurate location information; S4, introduce a progressive segmentation optimization network, use the initial segmentation results to guide the low-level features to further optimize and improve the segmentation details, and obtain the segmentation model; The progressive segmentation optimization network consists of a foreground guidance block, a background guidance block, and a multi-scale context-aware block. It uses the segmentation results of high-level features as prior knowledge to guide the foreground and background of low-level features respectively. S5, using the collected training samples to train the segmentation model, and using the test samples to test and select the optimal network parameters to form the final inconspicuous lesion segmentation model; S6. When applied, the image sequence containing the inconspicuous lesions is input into the final inconspicuous lesion segmentation model, and the inconspicuous lesion segmentation image is output after calculation.

2. The method for segmenting inconspicuous lesion areas in medical videos according to claim 1, characterized in that: The step S1 comprises: S11. Collect a dataset of gastrointestinal lesion images, perform preliminary screening on the images, and retain images containing unclear lesions and their segmentation labels; S12. Use the basic algorithm to filter out the unclear lesion datasets with segmentation accuracy less than 50% to ensure that the selected datasets can effectively challenge the segmentation ability of the model; S13, uniformly adjusting the image resolution to 256×256 by a bilinear interpolation method to facilitate subsequent processing, and dividing the adjusted data set into a training set and a test set; S14. Perform data augmentation on the training set, including random horizontal flipping, 90-degree and 180-degree clockwise rotation operations, to improve the generalization ability of the model; S15. The data-enhanced training set is used as input, and the output of the main decoder is used as the final result of the inconspicuous lesion segmentation model to achieve high-precision lesion segmentation.

3. The method for segmenting inconspicuous lesion areas in medical videos according to claim 2, characterized in that: The step S2 comprises: Construct a memory alignment network: The input image sequence of the inconspicuous lesion area shares the memory alignment network to obtain an enhanced feature set; Initialize the ResNet50 encoder with pre-trained weights and input the current frame and adjacent frames , extracting the feature representation of each image respectively; ResNet50 starts with a 7x7 convolutional layer with a stride of 2 and 64 filters, followed by a batch normalization layer (BN layer) and a ReLU activation layer, followed by a 3x3 maximum pooling layer with a stride of 2; the initial image size is 224×224, and after this stage the size becomes 56×56 pixels; In the cross-frame non-local attention module, the center feature is used As a query tensor , adjacent frame features { } as the key tensor and value tensor ; For each pixel in the query tensor, only non-local attention is performed on its spatial neighbor regions in the temporal neighbor frames; the dimension The K time slices of are expanded into 9×9 patches in the spatial domain; therefore, for each pixel There is a query vector , the vector corresponding to key becomes , the corresponding matrix becomes 1×81; Each row of is different; each Only the spatial neighborhood of a specific pixel in the current frame is represented; by superimposing the , we get the final correlation matrix Γ. For each row in Γ, we select only the largest entries: , ;in is the attention score, yes Column coordinates in ; and They form one-hot attention results respectively and , ; After this operation, , which is transformed into As the output of the cross-frame non-local attention module; The memory enhancement module contains a global memory block As the parameters of the network, non-local attention is used to query the current features in the global memory block M. ; Similar to the cross-frame non-local attention module, , which is transformed into As the output of the memory enhancement module; Will and After a convolution operation, the sum is performed. The above process is expressed as: , and then sent to the main decoder.

4. The method for segmenting inconspicuous lesion areas in medical videos according to claim 3, characterized in that: In step S23, the main network is divided into four stages, each of which contains a different number of residual blocks: The first stage: contains 3 residual blocks, each of which has two convolutional layers. The first is a 1x1 convolutional layer with 64 filters for dimensionality reduction; the second is a 3x3 convolutional layer with 64 filters. There is also a BN layer and a ReLU activation layer between these two convolutional layers. The output of each residual block is dimensionally matched through a 1x1 convolutional layer with 256 filters, and then added to the input. After processing in this stage, the image output is still 56×56, but the number of channels is increased from 64 to 256. The second stage: contains 4 residual blocks, each residual block has three convolutional layers. The first 1x1 convolutional layer has 128 filters, the second 3x3 convolutional layer also has 128 filters, and the third 1x1 convolutional layer has 256 filters. The first convolutional layer is followed by a BN layer and a ReLU activation layer, and the second convolutional layer is also followed by a BN layer and a ReLU activation layer. After the second stage, the image channels become 512 and the size becomes 28×28. The third stage: contains 6 residual blocks, each of which also has three convolutional layers. The first 1x1 convolutional layer has 256 filters, the second 3x3 convolutional layer also has 256 filters, and the third 1x1 convolutional layer has 512 filters. BN layers and ReLU activation layers are also interspersed between these convolutional layers. At this time, the number of image channels becomes 1024 and the size becomes 14×14. The fourth stage: contains 3 residual blocks, each residual block has three convolutional layers, the first 1x1 convolutional layer has 512 filters, the second 3x3 convolutional layer also has 512 filters, and the third 1x1 convolutional layer has 1024 filters; there are also BN layers and ReLU activation layers between these convolutional layers; at the beginning of each stage, in order to reduce the size of the feature map, the first 1x1 convolutional layer in the first residual block has a stride of 2, and an additional 1x1 convolutional layer with a stride of 2 is used to process the input to match the dimension; finally, the number of image channels becomes 2048 and the size becomes 7×7.

5. The method for segmenting inconspicuous lesion areas in medical videos according to claim 1, characterized in that: The step S3 comprises: S31, the main decoder consists of a multi-level feature fusion network; S32, the multi-level feature fusion network uses three highest-level features To perform adjacent fusion, specifically, first, the three layers of the highest level features extracted by the backbone network , perform 3x3 convolution operation to reduce the number of channels to 32; then use average pooling GAP and point convolution PWC to convolution the features Realize channel modulation and achieve channel semantic alignment ; Then, the high-level features are used to perform cross-layer semantic enhancement on the low-level features while ensuring the semantic consistency within the layer. The specific method is to Upsample twice and then compare with the feature Multiply them to get P, and do the same for the features Upsample twice and then compare with the feature Multiply them to get Q, then upsample P by two times and multiply it with Q to get R, and then add the feature Perform four times upsampling and then splice it with the obtained R, then go through 3×3 convolution and one Sigmoid function to get the initial segmentation map, which is passed into the progressive optimization network to optimize the segmentation result; the process is expressed as: in represents element-wise multiplication, represents a 3×3 convolutional block, Indicates two times upsampling.

6. The method for segmenting inconspicuous lesion areas in medical videos according to claim 1, characterized in that: The step S4 comprises: A progressive segmentation optimization network is introduced to capture the details of the lesions in the low-level feature maps and further optimize the segmentation results; First, use high-level predictions as a foreground guide After the reverse operation, the background guide map is obtained , a progressive iterative approach is used to guide the foreground and background respectively; Expand the number of channels of the current feature { } for i ∈ {1, 2, 3} through 1×1 convolution, and divide it into G groups along the channel dimension. Then, perform interpolation on the foreground mapping diagram and the background mapping diagram between the segmented features to obtain the feature . The first group g1 is further divided into three feature sets through a convolutional layer; among them, is used for iteration with the next group, and the other two are used for channel modulation; for the j-th group, 1 < j < G groups, connect the feature with that of the previous group along the channel dimension, followed by a convolutional layer and a segmentation operation. Similarly, divide this feature group into three feature sets. The output of the group only contains and for channel modulation. After iteration, connect and convert the feature through the global average pooling GAP and the pointwise convolution PWC layer into a feature modulation vector to weight the feature . Add the weighted feature to the input feature , and then obtain the foreground-guided feature and the background-guided feature through a convolutional layer; The multi-scale context-aware block consists of four branches; in each branch, 1×1 convolution is first used to reduce the channel, and in order to improve the inference efficiency without reducing the representation ability, the multi-scale context-aware block is composed of four branches. and convolution to capture multi-scale contextual information; followed by a 3×3 dilated convolution with a dilation rate of Output , i∈{1,2,3} will be sent to the next branch for further processing in a larger receptive field; Will , i∈{1,2,3,4} are set to 1,3,5,7 respectively, , i∈{1, 2, 3, 4} is set to 1, 2, 4, 8 respectively; the output of each branch is fused through a 1×1 convolution to obtain the final false positive feature , false negative features , and then correct the current features to obtain the corrected features, and further obtain a more accurate segmentation map.

7. The method for segmenting inconspicuous lesion areas in medical videos according to claim 1, characterized in that: The step S5 comprises: S51. Build a fully supervised neural network for segmentation of inconspicuous lesions, use the Adam optimizer, and set the initial learning rate to , reduced by 0.5 every 20 epochs; the input is a sequence of images of areas with inconspicuous lesions. After passing through the memory alignment network, the spatiotemporal information in the sequence is fully acquired, and the acquired spatiotemporal order is passed to the main decoder to output the segmentation result Mask; S52. During the training process, a 5-fold cross validation method will be used to evaluate the performance of the model. Assuming that the training set D contains labeled data, the training set D is evenly divided into 5 subsets, one of which is used as a validation set each time, and the remaining 4 subsets are used for training. S53, using a combination of cross entropy loss function and Dice loss function to guide the network training process to better achieve convergence effect; The loss function L of the neural network for inconspicuous lesion images is set as: is the cross entropy loss, which is used to measure the difference between the predicted value and the true value; is the true value, is the predicted value Dice loss is used to measure the similarity between two samples; is the predicted probability of the ith pixel in the predicted segmentation map, is the true label of the i-th pixel in the true segmentation map; Dice is used to deal with the imbalance of positive and negative samples; S54, training the model through the loss function, adjusting the parameters through back propagation, and selecting the model parameters with the highest accuracy in the test set for saving; S55. Select evaluation indicators including Dice coefficient, IOU index, Accuracy, Recall index, Precision index, Specificity coefficient, S-Measure index, and MAE index to evaluate model performance.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 6.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 6.

Images