Text reference based cross-modality medical video segmentation method and system
By constructing a cross-modal medical video segmentation model, the problems of inter-frame temporal dependency and cross-domain adaptability in medical video segmentation are solved, achieving higher accuracy in medical video segmentation and diagnostic assistance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN UNIV OF TECH
- Filing Date
- 2025-11-07
- Publication Date
- 2026-07-24
AI Technical Summary
Existing medical video segmentation methods fail to effectively capture global temporal dependencies between video frames and do not consider cross-domain adaptability, affecting segmentation performance and leading to inconsistent diagnoses and strain on medical resources.
A text-reference-based cross-modal medical video segmentation method is adopted. By constructing a cross-modal medical video segmentation model, including an encoding module, a cross-modal temporal context aggregation module, and a decoding module, the model is iteratively trained using a text-medical video segmentation dataset to enhance its ability to understand dynamic features and improve segmentation accuracy.
It achieves deep dynamic correlation of medical videos, outputs representative keyframes, improves the generalization and segmentation accuracy of the model, and provides doctors with more accurate diagnostic basis.
Smart Images
Figure CN121564006B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of medical video segmentation and computer vision technology, and in particular to a cross-modal medical video segmentation method and system based on text reference. Background Technology
[0002] In recent years, medical video segmentation has received widespread attention as a core supporting technology for clinical applications such as disease identification, lesion localization, disease severity classification, and real-time surgical navigation. This task aims to accurately segment regions of interest, pathological structures, or lesion areas from video data, providing crucial information for subsequent disease identification, localization, and real-time surgical navigation.
[0003] In the medical field, although there is a wealth of medical video data, traditional manual analysis relies on experienced professional doctors. However, the training of such doctors is a long process, which leads to a shortage of professional talents and a strain on medical resources. Furthermore, due to differences in doctors' knowledge and experience, as well as fatigue from frame-by-frame analysis, inconsistencies in diagnosis and misdiagnosis are prone to occur. At the same time, the research focus of a large amount of medical video data is still concentrated on static medical image analysis.
[0004] To address the above issues, many works have proposed deep learning segmentation methods, but most have failed to effectively capture the global temporal dependencies between video frames and have only roughly optimized the model for specific organs or modalities without considering cross-domain adaptability, ultimately affecting segmentation performance. Therefore, how to fully explore the global temporal relationships between frames, enhance the model's ability to understand dynamic features, and improve the model's generalization and segmentation accuracy across different medical video domains has become a key task in medical video segmentation. Summary of the Invention
[0005] This invention provides a text-reference-based cross-modal medical video segmentation method and system to solve the problem of how to segment medical videos to obtain representative keyframes, so as to provide doctors with more accurate diagnostic information.
[0006] The first aspect of this invention provides a text-referenced cross-modal medical video segmentation method, comprising: collecting cross-domain medical video and text annotation data to construct a text-medical video segmentation dataset; constructing a text-referenced cross-modal medical video segmentation model, the cross-modal medical video segmentation model comprising: an encoding module, a cross-modal temporal context aggregation module, and a decoding module; iteratively training the cross-modal medical video segmentation model using the text-medical video segmentation dataset until the training completion condition is met; the trained cross-modal medical video segmentation model can output segmented medical videos based on input medical video and text information.
[0007] In some implementations, the encoding module includes a text encoder, a query encoder, and a mask encoder; the text encoder employs a RoBERTa model based on a bidirectional Transformer architecture to extract semantic features of each word from the expression of multiple words in the input text. ,in, L The number of words in the language expression. C The feature dimension is defined as follows: The query encoder, for the input video frames, employs an improved ResNet-50 architecture that removes the classification head and the last convolutional stage. It is connected to the backbone network through two independent projection heads consisting of 3x3 convolutional layers, outputting query key feature maps respectively. and Query feature mapping ,in, and These represent the height and width of the feature map, respectively. and These represent the query key feature channel dimension and the query feature channel dimension, respectively; the mask encoder is used to map the query key features to the input prediction frame using a ResNet-18 backbone network. After the Query feature map Q is fused with the features of the last layer of the mask encoder, deep feature extraction is performed through two residual blocks and a convolutional attention module, finally outputting the mask value feature. ,in, This represents the dimension of the mask's Value feature channel.
[0008] In some implementations, the cross-modal temporal context aggregation module includes a visual-language fusion module, a segmentation selection module, and a memory storage module; wherein, the visual-language fusion module is used to process the semantic features. Multi-scale cross-modal fusion is performed to obtain cross-modal attention features. The segmentation selection module dynamically selects the optimal frame as the initial segmentation reference by predicting the segmentation confidence score of each frame; the memory storage module is used to store the memory key features of the memory frames respectively. and memory Value features Where T is the number of memory frames, it is also used to calculate the affinity matrix through a non-local matching mechanism, and then the affinity matrix is compared with the memory value feature. Combined with generated readout features .
[0009] In some implementations, the vision-language fusion module is used to construct a three-layer pyramid with a spatial span ranging from 4x to 16x downsampling for visual features in the last three stages of the query encoder. , , Language features of text encoders Perform multi-scale cross-modal fusion to obtain cross-modal attention features. .
[0010] In some implementations, the language features are... and n Visual features obtained by downsampling The process of performing multi-scale cross-modal fusion includes:
[0011] visual features and language features Using self-attention mechanisms, the two self-attention modules map the corresponding features to a common dimensional space for both modalities. This yields the joint cross-modal features corresponding to each spatial location and each word. ; for cross-modal features Applying a self-attention mechanism, three independent sets of two-dimensional convolutions are used to generate their corresponding key matrix k, query matrix q, and value matrix v. Cross-modal attention features with a spatial span downsampled by n times are calculated by estimating the correlation between visual features and word features. .
[0012] In some implementations, the step of dynamically selecting the optimal frame as the initial segmentation reference by predicting the segmentation confidence score of each frame includes: for a set of input video frame features, the segmentation selection module first reduces the number of channels to half of the original features using a 3×3 convolutional layer, and then feeds it into a fully connected layer to output the segmentation score of each frame. The optimal frame is dynamically selected by maximizing the sum of the segmentation scores of each frame.
[0013] In some implementations, the affinity matrix is calculated using a nonlocal matching mechanism and then compared with the memory value features. Combined with generated readout features The calculation formula includes:
[0014]
[0015] in, and These represent the first and second feature maps of the memory key and query key, respectively. Feature vectors at each position Indicates L2 similarity, This represents matrix multiplication.
[0016] In some embodiments, the decoding module is used to read the features. The query feature map output by the query encoder and cross-modal attention features As input, the use of historical frame information is optimized through a channel spatial attention mechanism, dynamically balancing the feature weights of historical frames and query frames, and employing a progressive fusion strategy to generate more discriminative feature representations, thereby obtaining the final prediction result. .
[0017] In some implementations, iteratively training the cross-modal medical video segmentation model using the text-medical video segmentation dataset until training completion is achieved includes: sequentially inputting the text-medical video segmentation dataset into the cross-modal medical video segmentation model; calculating the model's total loss function based on the output of the cross-modal medical video segmentation model and the labeled content of the text-medical video segmentation dataset, wherein the formula for calculating the model's total loss function is:
[0018]
[0019] in, , , These are the segmentation score loss, segmentation supervision loss, and temporal comparison loss, respectively. , , It is the coefficient that balances the three loss terms. Indicates the first The prediction results of the frame; the total loss function of the model is backpropagated, and the weights are optimized by the selected optimizer and corresponding parameters until the total loss function of the model continuously converges to a range less than a preset threshold.
[0020] A second aspect of this invention provides a text-reference-based cross-modal medical video segmentation system. The system includes: a data collection module for collecting cross-domain medical video and text annotation data to construct a text-medical video segmentation dataset; a model construction module for constructing a text-reference-based cross-modal medical video segmentation model, the model including: an encoding module, a cross-modal temporal context aggregation module, and a decoding module; and a training module for iteratively training the model using the text-medical video segmentation dataset until training completion is achieved. After training, the cross-modal medical video segmentation model can output segmented medical videos based on input medical video and text information.
[0021] By training a text-referenced cross-modal medical video segmentation model, the resulting model can deeply and dynamically correlate text and medical video, and segment representative keyframes based on temporal features. This enables doctors to provide more accurate diagnoses based on dynamic images. Moreover, since the training data is not limited to single-organ videos, the trained model exhibits high generalization ability. Attached Figure Description
[0022] Figure 1 A flowchart illustrating a text-reference-based cross-modal medical video segmentation method provided in an embodiment of the present invention;
[0023] Figure 2 A model architecture diagram for a text-reference-based cross-modal medical video segmentation model;
[0024] Figure 3 The flowchart shows the calculation process for cross-modal attention features downsampled by n times.
[0025] Figure 4 This is a schematic diagram of the structure of a text-reference-based cross-modal medical video segmentation system provided in an embodiment of the present invention. Detailed Implementation
[0026] The specific technical features described in the various embodiments in the detailed implementation can be combined in various ways without contradiction. For example, different implementation methods can be formed by combining different specific technical features. In order to avoid unnecessary repetition, the various possible combinations of the specific technical features in this invention will not be described separately.
[0027] It should also be noted that, in order to avoid obscuring the present invention with unnecessary details, only the structures and / or processing steps closely related to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.
[0028] Additionally, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the following description, the terms "first," "second," etc., are used merely to distinguish different objects and do not indicate any similarity or connection between them. It should be understood that the directional descriptions such as "above," "below," "inside," and "outside" refer to the orientation under normal use conditions.
[0029] In the following specific implementation, the text-reference-based cross-modal medical video segmentation method is used to segment medical videos into keyframes that facilitate diagnosis by doctors based on text descriptions. This segmentation method is only used to provide doctors with auxiliary references for diagnosis and is not a disease diagnosis method. The steps of the text-reference-based cross-modal medical video segmentation method are illustrated below with reference to embodiments.
[0030] In some embodiments, such as Figure 1 As shown, Figure 1 The following is a flowchart illustrating a text-reference-based cross-modal medical video segmentation method according to an embodiment of the present invention. The process of the cross-modal medical video segmentation method includes:
[0031] Step S101: Collect cross-domain medical video and text annotation data to construct a text-medical video segmentation dataset.
[0032] This can be understood as constructing a segmentation dataset with text-medical video associations by collecting rich cross-domain medical video and text annotation data. This dataset is used to train subsequent segmentation models. The segmentation dataset contains labeled frames that are representative of each medical video, i.e., these are labeled data.
[0033] Step S102: Construct a cross-modal medical video segmentation model based on text reference.
[0034] Specifically, such as Figure 2 As shown, the text-reference-based cross-modal medical video segmentation model includes an encoding module, a cross-modal temporal context aggregation module, and a decoding module. The structure and function of each module are illustrated below.
[0035] First, the encoding module is explained. It includes a text encoder, a query encoder, and a mask encoder. The text encoder uses a RoBERTa model based on a bidirectional Transformer architecture to extract semantic features for each word from an expression of L words in the input text. ,in, L The number of words in the language expression. C For feature dimensions;
[0036] The query encoder, for input video frames, employs an improved ResNet-50 architecture that removes the classification header and the last convolutional stage, encoding via two convolutional kernels of size [size missing]. The independent projection heads, composed of convolutional layers, are connected to the backbone network and output query key feature maps respectively. and Query feature mapping ,in, and These represent the height and width of the feature map, respectively. and These represent the Key feature channel dimension and the Query feature channel dimension, respectively. This can be understood as the query Key feature mapping. and Query feature mapping Each feature map can be represented as a feature map, with its height and width denoted as H and W, respectively. The feature channel dimensions of each pixel in the feature map are as follows: and The feature channel dimension can represent the color of the feature map. For example, if the feature channel dimension is 1, the feature map is a grayscale image, and if the feature channel dimension is 3, the feature map is a color image represented by the three primary colors.
[0037] A mask encoder, using ResNet-18 as the backbone network, maps the query key features to the input prediction frame (the keyframe labeled in the training data). After fusing the Query feature map Q with the features of the last layer of the mask encoder, deep feature extraction is performed through two residuals and a convolutional attention module, finally outputting the mask value feature. ,in, This represents the dimension of the mask's Value feature channel.
[0038] The following describes the cross-modal temporal context aggregation module, which includes a visual-language fusion module, a segmentation selection module, and a memory storage module. The visual-language fusion module constructs a three-layer pyramid with a spatial span ranging from 4x to 16x downsampling to process the visual features of the last three stages of the query encoder. , , Language features of text encoders Perform multi-scale cross-modal fusion to obtain cross-modal attention features. To more clearly illustrate the process of calculating visual features through downsampling, the following section combines... Figure 3 Visual features obtained by downsampling by n times and language features The process of multi-scale cross-modal fusion is illustrated by an example, where T represents the time dimension, i.e., the input T frames. , , These represent the number of feature channels, feature height, and feature width, respectively, downsampled by a factor of n.
[0039] Step 1: Visual features and language features Using self-attention mechanisms, the two self-attention modules map the corresponding features to a common dimensional space for both modalities. This yields the joint cross-modal features corresponding to each spatial location and each word. The calculation formula is as follows:
[0040]
[0041] in, Self-attention modules representing visual modalities Self-attention modules representing language modalities.
[0042] Step 2: Analyze cross-modal features By applying a self-attention mechanism, three independent sets of two-dimensional convolutions are used to generate the corresponding key matrix k, query matrix q, and value matrix v, respectively. The specific calculation is as follows:
[0043]
[0044] in, This represents the convolution operation.
[0045] The third step is to calculate cross-modal attention features by estimating the correlation between visual features and word features. The calculation formula is as follows:
[0046]
[0047] in, It is the softmax function.
[0048] The following describes the segmentation selection module within the cross-modal temporal context aggregation module. For a set of input video frame features, the segmentation selection module first reduces the number of channels to half the original feature count using a 3×3 convolutional layer, then feeds it into a fully connected layer, outputting the segmentation score for each frame. By adjusting the segmentation scores of each frame The optimal frame is dynamically selected by maximizing the sum of values. For example, during training, segmentation loss is used. To split the fraction Supervised, the true segmentation score for each frame. The calculation formula is as follows:
[0049]
[0050] in, Indicates the first video in a set of videos. frame, Indicates the first Predictive segmentation mask for frames Compared with the real segmentation mask The cross-entropy loss is used, and the higher the segmentation score S, the better the segmentation quality. The predicted segmentation mask is the output predicted segmentation mask.
[0051] The following describes the memory storage module in the cross-modal temporal context aggregation module, which is used to store the memory key features of memory frames respectively. and memory Value features Where T is the number of memory frames; the memory storage module is also used to calculate the affinity matrix through a non-local matching mechanism, and then match the affinity matrix with the memory Value feature. Combined with generated readout features The calculation formula is as follows:
[0052]
[0053] in, and These represent the first and second feature maps of the memory key and query key, respectively. Feature vectors at each position Indicates L2 similarity, This represents matrix multiplication.
[0054] The decoding module will now be introduced. The decoding module is used to process the read features. Query feature mapping output by the query encoder and cross-modal attention features As input, the use of historical frame information is optimized through a channel spatial attention mechanism, dynamically balancing the feature weights of historical frames and query frames, and employing a progressive fusion strategy to generate more discriminative feature representations, thereby obtaining the final prediction result. The specific steps are as follows:
[0055] Step 1: Query feature mapping Reading out features and cross-modal attention features Global average pooling and global max pooling are performed across spatial dimensions respectively, and the results are concatenated to obtain the aggregated spatial features. The calculation formula is as follows:
[0056]
[0057] in, and These represent global average pooling and global max pooling across spatial dimensions, respectively.
[0058] The second step is to aggregate spatial features. The channel weights are obtained by passing the input to three one-dimensional convolutions. , and Then, the normalized channel weights are obtained by applying the softmax function. , and .
[0059] The third step, similar to the calculation of channel weights, involves global average pooling and global max pooling across channel dimensions, followed by passing the aggregated channel features to three two-dimensional convolutions to obtain spatial weights. , and .
[0060] Fourth step: Add the channel weights and spatial weights using tensor broadcasting to obtain the weights of the current image features and the memory readout features. Then, combine the three weights with the input features. , and Multiply and sum to obtain the final output. The calculation formula is as follows:
[0061]
[0062] in, , .
[0063] Step S103: Iteratively train the cross-modal medical video segmentation model using the text-medical video segmentation dataset until the training completion condition is met.
[0064] The trained cross-modal medical video segmentation model can output segmented medical videos based on input medical video and text information. For example, the specific training process is as follows:
[0065] The first step is to input the text-medical video segmentation dataset into the cross-modal medical video segmentation model.
[0066] The second step involves calculating the model's total loss function based on the output of the cross-modal medical video segmentation model and the labeled content of the text-medical video segmentation dataset. The formula for calculating the model's total loss function is as follows:
[0067]
[0068] in, , , These are the segmentation score loss, segmentation supervision loss, and temporal comparison loss, respectively. , , It is the coefficient that balances the three loss terms. Indicates the first Frame prediction results, Indicates the first The calculation formula for the prediction results of each frame means that for each frame's prediction results, a segmentation score loss is calculated; simultaneously, when the prediction result is not empty, a mask loss is calculated to ensure better segmentation results. The calculation methods for the three loss terms are illustrated below:
[0069] For the segmentation fraction loss The calculation formula is as follows:
[0070]
[0071] in, It is the first Frame segmentation score, It is a balancing factor used to adjust the influence between positive and negative samples. It is the focus factor, used to adjust the weights of easy and difficult samples.
[0072] For segmented monitoring loss The calculation formula is as follows:
[0073]
[0074] in, For the first Predictive segmentation mask for frames, For the corresponding mask real label, For cross-entropy loss, Given a set of video frames, this loss only calculates the probability of a value less than a threshold. The pixel loss is used to prevent overtraining on simple samples.
[0075] For time comparison loss The calculation formula is as follows:
[0076]
[0077] in, Representing vectors and Cosine similarity between them This represents the feature of the t-th memory value in the memory bank. Add a small value to the denominator. This is to prevent division by zero errors in practice.
[0078] The third step is to backpropagate the total loss function of the model and optimize the weights in the cross-modal medical video segmentation model using the selected optimizer and response parameters. For example, the optimizer can be a gradient descent search to minimize the total loss function of the model in this training. Once the minimum value is reached, the training is considered complete. Then, new training data is input into the model in this training for the next training, that is, iterative training is performed until the total loss function of the model in multiple consecutive trainings continuously converges to a range less than a preset threshold. At this point, the model is considered to be able to accurately predict the required keyframes and the model training is considered complete.
[0079] This invention also provides a text-reference-based cross-modal medical video segmentation system, such as... Figure 4 As shown, the cross-modal medical video segmentation system includes a data collection module 100, a model building module 200, and a training module 300. The data collection module 100 collects cross-domain medical video and text annotation data to construct a text-medical video segmentation dataset. The model building module 200 constructs a text-referenced cross-modal medical video segmentation model, which includes an encoding module, a cross-modal temporal context aggregation module, and a decoding module. The training module 300 iteratively trains the cross-modal medical video segmentation model using the text-medical video segmentation dataset until the training completion condition is met. After training, the cross-modal medical video segmentation model can output segmented medical videos based on input medical video and text information.
[0080] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A cross-modal medical video segmentation method based on text reference, characterized in that, The cross-modal medical video segmentation method includes: Collect cross-domain medical video and text annotation data to construct a text-medical video segmentation dataset; A text-reference-based cross-modal medical video segmentation model is constructed, comprising an encoding module, a cross-modal temporal context aggregation module, and a decoding module. The encoding module includes a text encoder, a query encoder, and a mask encoder. The text encoder employs a RoBERTa model based on a bidirectional Transformer architecture to extract semantic features of each word from the expressions of multiple words in the input text. ,in, L The number of words in the language expression. C The feature dimension is defined as follows: The query encoder, for the input video frames, employs an improved ResNet-50 architecture that removes the classification head and the last convolutional stage. It is connected to the backbone network through two independent projection heads consisting of 3x3 convolutional layers, outputting query key feature maps respectively. and Query feature mapping ,in, and These represent the height and width of the feature map, respectively. and These represent the query key feature channel dimension and the query feature channel dimension, respectively; the mask encoder is used to map the query key features to the input prediction frame using a ResNet-18 backbone network. After the Query feature map Q is fused with the features of the last layer of the mask encoder, deep feature extraction is performed through two residual blocks and a convolutional attention module, finally outputting the mask value feature. ,in, The mask value feature channel dimension; the cross-modal temporal context aggregation module includes a visual-language fusion module, a segmentation selection module, and a memory storage module; The visual-language fusion module is used to process the semantic features. Multi-scale cross-modal fusion is performed to obtain cross-modal attention features. The segmentation selection module dynamically selects the optimal frame as the initial segmentation reference by predicting the segmentation confidence score of each frame; the memory storage module is used to store the memory key features of the memory frames respectively. and memory Value features Where T is the number of memory frames, it is also used to calculate the affinity matrix through a non-local matching mechanism, and then the affinity matrix is compared with the memory value feature. Combined with generated readout features ; The cross-modal medical video segmentation model is iteratively trained using the text-medical video segmentation dataset until the training completion condition is met. After training, the cross-modal medical video segmentation model can output segmented medical videos based on the input medical video and text information.
2. The cross-modal medical video segmentation method according to claim 1, characterized in that, The vision-language fusion module is used to construct a three-layer pyramid with a spatial span ranging from 4x to 16x downsampling, for processing the visual features of the last three stages of the query encoder. , , Language features of text encoders Perform multi-scale cross-modal fusion to obtain cross-modal attention features. .
3. The cross-modal medical video segmentation method according to claim 2, characterized in that, The language features and n Visual features obtained by downsampling The process of performing multi-scale cross-modal fusion includes: visual features and language features Using self-attention mechanisms, the two self-attention modules map the corresponding features to a common dimensional space for both modalities. This yields the joint cross-modal features corresponding to each spatial location and each word. ; For the joint cross-modal features By applying a self-attention mechanism, three independent two-dimensional convolutions are used to generate their corresponding key matrix k, query matrix q, and value matrix v. The cross-modal attention features, downsampled n times by a factor of n across the spatial span, are calculated by estimating the correlation between visual features and word features. .
4. The cross-modal medical video segmentation method according to claim 1, characterized in that, The step of dynamically selecting the optimal frame as the initial segmentation reference by predicting the segmentation confidence score of each frame includes: For a set of input video frame features, the segmentation selection module first reduces the number of channels to half of the original features using a 3×3 convolutional layer, and then feeds it into a fully connected layer to output the segmentation score for each frame. ; The optimal frame is dynamically selected by maximizing the sum of the segmentation scores of each frame.
5. The cross-modal medical video segmentation method according to claim 1, characterized in that, The affinity matrix is calculated using a nonlocal matching mechanism and then compared with the memory value features. Combined with generated readout features The calculation formula includes: in, and These represent the first and second feature maps of the memory key and query key, respectively. Feature vectors at each position Indicates L2 similarity, This represents matrix multiplication.
6. The cross-modal medical video segmentation method according to claim 3, characterized in that, The decoding module is used to read the features. The query feature map output by the query encoder and cross-modal attention features As input, the use of historical frame information is optimized through a channel spatial attention mechanism, dynamically balancing the feature weights of historical frames and query frames, and employing a progressive fusion strategy to generate more discriminative feature representations, thereby obtaining the final prediction result. .
7. The cross-modal medical video segmentation method according to claim 1, characterized in that, The cross-modal medical video segmentation model is iteratively trained using the text-medical video segmentation dataset until the training completion conditions are met, including: The text-medical video segmentation dataset is sequentially input into the cross-modal medical video segmentation model; Based on the output of the cross-modal medical video segmentation model and the labeled content of the text-medical video segmentation dataset, the total loss function of the model is calculated. The formula for calculating the total loss function is as follows: in, , , These are the segmentation score loss, segmentation supervision loss, and temporal comparison loss, respectively. , , It is the coefficient that balances the three loss terms. Indicates the first Frame prediction results; The total loss function of the model is backpropagated, and the weights are optimized using the selected optimizer and corresponding parameters until the total loss function of the model continuously converges to a range less than a preset threshold.
8. A text-reference-based cross-modal medical video segmentation system, characterized in that, The cross-modal medical video segmentation system includes: The data collection module is used to collect cross-domain medical video and text annotation data to construct a text-medical video segmentation dataset; The model building module is used to construct a text-reference-based cross-modal medical video segmentation model. This model includes an encoding module, a cross-modal temporal context aggregation module, and a decoding module. The encoding module comprises a text encoder, a query encoder, and a mask encoder. The text encoder uses a RoBERTa model based on a bidirectional Transformer architecture to extract semantic features of each word from the expressions of multiple words in the input text. ,in, L The number of words in the language expression. C The feature dimension is defined as follows: The query encoder, for the input video frames, employs an improved ResNet-50 architecture that removes the classification head and the last convolutional stage. It is connected to the backbone network through two independent projection heads consisting of 3x3 convolutional layers, outputting query key feature maps respectively. and Query feature mapping ,in, and These represent the height and width of the feature map, respectively. and These represent the query key feature channel dimension and the query feature channel dimension, respectively; the mask encoder is used to map the query key features to the input prediction frame using a ResNet-18 backbone network. After the Query feature map Q is fused with the features of the last layer of the mask encoder, deep feature extraction is performed through two residual blocks and a convolutional attention module, finally outputting the mask value feature. ,in, The mask value feature channel dimension; the cross-modal temporal context aggregation module includes a visual-language fusion module, a segmentation selection module, and a memory storage module; The visual-language fusion module is used to process the semantic features. Multi-scale cross-modal fusion is performed to obtain cross-modal attention features. The segmentation selection module dynamically selects the optimal frame as the initial segmentation reference by predicting the segmentation confidence score of each frame; the memory storage module is used to store the memory key features of the memory frames respectively. and memory Value features Where T is the number of memory frames, it is also used to calculate the affinity matrix through a non-local matching mechanism, and then the affinity matrix is compared with the memory value feature. Combined with generated readout features ; The training module is used to iteratively train the cross-modal medical video segmentation model using the text-medical video segmentation dataset until the training completion condition is met. After training, the cross-modal medical video segmentation model can output segmented medical videos based on the input medical video and text information.