Lesion segmentation method, device, medium and product based on correlation information between images

Abstract

The present application discloses a lesion segmentation method, device, medium and product based on correlation information between images, which relates to the field of image processing. The method includes obtaining a tumor slice sequence; inputting the tumor slice sequence into a trained Mesh‑SSM UNet sequence segmentation model to obtain a sequence of lesion area segmentation results; the Mesh‑SSM UNet sequence segmentation model adopts an improved UNet network model; in the improved UNet network model, the visual feature extraction unit of each encoding layer of the encoder adopts a 2D selective scanning module and a mesh propagation optional scanning module group connected in series, and the visual feature extraction unit of each decoding layer of the decoder adopts a 2D selective scanning module; the mesh propagation optional scanning module group is used to extract correlation features between image sequences from the input image sequence. The lesion segmentation method based on the present application can improve the segmentation accuracy of the lesion area.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device, medium and product for segmenting a lesion based on correlation information between images. Background Art

[0002] Medical image segmentation technology provides doctors with important visual information by segmenting key areas in the image from the background, which helps to accurately label and analyze disease status. Brain tumor segmentation is a challenging task. Its goal is to accurately extract and label the intracranial tumor area from brain MRI images. Due to the high mortality rate and difficult-to-cure characteristics of brain tumors, accurate positioning of the lesion area is crucial for subsequent treatment. The rapid development of computer-aided diagnosis technology enables doctors to more accurately diagnose diseases and tumor conditions with the help of computer image analysis capabilities. However, the irregular boundaries, variable positions and complex textures of brain tumors, as well as the uneven grayscale, low inter-class contrast, and blurred edges in MRI, make achieving stable and accurate segmentation performance a widely discussed topic in current research.

[0003] With the development of deep learning technology, especially the widespread application of convolutional neural networks (CNN), deep learning algorithms have gradually become the mainstream of MRI brain tumor image segmentation tasks with the help of the powerful feature extraction capabilities of convolutional neural networks, enabling image segmentation methods based on deep learning to use their self-learning capabilities to better learn task-related image features without manual intervention. UNet and its various variants have achieved remarkable results in the field of brain tumor image segmentation and have shown great potential.

[0004] Normally, brain tumor image data enters various models in the form of three-dimensional images or two-dimensional slices to complete the segmentation task. However, due to the continuity of spatial features such as lesion location and size between brain tumor MRI slices, the two-dimensional model is completely immersed in the features of a single image. Although the three-dimensional model can obtain a certain spatial feature extraction capability by sacrificing a large amount of computation, it is difficult to complete the specialized processing of sequential continuous information. Sequence models transplanted from the field of natural language processing, such as LSTM, Transformer, and Mamba SSM, are more often applied to internal sequences composed of single image blocks or image windows. There are still few studies that use brain tumor MRI slices as sequence input and design related algorithms. Summary of the invention

[0005] The purpose of this application is to provide a lesion segmentation method, device, medium and product based on image correlation information, which improves the accuracy of lesion area segmentation.

[0006] To achieve the above objectives, this application provides the following solutions.

[0007] In a first aspect, the present application provides a lesion segmentation method based on correlation information between images, comprising the following steps.

[0008] Obtain a tumor slice sequence to be segmented.

[0009] The tumor slice sequence to be segmented is input into the trained Mesh-SSM UNet sequence segmentation model to obtain a sequence of lesion area segmentation results; the Mesh-SSM UNet sequence segmentation model adopts an improved UNet network model; in the improved UNet network model, the visual feature extraction unit in each encoding layer of the encoder adopts a 2D selective scanning module and a mesh propagation optional scanning module group connected in series, and the visual feature extraction unit in each decoding layer of the decoder adopts a 2D selective scanning module; the mesh propagation optional scanning module group is used to extract correlation features between image sequences from the image sequence output by the corresponding 2D selective scanning module; the tumor slice sequence to be segmented is input into the first encoding layer of the encoder, and the lesion area segmentation result sequence is output by the last decoding layer of the decoder.

[0010] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned lesion segmentation method based on inter-image correlation information.

[0011] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned lesion segmentation method based on correlation information between images.

[0012] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned lesion segmentation method based on inter-image correlation information.

[0013] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0014] The present application provides a method, device, medium and product for lesion segmentation based on correlation information between images, wherein the method includes obtaining a tumor slice sequence; inputting the tumor slice sequence into a trained Mesh-SSM UNet sequence segmentation model to obtain a sequence of lesion area segmentation results; the Mesh-SSM UNet sequence segmentation model adopts an improved UNet network model; in the improved UNet network model, the visual feature extraction unit in each encoding layer of the encoder adopts a 2D selective scanning module and a mesh propagation optional scanning module group connected in series, and the visual feature extraction unit in each decoding layer of the decoder adopts a 2D selective scanning module; the mesh propagation optional scanning module group is used to extract correlation features between image sequences for the image sequence output by the corresponding 2D selective scanning module. In the present application, the tumor slice sequence is used as the input data of the segmentation model, and a Mesh-SSM UNet sequence segmentation model is designed, which can learn sequence correlation features from sequences and features, thereby improving the segmentation accuracy of the lesion area. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 This is a diagram of the application environment of a lesion segmentation method based on correlation information between images in one embodiment of the present application.

[0017] Figure 2 A flowchart of a lesion segmentation method based on inter-image correlation information provided in one embodiment of the present application.

[0018] Figure 3 A schematic diagram of the structure of the Mesh-SSM UNet sequence segmentation model provided in one embodiment of the present application.

[0019] Figure 4 A structural diagram of a mesh-cast optional scanning module group (Mesh-Cast SSMGroup) provided in one embodiment of the present application.

[0020] Figure 5 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0023] The lesion segmentation method based on the correlation information between images provided in the embodiment of the present application is specifically a lesion segmentation method based on the correlation information between medical sequence images, which can be applied to Figure 1 In the application environment shown. Among them, the terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be set up separately, integrated on the server, or placed on the cloud or other servers. The terminal can send the tumor slice sequence to be segmented to the server. After the server receives the tumor slice sequence to be segmented, the server inputs the tumor slice sequence to be processed into the trained Mesh-SSM UNet sequence segmentation model to obtain the lesion area segmentation result sequence; the Mesh-SSM UNet sequence segmentation model adopts an improved UNet network model; in the improved UNet network model, the visual feature extraction unit in each encoding layer of the encoder adopts a 2D selective scanning module and a mesh propagation optional scanning module group connected in series, and the visual feature extraction unit in each decoding layer of the decoder adopts a 2D selective scanning module; the mesh propagation optional scanning module group is used to extract the correlation features between the image sequences output by the corresponding 2D selective scanning module. The server can feed back the obtained lesion area segmentation result sequence to the terminal. In addition, in some embodiments, the lesion segmentation method based on the correlation information between images can also be implemented independently by a server or a terminal. For example, the terminal can directly perform lesion segmentation processing on the tumor slice sequence to be segmented, or the server can obtain the tumor slice sequence to be segmented from the data storage system and perform lesion segmentation processing.

[0024] The terminal may be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The server may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0025] In an exemplary embodiment, Figure 2 and Figure 3As shown, a lesion segmentation method based on image correlation information is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server in is taken as an example to illustrate, including the following steps 101 to 102.

[0026] Step 101, obtaining a tumor slice sequence to be segmented (ie, a tumor slice sequence to be segmented). The tumor slice sequence to be segmented is a two-dimensional slice sequence obtained by slicing a three-dimensional tumor image using existing slicing methods.

[0027] Because medical images are three-dimensional storage expressions, most of the current brain tumor segmentation algorithms use random division after slicing, and there is no certain sequential association between these slices. In this application, a sequential slicing method is used to retain the sequential adjacent association between images, so that there is spatial similarity between the two slices, such as the continuity of the lesion area.

[0028] Step 102: input the tumor slice sequence to be segmented into the trained Mesh-SSM UNet sequence segmentation model to obtain a lesion area segmentation result sequence.

[0029] The Mesh-SSM UNet sequence segmentation model (Mesh Selective Scanning Module UNetSequential Segmentation Model) mainly includes a visual sequence channel feature state space module group (Mesh-Cast SSM Group) and a UNet skeleton that uses the 2D selective scanning module (SS2D module for short, also called SS2D visual feature extraction module) in V-Mamba as a traditional visual module. Among them, the visual sequence channel feature state space module group can also be called a mesh propagation selectivity scanning module group. Furthermore, the Mesh-SSM UNet sequence segmentation model uses the encoder-decoder structure as the basic framework and uses the SS2D module as the visual feature extraction unit in each layer. After extracting the two-dimensional visual features, the Mesh-Cast SSM Group is used in each layer of the encoder to extract sequence features. Therefore, the Mesh-SSM UNet sequence segmentation model adopts an improved UNet network model; in the improved UNet network model, the visual feature extraction units in each encoding layer of the encoder and in each decoding layer of the decoder all adopt SS2D modules; the output end of the SS2D module of each encoding layer is also connected to a mesh-cast optional scanning module group (Mesh-Cast SSM Group). In the traditional UNet network model, the visual feature extraction units in each encoding layer and decoding layer adopt convolution modules. In this application, for the visual feature extraction unit in the encoding layer, the convolution module is replaced by a structure composed of SS2D module + mesh-cast optional scanning module group (Mesh-Cast SSM Group), and for the visual feature extraction unit in the decoding layer, the convolution module is replaced by an SS2D module. No changes have been made to other structures of the encoder and decoder, such as the jump connection method between the encoding layer and the decoding layer.

[0030] The mesh-cast optional scanning module group (Mesh-Cast SSM Group) has the ability to process multi-channel image sequences, and is used to extract correlation features between image sequences output by the corresponding SS2D module. The corresponding SS2D module here refers to the SS2D module connected to the mesh-cast optional scanning module group.

[0031] The tumor slice sequence to be segmented is input into the first encoding layer of the encoder, and the lesion area segmentation result sequence is output by the last decoding layer of the decoder. One tumor slice in the tumor slice sequence to be segmented corresponds to one lesion area segmentation result. The tumor slice sequence to be segmented is a multimodal MRI slice sequence or a multimodal CT slice sequence, which can be expressed as ,in is a single multimodal MRI slice or a single multimodal CT slice, and T is the number of slices.

[0032] The number of encoding layers in the encoder and decoding layers in the decoder can be set as required.

[0033] In this application, considering that most studies have ignored the sequence association in brain tumor slices, the existing segmentation methods still have limitations in the task of accurately segmenting brain tumor lesions. In this application, a sequence segmentation model suitable for brain tumor data is designed, namely the Mesh-SSM UNet sequence segmentation model, which breaks through the processing limitations of two-dimensional slice images or three-dimensional images and takes brain tumor slice sequences as input. In the segmentation model, Mamba SSM is applied to the sequence and channel dimensions through mesh propagation operations, and a mesh propagation optional scanning module group (Mesh-Cast SSM Group) is designed to extract the correlation features between image sequences from the image sequence output by the corresponding SS2D module, so that the segmentation model can fully learn the sequence correlation features from the sequence and between the features, and improve the accuracy of lesion area segmentation.

[0034] In another exemplary embodiment of the present application, the Mesh-Cast SSMGroup is the core module of the Mesh-SSM UNet sequence segmentation model, which is used to enhance the feature extraction capability in the sequence and channel dimensions. As an example, Figure 4 As shown in the figure, the Mesh-Cast SSM Group is composed of four Mesh-Cast SSM modules combined by a layer-attention module. Among them, the Mesh-Cast SSM module uses the S6 module (Selective Scan Space State Sequential Model, selective scanning module) as the core operator. The S6 module is a core module in the recently proposed Mamba algorithm, which realizes dynamic feature capture between slices across sequences. The S6 module is used to perceive the spatiotemporal features between image sequences, and selectively scans slices in the sequence dimension through the state space model to capture the associated features of lesions in different slices. This module can integrate the temporal information of the sequence into the feature representation of the model and improve the ability to capture the lesion area. Among them, in the Mesh-Cast SSM Group, the selective scanning module is called the selective scanning unit (S6 unit), and the S6 unit combines the layer-attention module to enhance the feature representation ability of the network layer by layer.

[0035] In the S6 unit, the state space model is used to selectively perceive the association of sequence features. Each element interacts with any previously scanned sample by compressing the hidden state, so that the model understands the common association between frames in the sequence. The calculation process is defined as follows.

[0036] First, enter the sequence is mapped to a hidden state space and processed in the form of a state space model. Initialize to zero vector or other set initial state.

[0037] During the state update process, the S6 unit uses the state space model (SSM) for calculation. The basic state update equation is as follows.

[0038] ;

[0039] .

[0040] in: is the state transfer matrix, which represents the change of hidden state over time. is the input influence matrix, which is the input Projected into the hidden state space. is the output matrix, and the hidden state Convert to output. Is a direct influence matrix, directly acting on the input To affect the output These matrix parameters are It is acquired through learning.

[0041] In order to improve computational efficiency, the S6 unit effectively reduces the computational complexity through a selective scanning mechanism, enabling it to achieve linear complexity calculations without losing the global receptive field.

[0042] Based on the above, if Figure 4 As shown, the mesh-cast optional scanning module group (Mesh-Cast SSMGroup) includes a layer attention mechanism module and a plurality of Mesh-Cast SSM modules connected in series; the output end of each Mesh-CastSSM module is also connected to the input end of the layer attention mechanism module.

[0043] The Mesh-Cast SSM module is used to selectively perceive the image sequence output by the corresponding SS2D module in the sequence and channel dimensions to perceive the sequence changes of tumor lesions. The corresponding SS2D module refers to the SS2D module connected to the Mesh-Cast SSM module.

[0044] The layer attention mechanism module is used to fuse the output of each Mesh-Cast SSM module to obtain a fused feature.

[0045] In another exemplary embodiment of the present application, a mesh-cast selective scanning module group (Mesh-Cast SSMGroup) is used to enhance feature extraction capabilities in the sequence and channel dimensions, which mainly includes a Mesh-Cast SSM module using a Mesh-Cast algorithm and an S6 algorithm, and its purpose is to selectively perceive a multi-channel MRI (or CT) sequence in the sequence and channel dimensions, and focus the model's attention on the sequence changes of the lesion, such as Figure 4 Take a Mesh-Cast SSM module as an example. First, input the frame number of the image sequence As the sequence length, the number of channels As the model dimension, the first selective scanning unit S6 is input for attention perception processing. Here, the input image sequence of the first selective scanning unit S6 is It can be expressed as the following formula.

[0046] .

[0047] The output generated by the first selective scanning unit S6 is expressed as follows.

[0048] .

[0049] Afterwards, The number of input channels in As the sequence length, the number of frames As the model dimension, the reorganized sequence is expressed as follows.

[0050] .

[0051] Next, the reorganized sequence is input into the second selective scanning unit S6 for attention perception processing, and the output is expressed as follows.

[0052] .

[0053] Finally, the output Reorganize into the initial dimension (i.e., in terms of the number of frames) As the sequence length, the number of channels As the model dimension), the perception output that has been selectively scanned in both the sequence and channel dimensions is obtained and input into the next Mesh-Cast SSM module in series.

[0054] Based on the above content, each of the Mesh-Cast SSM modules includes a first selective scanning unit (first S6 unit), a first Mesh-Cast unit, a second selective scanning unit (second S6 unit) and a second Mesh-Cast unit which are connected in sequence.

[0055] The first selective scanning unit is used to perform attention perception processing on the corresponding input image sequence (i.e., the image sequence input to the first selective scanning unit) in the sequence and channel dimensions to obtain a first perception feature sequence (corresponding to ); In the input image sequence of the first selective scanning unit, the number of frames is used as the sequence length and the number of channels is used as the model dimension.

[0056] The first Mesh-Cast unit is used to use the channel dimension as the sequence length and the number of frames as the model dimension to interchange the sequence and channel dimensions of the first perceptual feature sequence to obtain a first reorganized sequence (corresponding to ).

[0057] The second selective scanning unit is used to perform attention perception on the first reorganized sequence in the sequence and channel dimensions to obtain a second perception feature sequence (corresponding to ).

[0058] The second Mesh-Cast unit is used to use the number of frames as the sequence length and the number of channels as the model dimension, and to interchange and reorganize the sequence and channel dimensions of the second perceptual feature sequence to obtain a second reorganized sequence.

[0059] Each Mesh-Cast SSM module also includes a Mesh-Cast structure (ie, a first Mesh-Cast unit and a second Mesh-Cast unit) for dimensional reorganization and linking.

[0060] In another exemplary embodiment of the present application, the outputs of multiple (e.g., 4) Mesh-Cast SSM modules in the Mesh-Cast SSMGroup are connected in series to obtain summary information of the lesion in the sequence and channel dimensions, and enhance the information as the number of layers increases. In order to maintain the balance between sequence, channel, and 2D semantic features and obtain the most representative features, a hierarchical attention mechanism is used to obtain the most balanced output features that are directly multiplied by the input information, and the output features of the remaining layers are added as auxiliary, such as Figure 4 As shown. The specific calculation process of the hierarchical attention mechanism module is as follows:

[0061] Set up The input of a Mesh-Cast SSM module is , the output is The series relationship can be expressed as follows.

[0062]

[0063] in, Indicates that the output of the previous Mesh-Cast SSM module serves as the input of the next Mesh-Cast SSM module.

[0064] Take 4 Mesh-Cast SSM modules as an example. For the first Mesh-Cast SSM module, there are , which is the initial input data. The output of the last Mesh-Cast SSM module It will contain enhanced feature information after being processed by 4 layers of Mesh-Cast SSM modules. As the number of layers increases, the structure of the Mesh-Cast SSMGroup allows the model to gradually enhance its ability to capture the features of lesions in the input data.

[0065] The output characteristics of the Mesh-Cast SSM module for each layer are m= 1, 2, 3, 4), the weights of the features at each layer are calculated through the SE-based hierarchical attention mechanism , to measure the importance of each layer of features.

[0066] .

[0067] Here, SE(·) represents the Squeeze-and-Excitation operation.

[0068] Then, according to the weight Select the most balanced features , the most balanced features Direct and input information Multiply, and perform weighted addition of the output features of each layer as auxiliary information. Taking 4 Mesh-Cast SSM modules as an example, assuming is the output feature with the largest weight, then the output of the hierarchical attention mechanism module is expressed as follows.

[0069] .

[0070] Among them, the output of the hierarchical attention mechanism module is the final output of the mesh-cast optional scanning module group (Mesh-Cast SSMGroup). is the weighting coefficient of the auxiliary layer features, which is usually set to a value less than 1 to avoid over-enhancing non-main features. The output of the hierarchical attention mechanism module It integrates features of different dimensions and ensures the balance between sequence, channel and 2D semantic features through a hierarchical attention mechanism.

[0071] Based on the above content, in the hierarchical attention mechanism module, the output of each Mesh-Cast SSM module is fused to obtain the fusion features, including:

[0072] (1) Apply the SE module-based hierarchical attention mechanism to calculate the weight of the output features of each Mesh-Cast SSM module.

[0073] (2) The output feature corresponding to the maximum weight is determined as the most balanced output feature.

[0074] (3) The output features corresponding to other weight values ​​are determined as auxiliary output features. Other weight values ​​refer to the remaining weight values ​​after removing the maximum weight value.

[0075] (4) After multiplying the most balanced output feature and the input data of the first Mesh-Cast SSM module, the resultant is added to the auxiliary output feature to obtain the fused feature.

[0076] Among them, the calculation expression of the fusion feature is expressed by the following formula.

[0077] .

[0078] In the formula, Indicates fusion features; Represents the input data of the first Mesh-Cast SSM module; Represents the most balanced output characteristics; Indicates m Auxiliary output features; represents the weighting coefficient; M Indicates the number of Mesh-Cast SSM modules.

[0079] In another exemplary embodiment of the present application, usually, the input of the sequence model is composed of a fixed sequence of a certain length, but such an input method lacks randomness compared to the input of the ordinary 2D model with scrambling. On the other hand, using the scrambled sequence as input will cause the model to be unable to learn the true lesion sequence characteristics. These changes can be regarded as differences caused by different frequencies of the sequence images. To this end, a SOST strategy (Shuffled-OrderedSequence Training Strategy) is proposed for the training of the sequence model, so that the model learns on the scrambled sequence and the original sequence successively, obtains the ability to identify the commonality and specific sequence characteristics of the lesion, and balances the weights between the image features and sequence features of the sequence model. The SOST strategy is a two-stage training strategy designed to enhance the generalization ability of the model. The strategy improves the adaptability of the model to different sequence patterns by introducing randomness and diversity, while retaining the model's understanding and processing capabilities of the original sequence structure. This method is particularly suitable for tasks that require learning 2D semantic features and sequence features at the same time, such as the segmentation and analysis of brain tumor slices. Its operation process is as follows.

[0080] Set initial input is an original sample data set consisting of Tumor slice sample sequence Composition, one sequence is one sample, each tumor slice sample sequence Include elements. That is, the original sample data set of the initial input , each tumor slice sample sequence It is expressed as the following formula.

[0081] .

[0082] In the data shuffling stage, each tumor slice sample sequence element in the dataset Completely shuffled to generate a shuffled data set , whose elements can come from any sequence and any position .

[0083] .

[0084] in, Represents any sequence element in the original sample data set after shuffling. The elements of these sequences will be completely shuffled globally, that is, the elements of each new sequence may come from any position in any other sequence. When training the model for input, the samples and their corresponding labels are shuffled, that is, the shuffled order of the samples is the same as the shuffled order of the labels. The model can better adapt to different input modes and learn a wider range of features.

[0085] After training on the shuffled dataset, the segmentation model is trained using the original sequence order. The goal of this stage is to fine-tune the model to ensure that it can accurately understand and process data arranged in the correct order. Thanks to the sequence generalization ability of the Mesh-Cast SSM module, the SOST strategy enables the segmentation model to understand the common information of lesions and the individual characteristics of specific sequence instances, thereby improving the segmentation model's ability to learn sequence features.

[0086] Based on the above, if Figure 2 As shown, the training process of the Mesh-SSM UNet sequence segmentation model specifically includes the following steps.

[0087] (1) Obtaining an original model training data set; the original model training data set includes multiple tumor slice sample sequences and corresponding lesion area segmentation result sample sequences; each tumor slice sample sequence includes multiple tumor slice samples. The tumor slice samples in the tumor slice sample sequence are sorted in slice order. The tumor slice sample sequence images are derived from the open source data set of MICCAI.

[0088] (2) Performing a global shuffling process on the positions of the tumor slice samples in the tumor slice sample sequence in the original model training data set to obtain a shuffled model training data set.

[0089] (3) Using the tumor slice sample sequence in the shuffled model training data set as input and the corresponding lesion area segmentation result sample sequence as output, the initial Mesh-SSM UNet sequence segmentation model is trained.

[0090] (4) During the training process, the first model loss error of the Mesh-SSM UNet sequence segmentation model is calculated.

[0091] (5) When the loss error of the first model converges, the Mesh-SSM UNet sequence segmentation model after the first stage of training is obtained.

[0092] Steps (1) to (5) are the first stage model training process.

[0093] (6) Using the tumor slice sample sequence in the original model training data set as input and the corresponding lesion area segmentation result sample sequence as output, fine-tuning the Mesh-SSM UNet sequence segmentation model trained in the first stage.

[0094] (7) During fine-tuning, the second model loss error of the Mesh-SSM UNet sequence segmentation model is calculated.

[0095] (8) When the loss error of the second model converges, the Mesh-SSM UNet sequence segmentation model after the second stage training is obtained; the Mesh-SSM UNet sequence segmentation model after the second stage training is the final trained Mesh-SSM UNet sequence segmentation model.

[0096] Steps (6) to (8) are the second stage model training process.

[0097] In another exemplary embodiment of the present application, for the loss function of the segmentation model, the multi-category segmentation task of brain tumors can be converted into a multi-channel single-category segmentation task. In training, in order to reduce the complexity of the task, the segmentation tasks of various lesion areas are divided into single-category segmentation tasks with multiple channels, and the losses are calculated separately, and the joint loss function of the BCE loss function and the Dice loss function is used as the loss function of the segmentation model. The BCE loss function is defined as follows.

[0098] .

[0099] in, and Represents the width and height of the image respectively.

[0100] The Dice loss function is a loss function specially designed to deal with the problem of class imbalance. In the image segmentation task, the Dice loss function can more accurately reflect the segmentation effect of the model on small-scale lesion areas. The Dice loss function can be expressed as the following formula.

[0101] .

[0102] Based on the above two loss functions, the combined joint loss function is:

[0103] .

[0104] in, and Respectively represent k Dice loss for output channels and cross entropy loss ; and Respectively represent the width and height of the image; Represents the sample prediction image of the lesion area segmentation result P middle The pixel value at ; Indicates the real image of the lesion area segmentation result sample T middle The pixel value at ; N represents the number of pixels in the image; Represents the sample prediction image of the lesion area segmentation result P Middle i pixel value; Indicates the real image of the lesion area segmentation result sample T Middle i pixel value; represents the minimum constant; Is the weight value for balancing different losses, and its value range is The values ​​of the output channel k refer to three output channels, specifically three regions of interest of brain tumors: the enhanced tumor region (ET), the tumor core region (TC), and the entire tumor region (WT).

[0105] The following is the verification process of the lesion segmentation algorithm of this application.

[0106] BRATS-2019 is a brain tumor dataset released by MICCAI 2019. The training dataset includes 259 high-grade glioma (HGG) patients and 76 low-grade glioma (LGG) patients. The size of each MRI image is . 207 and 60 subjects were randomly selected from HGG and LGG for training, and 52 and 16 subjects were randomly selected for testing, with a training-test ratio of 8:2. The data of each subject included 4 MRI modes (T1, T1c, T2, and FLAIR) and 4 labels (0 for non-tumor, 1 for necrosis and non-enhancing tumor, 2 for edema, and 4 for enhancing tumor). The regions to be segmented include enhancing tumor (ET), tumor core (TC), and entire tumor (WT), and the corresponding labels are 4, 1+4, and 1+2+4, respectively.

[0107] Considering that the black background in MRI images is not helpful for segmentation, in order to eliminate the imbalance between target and background data, the black background is cropped. After processing, the size of each image is In addition, since the contrast of MRI images of different modalities is different, the foreground area of ​​each image is Z-score normalized separately.

[0108] The segmentation model of the present invention is implemented using the PyTorch framework under Ubuntu 22.24, and the specific version is similar to that required by MambaSSM. All experiments are implemented on an NVIDIA RTX 2080Ti. Considering the limitations of video memory and computing power, the input sequence length of the Mesh-SSM UNet sequence segmentation model is limited to 15.

[0109] All algorithms will use Dice Score and Hausdorff95 Dictance to evaluate and measure the algorithm effect. Dice Score is an indicator used to measure the similarity between two samples, especially in medical image segmentation to evaluate the consistency between the predicted segmentation and the true segmentation. Its calculation method is as follows.

[0110] .

[0111] in, (True Positives) represents the number of correctly predicted positive samples, (False Positives) represents the number of incorrectly predicted positive samples, (False Negatives) represents the number of positive samples that failed to be predicted correctly.

[0112] Hausdorff95 is an indicator that measures the distance between two sets. It calculates the average minimum distance from all elements in a set to the nearest point in another set. In the context of image segmentation, this indicator can be used to measure the spatial difference between the segmentation result and the true annotation. The calculation process is as follows.

[0113] .

[0114] In the formula, Represents the Hausdorff distance between set A and set B; and Represent the sets of two segmentation results respectively; Represents a collection of elements Elements in To the closest point (element set Middle distance element collection the distance to the nearest element); Represents a collection of elements Elements in To the closest point (element set Middle distance element collection the distance to the nearest element); and Respectively represent sets and A collection of elements. Representing a collection Elements in Representing a collection The elements in .

[0115] In order to analyze the role of the sequence and channel parts in the proposed Mesh-Cast SSM Group and the impact of different training strategies on performance, this application uses Vision Mamba as the U-type network (MambaUNet) of the basic encoding module as the BaseLine (benchmark) to design an ablation experiment, as shown in Table 1. Among them, S and C represent the sequence and channel associations of interest in the Mesh-Cast SSM module. E and D represent the encoder and decoder after adding the Mesh-Cast SSM Group, respectively, E (Encider) is the encoder, and D (Decoder) is the decoder.

[0116] Table 1 Ablation experiment results

[0117]

[0118] When using the same Ordered sequence input, the Mesh-Cast SSM Group containing sequence and channel information achieved the best results compared to BaseLine and a single sequence SSM. The experimental results using the same Shuffle sequence input also show that using the Mesh-Cast SSMGroup in the encoder is better than not using it or using it in both the encoder and decoder. This shows that using the Mesh-Cast SSM Group with sequence and channel dimension information in the encoder part is optimal. In addition, under the same Mesh-SSM UNet framework, the model trained using the SOST strategy achieved the best performance, which also confirms the effectiveness of the SOST strategy, which will also serve as the final configuration of the Mesh-SSM UNet sequence segmentation model.

[0119] As shown in Table 2, the comparison results with other medical image segmentation methods on the BRATS-2019 dataset are shown. The proposed Mesh-SSM UNet sequence segmentation model is compared with several advanced methods, including U-Net, TransUNet, TransNorm, UNETR, SwinUNETR, SegResNet, MedNeXt, SLf-UNet, MedSAM and Mamba UNet. To ensure a fair comparison, each algorithm will use the same dataset and loss function, and the training rounds of all algorithms follow the early stopping strategy of a maximum of 300 rounds, with an early stopping round of 30. The optimal indicators are indicated by bold underline, while the suboptimal indicators are only in bold. The indicators in Table 2 show that the proposed Mesh-SSM UNet sequence segmentation algorithm (i.e., Mesh-SSM UNet sequence segmentation model) outperforms all the above algorithms in each indicator in each area, and has a large degree of leading. The Mesh-SSM UNet sequence segmentation algorithm achieved Dice scores of 88.39, 90.52, and 91.43 in the WT, TC, and ET regions, and Hausdorff95 scores of 1.2869, 0.8154, and 0.6571, reaching the current highest level.

[0120] Table 2 Comparison results with other medical image segmentation methods on the BRATS-2019 dataset

[0121]

[0122] Experimental results on the BRATS-2019 dataset show that the proposed method has achieved significant improvements in segmentation accuracy and visual quality compared with existing algorithms, reaching the current highest level.

[0123] The beneficial effects of this application are as follows.

[0124] (1) It can be seen from the above experimental results that the present application improves the accuracy of lesion segmentation.

[0125] (2) The algorithm of this application can adapt to different types of medical image segmentation requirements. This method can be used to perform more effective segmentation on three-dimensional MRI images or CT images. It can provide doctors with more accurate image analysis results and assist in making more accurate clinical decisions. In particular, the random original sequence training strategy can also be applied separately to other sequence image processing applications, such as target tracking and detection in natural scenes.

[0126] The present application also provides an application scenario, which applies the above-mentioned lesion segmentation method based on image correlation information. Specifically: The lesion segmentation method based on image correlation information provided in this embodiment can be applied in tumor image processing scenarios. The scenario includes an image slice processing link, an image processing link, and a processing result display link; the image slice processing link is used to obtain a tumor slice sequence based on a three-dimensional tumor image using existing slice processing means; the image processing link is used to obtain a tumor slice sequence, and input the tumor slice sequence into the trained Mesh-SSM UNet sequence segmentation model to obtain a lesion area segmentation result sequence; the processing result display link is used to display the lesion area segmentation results. The lesion segmentation method based on image correlation information provided in this embodiment belongs to the image processing link.

[0127] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store lesion area segmentation results. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a lesion segmentation method based on image correlation information is implemented.

[0128] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0129] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0130] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0133] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0134] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A lesion segmentation method based on inter-image correlation information, characterized in that: The lesion segmentation method based on inter-image correlation information includes: Acquire a sequence of tumor slices to be segmented; The tumor slice sequence to be segmented is input into the trained Mesh-SSM UNet sequence segmentation model to obtain a sequence of lesion area segmentation results; the Mesh-SSM UNet sequence segmentation model adopts an improved UNet network model; in the improved UNet network model, the visual feature extraction unit in each encoding layer of the encoder adopts a 2D selective scanning module and a mesh propagation optional scanning module group connected in series, and the visual feature extraction unit in each decoding layer of the decoder adopts a 2D selective scanning module; the mesh propagation optional scanning module group is used to extract correlation features between image sequences from the image sequence output by the corresponding 2D selective scanning module; the tumor slice sequence to be segmented is input into the first encoding layer of the encoder, and the lesion area segmentation result sequence is output by the last decoding layer of the decoder; The mesh propagation optional scanning module group includes a layer attention mechanism module and a plurality of Mesh-CastSSM modules connected in series; the output end of each Mesh-Cast SSM module is also connected to the input end of the layer attention mechanism module; The Mesh-Cast SSM module is used to selectively perceive the image sequence output by the corresponding 2D selective scanning module in the sequence and channel dimensions to perceive the sequence changes of tumor lesions; The layer attention mechanism module is used to fuse the output of each Mesh-Cast SSM module to obtain a fused feature.

2. The lesion segmentation method based on inter-image correlation information according to claim 1, characterized in that: The output of each Mesh-Cast SSM module is fused to obtain the fusion features, including: Apply the SE module-based hierarchical attention mechanism to calculate the weight of the output features of each Mesh-Cast SSM module; The output feature corresponding to the maximum weight is determined as the most balanced output feature; Determine the output features corresponding to other weight values ​​as auxiliary output features; other weight values ​​refer to the remaining weight values ​​after removing the maximum weight value; The most balanced output feature is multiplied by the input data of the first Mesh-Cast SSM module, and then added to the auxiliary output feature to obtain the fused feature.

3. The lesion segmentation method based on inter-image correlation information according to claim 2, characterized in that: The calculation expression of the fusion feature is: ； In the formula, Indicates fusion features; Represents the input data of the first Mesh-Cast SSM module; Represents the most balanced output characteristics; Indicates m Auxiliary output features; represents the weighting coefficient; M Indicates the number of Mesh-Cast SSM modules.

4. The lesion segmentation method based on inter-image correlation information according to claim 1, characterized in that: The Mesh-Cast SSM module includes a first selective scanning unit, a first Mesh-Cast unit, a second selective scanning unit, and a second Mesh-Cast unit connected in sequence; The first selective scanning unit is used to perform attention perception processing on the corresponding input image sequence in the sequence and channel dimensions to obtain a first perception feature sequence; in the input image sequence of the first selective scanning unit, the number of frames is used as the sequence length, and the number of channels is used as the model dimension; The first Mesh-Cast unit is used to use the channel dimension as the sequence length and the number of frames as the model dimension, and to interchange and reorganize the sequence and channel dimensions of the first perceptual feature sequence to obtain a first reorganized sequence; The second selective scanning unit is used to perform attention perception on the first reorganized sequence in sequence and channel dimensions to obtain a second perception feature sequence; The second Mesh-Cast unit is used to use the number of frames as the sequence length and the number of channels as the model dimension, and to interchange and reorganize the sequence and channel dimensions of the second perceptual feature sequence to obtain a second reorganized sequence.

5. The lesion segmentation method based on inter-image correlation information according to claim 1, characterized in that: The training process of the Mesh-SSM UNet sequence segmentation model specifically includes: Acquire an original model training data set; the original model training data set includes a plurality of tumor slice sample sequences and corresponding lesion area segmentation result sample sequences; each tumor slice sample sequence includes a plurality of tumor slice samples; Performing a global position shuffling process on the tumor slice samples of the tumor slice sample sequence in the original model training data set to obtain a shuffled model training data set; Taking the tumor slice sample sequence in the shuffled model training data set as input and the corresponding lesion area segmentation result sample sequence as output, training the initial Mesh-SSM UNet sequence segmentation model; During the training process, the first model loss error of the Mesh-SSM UNet sequence segmentation model is calculated; When the loss error of the first model converges, the Mesh-SSM UNet sequence segmentation model after the first stage training is obtained; Taking the tumor slice sample sequence in the original model training data set as input and the corresponding lesion area segmentation result sample sequence as output, fine-tuning the Mesh-SSM UNet sequence segmentation model trained in the first stage; During fine-tuning, the second model loss error of the Mesh-SSM UNet sequence segmentation model is calculated; When the loss error of the second model converges, the Mesh-SSM UNet sequence segmentation model after the second stage training is obtained; the Mesh-SSM UNet sequence segmentation model after the second stage training is the final trained Mesh-SSM UNet sequence segmentation model.

6. The method for lesion segmentation based on inter-image correlation information according to claim 5, characterized in that: The expression of the loss error of the first model is: ； in, ； ； In the formula, represents the first model loss error, and Respectively represent k Output channels Dice loss and cross entropy loss ; Represents the weight value for balancing different losses; and Respectively represent the width and height of the image; Represents the sample prediction image of the lesion area segmentation result P middle The pixel value at ; Indicates the real image of the lesion area segmentation result sample T middle The pixel value at ; N Represents the number of pixels in the image; Represents the sample prediction image of the lesion area segmentation result P Middle pixel value; Indicates the real image of the lesion area segmentation result sample T Middle pixel value; Represents a constant.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the lesion segmentation method based on inter-image correlation information as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for segmenting a lesion based on correlation information between images described in any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for segmenting a lesion based on correlation information between images described in any one of claims 1 to 6 is implemented.

Images