A medical image segmentation method and system based on structural constraints

Abstract

The present invention relates to the field of medical image segmentation, and discloses a medical image segmentation method and system based on structural constraints, the method comprising: performing global feature extraction on a medical image data set to obtain a first feature representation; performing encoding, decoding and deep convolution processing on the first feature representation to obtain a feature image map and visual features; performing feature generation and feature fusion on the feature image map and visual features using a self-attention mechanism and a cross-attention mechanism to obtain a second feature representation; fusing the first feature representation and the second feature representation through a residual connection and performing feature decoding to obtain a segmented image corresponding to the medical image data set. The present invention significantly reduces the semantic gap between the feature image map and the visual features through the effective fusion of the two, thereby improving the accuracy of medical image segmentation. At the same time, by utilizing the structural constraint relationship in the feature image map, the present invention achieves accurate segmentation of medical images.

Description

Technical Field

[0001] The present invention relates to the field of medical image segmentation, and in particular to a medical image segmentation method and system based on structural constraints. Background Art

[0002] Medical image segmentation, as a core step in medical image analysis, aims to accurately extract specific areas or structures in images, thereby helping doctors more accurately identify lesions, assess conditions, and develop treatment plans. This technology plays a pivotal role in clinical practice.

[0003] First, accurate medical image segmentation enables doctors to more clearly identify the lesion area, accurately assess the severity of the disease, and then formulate personalized treatment plans for patients. In addition, this technology also plays a key role in complex medical scenarios such as three-dimensional reconstruction and surgical navigation, significantly improving the accuracy and safety of surgery.

[0004] Secondly, medical image segmentation covers a variety of methods. Traditional segmentation methods, such as threshold, edge and region-based segmentation, are mainly based on image features such as grayscale, color and texture, but are often affected by noise, lighting conditions and other interference.

[0005] Therefore, medical image segmentation still faces many challenges. For example, the complexity of medical images and factors such as partial volume effect, grayscale inhomogeneity, and artifacts make it a very challenging task to automatically segment the target area from the image. In summary, how to further improve the accuracy and speed of the segmentation algorithm to meet the real-time and accuracy requirements of clinical applications has become a problem that needs to be solved urgently. Summary of the invention

[0006] The present invention provides a medical image segmentation method, system, electronic device and computer-readable storage medium based on structural constraints, the main purpose of which is to solve the problem of how to further improve the accuracy and speed of the segmentation algorithm to meet the real-time and accuracy requirements of clinical applications.

[0007] To achieve the above object, the present invention provides a medical image segmentation method based on structural constraints, comprising:

[0008] Perform global feature extraction on the medical image dataset to obtain the first feature representation;

[0009] Performing encoding, decoding and deep convolution processing on the first feature representation respectively to obtain a feature image and a visual feature;

[0010] Using a self-attention mechanism to generate features for the feature image and the visual features, and using a cross-attention mechanism to fuse the generated features to obtain a second feature representation;

[0011] The first feature representation and the second feature representation are fused through residual connection and feature decoding is performed to obtain a segmented image corresponding to the medical image data set.

[0012] In one embodiment of the present invention, global feature extraction is performed on a medical image data set to obtain a first feature representation, including:

[0013] Preprocessing the medical image data set to obtain a preprocessed image;

[0014] Convolving and downsampling the preprocessed image to obtain a deep feature;

[0015] Deconvolution is performed on the deep features to obtain the first feature representation.

[0016] In one embodiment of the present invention, encoding and decoding the first feature representation to obtain a feature image map includes:

[0017] Using a self-attention encoder in a preset self-attention model to perform feature conversion on the first feature representation to obtain a corresponding unit vector;

[0018] Determine the constraint relationship between the unit vectors using a self-attention decoder in the self-attention model;

[0019] The feature image map is generated according to the unit vector and the corresponding constraint relationship.

[0020] In one embodiment of the present invention, deep convolution processing is performed on the first feature representation to obtain visual features, including:

[0021] Performing a depth-wise separable convolution on the first feature representation to obtain a depth-wise convolution feature;

[0022] Performing point-by-point convolution on the deep convolutional features to obtain a third feature representation;

[0023] The third feature representation is flattened to obtain the visual feature.

[0024] In one embodiment of the present invention, the feature image and the visual feature are generated using a self-attention mechanism, and the generated features are fused using a cross-attention mechanism to obtain a second feature representation, including:

[0025] Perform feature updating according to the node similarity between each node in the feature image graph and the feature similarity between each feature in the visual feature, to obtain a target feature image graph and a target visual feature;

[0026] Calculating the target similarity between the target feature image and the target visual feature;

[0027] Generate a cross-attention weight matrix according to the target similarity;

[0028] The target feature image and the target visual feature are subjected to feature fusion according to the cross-attention weight matrix to obtain the second feature representation.

[0029] In one embodiment of the present invention, feature updating is performed according to the node similarity between each node in the feature image graph and the feature similarity between each feature in the visual feature to obtain a target feature image graph and a target visual feature, including:

[0030] Calculating the node similarity between each node in the feature image graph and the feature similarity between each feature in the visual feature respectively;

[0031] Generate a corresponding first attention weight matrix and a second attention weight matrix according to the node similarity and the feature similarity;

[0032] The feature image map and the visual features are weighted according to the first attention weight matrix and the second attention weight matrix respectively to obtain the target feature image map and the target visual features.

[0033] In one embodiment of the present invention, generating an attention weight matrix according to the similarity includes:

[0034] Normalizing the similarity to obtain an attention weight;

[0035] The attention weight matrix is ​​generated according to the attention weight and the vector dimension of each node in the feature image graph.

[0036] In one embodiment of the present invention, the target feature image and the target visual feature are subjected to feature fusion according to the cross attention weight matrix to obtain the second feature representation, including:

[0037] weighting the target visual feature according to the cross attention weight matrix to obtain a weighted visual feature representation;

[0038] The weighted visual feature representation is fused into the target feature image to obtain the second feature representation.

[0039] In one embodiment of the present invention, fusing the first feature representation and the second feature representation through residual connection and performing feature decoding to obtain a segmented image corresponding to the medical image dataset includes:

[0040] Performing a residual connection on the first feature representation and the second feature representation to obtain a fused feature;

[0041] The fused features are feature decoded to obtain a decoded feature map, and the segmented image is determined according to the decoded feature map.

[0042] In order to solve the above problems, the present invention also provides a medical image segmentation system based on structural constraints, the system comprising:

[0043] A feature extraction module, used for performing global feature extraction on the medical image data set to obtain a first feature representation;

[0044] An image map and visual feature generation module, used for performing encoding and decoding and deep convolution processing on the first feature representation to obtain a feature image map and a visual feature;

[0045] An image graph embedding module, used for generating features of the feature image graph and the visual features by using a self-attention mechanism, and performing feature fusion on the generated features by using a cross-attention mechanism to obtain a second feature representation;

[0046] A decoding and segmentation module is used to fuse the first feature representation and the second feature representation through a residual connection and perform feature decoding to obtain a segmented image corresponding to the medical image data set.

[0047] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0048] at least one processor; and,

[0049] a memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the steps of the above-mentioned medical image segmentation method based on structural constraints.

[0051] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the steps of the above-mentioned medical image segmentation method based on structural constraints.

[0052] The present invention can solve the problem of difficult image segmentation caused by the target area in the medical image being often blocked by other areas or part of itself, or the tissue boundary being blurred or discontinuous, through the structural constraint relationship in the feature image map and the visual features, thereby improving the accuracy of image segmentation; based on global feature extraction, deep feature extraction and other operations, it can realize the processing of multi-scale and multi-type medical images; through the effective fusion of the feature image map and the visual features, it can effectively alleviate the problem of the large semantic gap between the structural constraints represented by the image map and the visual features, thereby realizing the accurate segmentation of medical images. Therefore, the main purpose of the medical image segmentation method, system, electronic device and computer-readable storage medium based on structural constraints proposed in the present invention is to solve the problem of how to further improve the accuracy and speed of the segmentation algorithm to meet the real-time and accuracy requirements of clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a flow chart of a medical image segmentation method based on structural constraints provided by an embodiment of the present invention;

[0054] Figure 2 A network framework diagram of medical image segmentation provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a process for generating a second feature representation provided by an embodiment of the present invention;

[0056] Figure 4 A functional module diagram of a medical image segmentation system based on structural constraints provided by an embodiment of the present invention;

[0057] Figure 5 A schematic diagram of the structure of an electronic device for implementing the medical image segmentation method based on structural constraints provided by an embodiment of the present invention.

[0058] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0059] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0060] The target area in medical images has the characteristics of occlusion and weak features, which makes it difficult for existing image segmentation methods to obtain good segmentation results. In addition, due to the influence of factors such as imaging environment and patient lesions, the content of different medical images is generally quite different, which brings certain difficulties to the mining of target structural constraints. In addition, there is a large semantic gap between the structural constraints represented by image graphs and visual features. How to achieve effective fusion is also a major technical problem at present.

[0061] In medical images, there is strong structural constraint information between target areas, including positional relationships, inclusion relationships, etc. How to mine and utilize this information plays an important role in achieving accurate segmentation of medical images. To this end, the present application discloses a medical image segmentation method and system based on structural constraints, which mines and utilizes the structural constraint information between target areas, generates an image map to express the structural constraints between targets, and combines visual features and influence feature maps to effectively enhance target features and achieve accurate segmentation of medical images.

[0062] Based on the problems existing in the prior art, the technical solution of the present invention can achieve the following technical effects:

[0063] 1. Occlusion problem in images: In medical images, the target area is often occluded by other areas or parts of itself, which increases the difficulty of segmentation. The structural constraints extracted in the present invention can help the model to handle such occlusion to obtain a complete segmentation result.

[0064] 2. Boundary blur and discontinuity: In the image segmentation process, accurate extraction of edge information is key. However, due to factors such as weak features, noise, blur and low contrast, tissue boundaries in medical images may be blurred or discontinuous, which makes segmentation difficult. The structural constraints extracted in the present invention can help the model better identify and understand these boundaries, thereby improving the accuracy of segmentation.

[0065] 3. Objects of different scales and categories: Medical images may contain objects of different scales and categories, requiring the model to be able to process multi-scale information and multi-category objects. Segmentation models based on structural constraints and multiple feature transformations such as global feature extraction and deep feature extraction can achieve accurate segmentation of objects of different scales and categories.

[0066] 4. Complexity of medical images: Medical images usually contain a variety of different tissues and structures, and are affected by various factors such as noise and artifacts, making the segmentation task complicated. The prior knowledge or structural information introduced in the segmentation model based on structural constraints can improve the accuracy and robustness of segmentation.

[0067] The present invention is mainly used in the field of medical image processing, especially in the precise segmentation and analysis of medical images. Through the medical image segmentation method based on structural constraints, key information can be extracted from medical images more accurately, providing strong support for the diagnosis, treatment and prognosis evaluation of diseases. Specifically, the system can be applied to various medical imaging technologies, such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (Ultrasound) and X-ray (X-ray), etc., to achieve precise segmentation of different organs, lesions or other biological structures. For example, in chest X-ray images, this method can effectively segment lung areas and lesions, such as lung nodules, pneumonia, etc.; in brain MRI images, segmentation of different brain regions can be achieved, which is helpful to study the structure and function of the brain.

[0068] The embodiment of the present application provides a medical image segmentation method based on structural constraints. The execution subject of the medical image segmentation method based on structural constraints includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the medical image segmentation method based on structural constraints can be executed by software or hardware installed in a terminal device or a server device.

[0069] Reference Figure 1 FIG. 1 is a flow chart of a medical image segmentation method based on structural constraints provided by an embodiment of the present invention. In this embodiment, the medical image segmentation method based on structural constraints includes:

[0070] S1. Perform global feature extraction on the medical image dataset to obtain a first feature representation.

[0071] In the embodiment of the present invention, the medical image data set includes multiple medical images, such as X-ray images, angiography images, computer tomography images, magnetic resonance imaging images, etc.

[0072] In an embodiment of the present invention, global feature extraction is performed on a medical image data set to obtain a first feature representation, including: preprocessing the medical image data set to obtain a preprocessed image; convolving and downsampling the preprocessed image to obtain a deep feature; and deconvolving the deep feature to obtain the first feature representation.

[0073] In the embodiment of the present invention, the preprocessing operation includes image standardization, denoising, image segmentation, etc., and the medical image data set is preprocessed so as to subsequently extract different target areas of the image in the medical image data set.

[0074] In medical image segmentation, different target regions mainly refer to various anatomical structures in the image or regions with specific pathological characteristics. The main reasons why medical image segmentation divides an image into multiple regions are as follows:

[0075] 1. Complexity of anatomical structure: The human body has a complex anatomical structure, including multiple organs, tissues, and blood vessels, etc. These structures need to be identified and processed separately in medical images. For example, in chest X-ray images, according to the chest anatomical structure, medical imaging, and the characteristics of imaging examination reports, the chest can be divided into: thorax, trachea, mediastinum, lungs, heart shadow, aorta, diaphragm, costophrenic angle, bone and other regions; according to different organs, it can also be divided into: heart, lungs, liver, kidneys and other regions, which occupy specific positions in medical images and have unique shapes and characteristics; according to different lesion areas, it can be divided into: tumors, cysts, inflammation, etc., these areas are usually shown in medical images with different grayscale, texture or shape from normal tissues.

[0076] 2. Diversity of pathological features: The lesion area may appear in different grayscales, textures or shapes in medical images. In order to accurately diagnose and evaluate the disease, these areas need to be segmented separately.

[0077] 3. Diagnosis and treatment needs: The results of medical image segmentation can be used in a variety of medical applications, such as surgical planning, lesion localization, lesion measurement, etc. By dividing the image into multiple regions, doctors can more accurately determine the location, size and shape of the lesion, thereby formulating a more effective treatment plan.

[0078] In an embodiment of the present invention, the preprocessed image can be input into the UTNet deep model for global feature extraction. The model includes an encoder, a decoder and an output layer. The encoder combines a convolutional layer and a downsampling module, which can effectively capture global feature information and improve feature resolution, and extract deep features of medical images; the deconvolution operation of the UTNet deep model can maintain global feature information while improving feature resolution, which helps to more accurately understand and analyze medical images; the output layer can output a first feature representation with rich semantic information and high resolution.

[0079] The encoder may include five convolutional layers, which are used to extract features from medical images. Each convolutional layer learns and extracts information at different levels in the image. Between the convolutional layers, downsampling modules (such as the maximum pooling layer) are added to gradually reduce the spatial resolution of the features and increase the receptive field to capture a wider range of information. After five layers of convolution and downsampling operations, the deep features of medical images can be effectively extracted. The deep features contain rich semantic information, which is helpful for subsequent image analysis.

[0080] In detail, in order to improve the resolution of deep features and capture global feature information, a five-layer deconvolution operation can be used. Deconvolution (also known as transposed convolution) can gradually increase the spatial size of the feature map while maintaining or enhancing the learned features. During the deconvolution process, the size and step size of the convolution kernel are adjusted to accurately control the size change of the feature map, ensuring that the final output feature map maintains a certain correspondence with the original medical image in terms of spatial size.

[0081] S2. Perform encoding, decoding and deep convolution processing on the first feature representation to obtain a feature image and visual features.

[0082] In an embodiment of the present invention, in order to overcome the influence of factors such as the imaging environment, patient lesions, and differences in the contents of different medical images on the mining of target structural constraints, the structural constraints of different target areas in the medical image can be mined, thereby generating an image map with clear regional relationships and constraint relationships.

[0083] In an embodiment of the present invention, the first feature representation is encoded and decoded to obtain a feature image map, including: using a self-attention encoder in a preset self-attention model to perform feature conversion on the first feature representation to obtain a corresponding unit vector; using a self-attention decoder in the self-attention model to determine the constraint relationship between the unit vectors; and generating the feature image map based on the unit vectors and the corresponding constraint relationship.

[0084] See also Figure 2 As shown, it is a network framework diagram of image segmentation provided by the present invention. The self-attention encoder (Transformer Encoder) and the self-attention decoder (Transformer Decoder) are two important parts of the self-attention model (Transformer model), which are used to encode the input sequence and generate the output sequence respectively.

[0085] Among them, the Transformer Encoder is composed of multiple identical encoder layers, each of which contains a multi-head self-attention mechanism and a feedforward neural network to encode the information in the input sequence; the function of the Encoder is to convert the feature sequence represented by the first feature of the input into a series of unit vectors (Tokens), each Token represents the information of different target areas in the medical image.

[0086] In medical image segmentation, the relationship between different regions usually refers to their mutual correlation in anatomical structure, spatial position and possible pathological changes. Take chest X-ray images as an example:

[0087] 1. Relationship of anatomical structures: Chest X-ray images mainly include anatomical structures such as the lungs, heart, ribs, thorax, and mediastinum. These structures each occupy a specific position on the image and have a clear anatomical relationship with each other. For example, the heart is located in the mediastinum, and its shape and size can be clearly shown on the X-ray image. The ribs surround the thorax to provide protection for the chest cavity. The lungs occupy most of the space in the chest cavity and are adjacent to the heart and mediastinum.

[0088] 2. Spatial position relationship: Different areas on the chest X-ray image are arranged according to their anatomical positions to form an orderly spatial distribution. For example, from the top to the bottom of the image, you can see the clavicle, ribs, lungs, heart and diaphragm in turn. From left to right, you can see the symmetrical thorax, ribs and lungs on both sides.

[0089] 3. Pathological connection: In pathological conditions, there may be correlation or influence between different areas on the chest X-ray image.

[0090] In an actual application scenario, when inflammation occurs in the lungs, it may increase the density of the surrounding lung tissue, forming a patchy shadow. At the same time, inflammation may also affect the structure of the mediastinum and chest wall, leading to signs such as widening of the mediastinum or thickening of the chest wall. Heart diseases such as heart failure may cause congestion in the lungs, which can be seen on X-ray images as enlarged hilar shadows and increased lung texture. The above changes are not limited to the heart area, but also involve the lung tissue adjacent to the heart.

[0091] Suppose a round shadow with clear edges and uniform density is found in the right upper lobe on a chest X-ray image. Based on the anatomical structure relationship, it can be judged that the shadow may be located in a certain part of the right upper lobe, adjacent to the surrounding lung tissue, ribs, mediastinum and other structures. Further observation of the spatial position relationship shows that the shadow is located in the middle and upper part of the chest, close to the mediastinum and ribs. This position relationship helps to locate the lesion more accurately. Considering the pathological connection, if the shadow is a pulmonary nodule, it is necessary to further evaluate its relationship with the surrounding tissues and possible pathological changes.

[0092] In an embodiment of the present invention, in the precise segmentation of medical images, the Transformer Decoder is composed of multiple identical decoder layers, each decoder layer contains a multi-head self-attention mechanism, a multi-head attention mechanism and a feedforward neural network, which is used to generate an output sequence; the Transformer Decoder models the relationship between different target areas, and learns and understands the relationship between tokens through the self-attention mechanism and the encoder-decoder attention mechanism. In the precise segmentation of medical images, these relationships include spatial position relationships, anatomical structure relationships, etc.; based on these relationships, the self-attention model can capture the constraint relationships between target areas and generate accurate segmentation results that meet these constraints.

[0093] In detail, in the Transformer Decoder, the self-attention mechanism allows each token to pay attention to all other tokens in the sequence. In medical image segmentation, a token represents a pixel block (also called a patch) or a feature vector in the image. By calculating the attention weight between each token and other tokens, the model can learn the dependencies between tokens. These dependencies may include proximity in spatial position or similarity in feature values.

[0094] The encoder-decoder attention mechanism allows the tokens in the decoder to pay attention to all the tokens in the encoder output. In medical image segmentation, the encoder is usually used to extract feature information in the image, while the decoder is responsible for generating segmentation results based on this feature information. By calculating the attention weights between the tokens in the decoder and the tokens in the encoder output, the model can understand the correspondence between the input image and the output segmentation result, thereby more accurately capturing the constraint relationship between the target areas.

[0095] Since the anatomical structures in medical images usually have fixed spatial position relationships, the spatial position relationship between tokens can be used as a constraint. For example, the left and right lungs are located in the left and right chest cavities, respectively, and the heart is located in the middle mediastinum. By learning the spatial position relationship between tokens, the model can ensure that these spatial position constraints are followed during the segmentation process, thereby improving the accuracy of segmentation.

[0096] The anatomical structures in medical images have relatively fixed shapes and sizes. Therefore, during the segmentation process, the model needs to follow these anatomical constraints to ensure the accuracy of the segmentation results. By learning the relationship between tokens and combining known anatomical information, the model can understand which regions correspond to which anatomical structures, and generate segmentation results that meet the anatomical constraints.

[0097] Furthermore, the feature image graph is represented in the form of a graph structure, which is a directed or undirected graph consisting of nodes and edges. The feature image graph uses vector form to represent nodes and relationships. Through word embedding (Word Embedding) or graph embedding (Graph Embedding), nodes and relationships can be mapped to a low-dimensional vector space, which facilitates efficient calculation and reasoning.

[0098] In detail, nodes represent entities, which represent different target areas. For example, in the image of a chest X-ray image, nodes may include thorax, trachea, mediastinum, lung texture, clavicle, scapula, ribs, etc.

[0099] Edges represent the relationship between nodes and the constraint relationship between regions. For example, the left and right are located in the left and right chest cavities, and the heart is located in the mediastinum. There may be a "located in" or "occupied" relationship between "lungs" and "chest cavity". "Entity-relationship-entity" constitutes a triple. For example, "(lung, located in, chest cavity)" is a triple.

[0100] In another optional embodiment of the present invention, the self-attention model can be optimized by an optimization algorithm, and the generated feature image map can be evaluated using appropriate evaluation indicators to ensure that it can accurately reflect the structural constraints of the target area in the medical image.

[0101] In an embodiment of the present invention, in order to further extract and refine features, the first type of feature map is input into a depthwise separable convolution (Depthwise Conv) for deep feature extraction. These feature maps retain the key information in the original image and are processed by the encoder so that subsequent operations can focus more on extracting features that are beneficial to the segmentation task.

[0102] Specifically, the first feature representation is subjected to deep convolution processing to obtain visual features, including: performing deep separable convolution on the first feature representation to obtain deep convolution features; performing point-by-point convolution on the deep convolution features to obtain a third feature representation; and flattening the third feature representation to obtain the visual features.

[0103] Among them, a convolution kernel is applied to each channel of the first feature representation to extract local features on each channel; in the depthwise separable convolution stage, each input feature map is individually subjected to a depthwise convolution operation, and each convolution kernel is convolved on only one feature map, thereby further extracting and refining features. This operation helps to separate information between different feature maps and emphasizes local patterns within each feature map.

[0104] In detail, pointwise convolution (1x1 convolution) is used to fuse all deep convolution features to obtain the third feature representation; pointwise convolution can combine feature information across channels, thereby capturing the interactions and dependencies between feature maps, and helps to fuse information from different feature maps to form a more comprehensive third feature representation.

[0105] The third feature not only contains the original features extracted by the UTNet encoder, but also has been further processed and refined by the depthwise separable convolution. By combining the UTNet encoder and the depthwise separable convolution, the advantages of both can be fully utilized to achieve more efficient and accurate feature extraction.

[0106] Furthermore, the third feature representation is flattened to obtain a visual feature representation of a one-dimensional vector of the third feature representation, and the one-dimensional vector can be input into a linear layer or a fully connected layer for further processing to obtain visual features. The one-dimensional vector representation can capture the key information of the image and is convenient for fusion with other types of data.

[0107] S3. Use the self-attention mechanism to generate features for the feature image and the visual features, and use the cross-attention mechanism to fuse the generated features to obtain a second feature representation.

[0108] In an embodiment of the present invention, in order to alleviate the problem that there is a large semantic gap between the structural constraints and visual features represented by the feature image graph, an effective fusion of the structural constraints and visual features represented by the image graph is achieved through an image graph embedding method. That is, the self-attention mechanism and the cross-attention mechanism are combined to achieve the enhancement and fusion of the image graph and the visual features. In this way, the structural constraint information of the image graph is fully mined, the mining and utilization of the structural constraint information of the image graph is achieved, and the accurate segmentation of the medical image is achieved through the effective fusion of the structural constraints and visual features represented by the image graph.

[0109] See also Figure 3 As shown, in the embodiment of the present invention, the feature image and the visual feature are generated by using a self-attention mechanism, and the generated features are fused by using a cross-attention mechanism to obtain a second feature representation, including:

[0110] S31, performing feature update according to the node similarity between each node in the feature image graph and the feature similarity between each feature in the visual feature, to obtain a target feature image graph and a target visual feature;

[0111] S32, calculating the target similarity between the target feature image and the target visual feature;

[0112] S33, generating a cross attention weight matrix according to the target similarity;

[0113] S34. Perform feature fusion on the target feature image and the target visual feature according to the cross-attention weight matrix to obtain the second feature representation.

[0114] In detail, feature updating is performed according to the node similarity between each node in the feature image map and the feature similarity between each feature in the visual features to obtain the target feature image map and the target visual features, including: respectively calculating the node similarity between each node in the feature image map and the feature similarity between each feature in the visual features; generating a corresponding first attention weight matrix and a second attention weight matrix according to the node similarity and the feature similarity; and weighting the feature image map and the visual features according to the first attention weight matrix and the second attention weight matrix, respectively, to obtain the target feature image map and the target visual features.

[0115] Among them, generating an attention weight matrix according to the similarity includes: normalizing the similarity to obtain attention weight; generating the attention weight matrix according to the attention weight and the vector dimension of each node in the feature image.

[0116] Furthermore, the target feature image map and the target visual features are feature fused according to the cross-attention weight matrix to obtain the second feature representation, including: weighting the target visual features according to the cross-attention weight matrix to obtain a weighted visual feature representation; fusing the weighted visual feature representation into the target feature image map to obtain the second feature representation.

[0117] For the feature update of the feature image graph, three different linear transformations or weight matrices can be used to generate the query Q1, key K1 and value V1 vectors corresponding to the feature image graph, which are used to calculate the similarity and generate the attention weight; for each node in the image graph, the similarity is calculated using its query vector Q1 and the key vector K1 of all nodes; the similarity score is normalized by the softmax function to obtain the attention weight, which represents the degree of attention of each node to other nodes; the attention weights of all nodes are combined into a matrix, namely the attention weight matrix A1, and each element in the attention weight matrix A1 represents the attention weight of the corresponding node.

[0118] Each weight in the attention weight matrix A1 is element-wise multiplied with the feature vector of the corresponding node in the feature image. By weighting the feature vectors, important positions are emphasized and unimportant positions are weakened. The resulting target feature image M1 is a reweighted feature representation that contains the same number of feature vectors as the feature representation in the original feature image, but the value of each feature vector is adjusted according to the attention weight.

[0119] For the feature update of visual features, three different linear transformations or weight matrices can be used to generate query Q2, key K2 and value V2 vectors for calculating similarity and generating attention weights; for each feature in the visual features, the similarity between its query vector Q2 and the key vector K2 of all features is calculated; the similarity scores are normalized by the softmax function to obtain the attention weight of each feature.

[0120] The generated attention weights are used to perform weighted summation on the value V2 vector to obtain a new feature representation, namely the target visual feature M2. By aggregating the features, the correlation between different features is considered; the target visual feature not only contains the information of the original visual features, but also incorporates the dependency between different features, making it more expressive.

[0121] In the embodiment of the present invention, after the target feature image map and the target visual features are generated, the target feature image map M1 can be used as a "query". Specifically, the target feature image map M1 is subjected to global average pooling (GAP) to obtain a vector of a fixed size, which can be used as a query.

[0122] The target visual feature M2 is used as a “key-value pair”. Specifically, the target visual feature M2 is subjected to global average pooling (GAP) to obtain a vector of a fixed size, which can be used as a key-value pair.

[0123] Furthermore, the similarity between the query (target feature image M1) and each key (target visual feature M2) is calculated, and then the similarity scores are normalized by the softmax function to obtain the cross-attention weight matrix B2.

[0124] In the embodiment of the present invention, the cross-attention weight matrix B2 is used to weight the target visual feature M2 to obtain a weighted feature representation. The weighted feature representation contains information from the entire feature map, where the contribution of different features is weighted according to their similarity to the query; the weighted feature representation is fused into the feature representation of the target feature image map, and then the relevant information in the visual feature is fused into the image map feature to obtain a second feature representation, which effectively combines the information of the image map and the visual feature.

[0125] The present invention realizes the mining and utilization of image image structural constraint information by extracting constraint relationships, and effectively alleviates the problem of a large semantic gap between the structural constraints represented by image images and the visual features by effectively integrating the structural constraints represented by image images with the visual features, thus realizing the accurate segmentation of medical images.

[0126] S4. Fusing the first feature representation and the second feature representation through residual connection and performing feature decoding to obtain a segmented image corresponding to the medical image dataset.

[0127] In an embodiment of the present invention, the first feature representation and the second feature representation are fused through residual connection and feature decoding is performed to obtain a segmented image corresponding to the medical image data set, including: performing residual connection on the first feature representation and the second feature representation to obtain a fused feature; performing feature decoding on the fused feature to obtain a decoded feature map, and determining the segmented image based on the decoded feature map.

[0128] Among them, the decoder consists of an upsampling layer, a convolution layer, a fusion layer, an activation function, and an output layer. The decoder is used to map the fused features back to the high-resolution space and output the decoded feature map.

[0129] like Figure 4 , is a functional module diagram of a medical image segmentation system based on structural constraints provided by an embodiment of the present invention. According to the functions to be implemented, the medical image segmentation system based on structural constraints 400 may include a feature extraction module 401, an image map and visual feature generation module 402, an image map embedding module 403, and a decoding segmentation module 404. The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of the medical image segmentation system based on structural constraints and can complete fixed functions, which are stored in the memory of the medical image segmentation system based on structural constraints.

[0130] In this embodiment, the functions of each module / unit are as follows:

[0131] The feature extraction module 401 is used to perform global feature extraction on the medical image data set to obtain a first feature representation;

[0132] The image map and visual feature generation module 402 is used to perform encoding and decoding and deep convolution processing on the first feature representation to obtain a feature image map and a visual feature;

[0133] The image embedding module 403 is used to generate features from the feature image and the visual features using a self-attention mechanism, and to fuse the generated features using a cross-attention mechanism to obtain a second feature representation;

[0134] The decoding and segmentation module 404 is used to fuse the first feature representation and the second feature representation through residual connection and perform feature decoding to obtain a segmented image corresponding to the medical image dataset.

[0135] In detail, each module described in the structural constraint-based medical image segmentation system 400 described in the embodiment of the present invention adopts the same technical means as the structural constraint-based medical image segmentation method described in the accompanying drawings when used, and can produce the same technical effects, which will not be repeated here.

[0136] like Figure 5 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing a medical image segmentation method based on structural constraints provided by an embodiment of the present invention.

[0137] The electronic device 500 may include a processor 501 , a memory 502 , a communication bus 503 , and a communication interface 504 , and may also include a computer program stored in the memory 502 and executable on the processor 501 , such as a medical image segmentation program based on structural constraints.

[0138] The processor 501 is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect the various components of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory 502 (for example, executing a medical image segmentation program based on structural constraints, etc.), and calling data stored in the memory 502.

[0139] The memory 502 includes at least one type of readable storage medium. In some embodiments, the memory 502 may include both an internal storage unit of the electronic device and an external storage device. The memory 502 may be used not only to store application software and various data installed in the electronic device, such as the code of the medical image segmentation program based on structural constraints, but also to temporarily store data that has been output or is to be output.

[0140] The communication bus 503 can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 502 and at least one processor 501, etc.

[0141] The communication interface 504 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the electronic device and other electronic devices.

[0142] The medical image segmentation program based on structural constraints stored in the memory 502 in the electronic device 500 is a combination of multiple instructions. When executed in the processor 501, the steps in the above-mentioned medical image segmentation method based on structural constraints can be implemented.

[0143] Specifically, the specific implementation system of the processor 501 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0144] Furthermore, if the modules / units integrated in the electronic device 500 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0145] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the steps in the above-mentioned medical image segmentation method based on structural constraints can be implemented.

[0146] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0147] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0149] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0150] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A medical image segmentation method based on structural constraints, characterized in that: The method comprises: Perform global feature extraction on the medical image dataset to obtain the first feature representation; Performing encoding, decoding and deep convolution processing on the first feature representation respectively to obtain a feature image and a visual feature; Using a self-attention mechanism to generate features for the feature image and the visual features, and using a cross-attention mechanism to fuse the generated features to obtain a second feature representation; fusing the first feature representation and the second feature representation through a residual connection and performing feature decoding to obtain a segmented image corresponding to the medical image dataset; The feature image graph is a directed graph or an undirected graph consisting of nodes and edges, and the feature image graph uses vectors to represent nodes and relationships; nodes represent different target areas, and edges represent constraint relationships between areas; The feature image and the visual feature are generated by using a self-attention mechanism, and the generated features are fused by using a cross-attention mechanism to obtain a second feature representation, including: Perform feature updating according to the node similarity between each node in the feature image graph and the feature similarity between each feature in the visual feature, to obtain a target feature image graph and a target visual feature; Calculating the target similarity between the target feature image and the target visual feature; Generate a cross-attention weight matrix according to the target similarity; The target feature image and the target visual feature are subjected to feature fusion according to the cross-attention weight matrix to obtain the second feature representation.

2. The medical image segmentation method based on structural constraints according to claim 1, characterized in that: Perform global feature extraction on the medical image dataset to obtain the first feature representation, including: Preprocessing the medical image data set to obtain a preprocessed image; Convolving and downsampling the preprocessed image to obtain a deep feature; Deconvolution is performed on the deep features to obtain the first feature representation.

3. The medical image segmentation method based on structural constraints according to claim 1, characterized in that: Encoding and decoding the first feature representation to obtain a feature image map includes: Using a self-attention encoder in a preset self-attention model to perform feature conversion on the first feature representation to obtain a corresponding unit vector; Determine the constraint relationship between the unit vectors using a self-attention decoder in the self-attention model; The feature image map is generated according to the unit vector and the corresponding constraint relationship.

4. The medical image segmentation method based on structural constraints according to claim 1, characterized in that: Performing deep convolution processing on the first feature representation to obtain visual features includes: Performing a depth-wise separable convolution on the first feature representation to obtain a depth-wise convolution feature; Performing point-by-point convolution on the deep convolutional features to obtain a third feature representation; The third feature representation is flattened to obtain the visual feature.

5. The medical image segmentation method based on structural constraints according to claim 1, characterized in that: Performing feature updating according to the node similarity between each node in the feature image graph and the feature similarity between each feature in the visual feature to obtain a target feature image graph and a target visual feature, including: Calculating the node similarity between each node in the feature image graph and the feature similarity between each feature in the visual feature respectively; Generate a corresponding first attention weight matrix and a second attention weight matrix according to the node similarity and the feature similarity; The feature image map and the visual features are weighted according to the first attention weight matrix and the second attention weight matrix respectively to obtain the target feature image map and the target visual features.

6. The medical image segmentation method based on structural constraints according to claim 5, characterized in that: Generate an attention weight matrix according to the similarity, including: Normalizing the similarity to obtain an attention weight; The attention weight matrix is ​​generated according to the attention weight and the vector dimension of each node in the feature image graph.

7. The medical image segmentation method based on structural constraints according to claim 1, characterized in that: Performing feature fusion on the target feature image and the target visual feature according to the cross attention weight matrix to obtain the second feature representation includes: weighting the target visual feature according to the cross attention weight matrix to obtain a weighted visual feature representation; The weighted visual feature representation is fused into the target feature image to obtain the second feature representation.

8. The medical image segmentation method based on structural constraints according to claim 1, characterized in that: The first feature representation and the second feature representation are fused by residual connection and feature decoding is performed to obtain a segmented image corresponding to the medical image data set, including: Performing a residual connection on the first feature representation and the second feature representation to obtain a fused feature; The fused features are feature decoded to obtain a decoded feature map, and the segmented image is determined according to the decoded feature map.

9. A medical image segmentation system based on structural constraints, characterized in that: The system comprises: A feature extraction module, used for performing global feature extraction on the medical image data set to obtain a first feature representation; An image map and visual feature generation module, used for performing encoding and decoding and deep convolution processing on the first feature representation to obtain a feature image map and a visual feature; An image graph embedding module, used for generating features of the feature image graph and the visual features by using a self-attention mechanism, and performing feature fusion on the generated features by using a cross-attention mechanism to obtain a second feature representation; A decoding and segmentation module, configured to fuse the first feature representation and the second feature representation through a residual connection and perform feature decoding to obtain a segmented image corresponding to the medical image dataset; The feature image graph is a directed graph or an undirected graph consisting of nodes and edges, and the feature image graph uses vectors to represent nodes and relationships; nodes represent different target areas, and edges represent constraint relationships between areas; The feature image and the visual feature are generated by using a self-attention mechanism, and the generated features are fused by using a cross-attention mechanism to obtain a second feature representation, including: Perform feature updating according to the node similarity between each node in the feature image graph and the feature similarity between each feature in the visual feature, to obtain a target feature image graph and a target visual feature; Calculating the target similarity between the target feature image and the target visual feature; Generate a cross-attention weight matrix according to the target similarity; The target feature image and the target visual feature are subjected to feature fusion according to the cross-attention weight matrix to obtain the second feature representation.

Images