Image processing method and device, terminal equipment and storage medium

Abstract

The application discloses an image processing method and device, a terminal equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring an image to be processed; processing the image to be processed based on a pre-trained multi-head attention mechanism model to obtain an image block segmentation result; learning the image block segmentation result based on a pre-trained deep bidirectional learning model to obtain a primary segmentation image; and processing the primary segmentation image by using a post-processing algorithm to obtain a target segmentation image. The image to be processed is processed by using the multi-head attention mechanism model to obtain the image block segmentation result, and then the deep bidirectional learning model is used to learn the image block segmentation result to obtain the primary segmentation image. In this way, the adjacent image blocks are connected, the rationality and integrity of the entire segmentation result are improved, and then the post-processing algorithm is used to process the primary segmentation image to obtain the target segmentation image, so that the accuracy of image segmentation is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an image processing method, apparatus, terminal device, and storage medium. Background Technology

[0002] With the rapid development of medical imaging equipment, doctors can use CTA and MRA to image blood vessels in various parts of a patient's body. However, manually analyzing vascular lesions from massive amounts of image data is obviously a very time-consuming and labor-intensive task for radiologists. In recent years, with the development of deep learning, automatic blood vessel segmentation methods based on convolutional neural networks (CNNs) have achieved significant results in analyzing vascular images. However, due to the locality of convolution operations, CNN-based methods struggle to learn global contextual information and long-distance spatial dependencies. Furthermore, since direct processing of 3D medical data is very computationally intensive, each image patch is often processed separately and then integrated into the final segmentation result. However, this approach does not consider the interdependencies between adjacent image patches, thus failing to accurately segment complete blood vessels.

[0003] Therefore, it is necessary to propose a solution to improve the accuracy of image segmentation.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide an image processing method, apparatus, terminal device, and storage medium, which aims to improve the accuracy of image segmentation.

[0006] To achieve the above objectives, the present invention provides an image processing method, the image processing method comprising:

[0007] Obtain the image to be processed;

[0008] The image to be processed is processed based on a pre-trained multi-head attention mechanism model to obtain image block segmentation results;

[0009] The image patch segmentation results are learned based on a pre-trained deep bidirectional learning model to obtain an initial segmented image;

[0010] The initial segmented image is processed by a post-processing algorithm to obtain the target segmented image.

[0011] Optionally, the step of acquiring the image to be processed includes:

[0012] Obtain the original image;

[0013] The original image is normalized to obtain a standardized image;

[0014] The standardized image is cropped to grayscale to obtain the image to be processed.

[0015] Optionally, the multi-head attention mechanism model includes an encoder and a decoder, and the encoder and the decoder use skip connections and an attention mechanism for feature fusion. Before the step of processing the image to be processed based on the pre-trained multi-head attention mechanism model to obtain the image patch segmentation result, the model further includes:

[0016] Obtain sample images and their corresponding ground truth labels;

[0017] The sample image is input into the encoder for abstract feature extraction to obtain fused abstract features;

[0018] The fused abstract features are passed through the decoder layer by layer to obtain the corresponding probability map;

[0019] Calculate the loss of the probability map output by each layer in the decoder with respect to the corresponding real label to obtain the total loss;

[0020] This process is repeated to iterate the parameters until the total loss converges, thus obtaining the multi-head attention mechanism model.

[0021] Optionally, the deep bidirectional learning model includes sequence learning layers, convolutional layers, and logistic regression layers. Before the step of learning the image patch segmentation results based on the pre-trained deep bidirectional learning model to obtain the initial segmented image, the model further includes:

[0022] Obtain the sample image and its corresponding real label;

[0023] The feature sequences in the sample images are sequentially input into a sequence learning layer for concatenation and fusion to obtain the first learning information;

[0024] The segmentation probability map is obtained based on the first learning information through the convolutional layer and the logistic regression layer;

[0025] Calculate the loss of the segmentation probability map with respect to the corresponding true label to obtain the prediction loss;

[0026] This process is repeated to iterate the parameters until the prediction loss converges, thus obtaining the deep bidirectional learning model.

[0027] Optionally, the post-processing algorithm includes a connected component volume post-processing algorithm and / or a connected component distance post-processing algorithm, and the step of processing the initial segmented image using the post-processing algorithm to obtain the target segmented image includes:

[0028] The initial segmented image is post-processed using the connected component volume post-processing algorithm to obtain a volume-post-processed image.

[0029] The target segmentation image is obtained by performing distance post-processing on the volume post-processed image using the connected component distance post-processing algorithm.

[0030] Optionally, the step of performing volume post-processing on the initial segmented image using the connected component volume post-processing algorithm to obtain a volume post-processed image includes:

[0031] Obtain each connected component in the initial segmented image and calculate the volume of each connected component in the initial segmented image;

[0032] Calculate the first volume of each connected component based on the volume of each connected component in the initial segmented image;

[0033] The rejection rate of each connected component in the initial segmentation image is calculated based on the volume of each connected component and the first volume of the connected components.

[0034] The connected components in the initial segmented image with a rejection rate less than a preset rejection rate threshold are cleared to obtain the volume post-processed image.

[0035] Optionally, the step of performing distance post-processing on the volume post-processed image using the connected component distance post-processing algorithm to obtain the target segmentation image includes:

[0036] Obtain each connected component of the volume post-processed image, and calculate the volume of each connected component in the volume post-processed image;

[0037] Starting connected components are selected from each connected component in the volume post-processed image based on preset rules.

[0038] Calculate the representative coordinates of the initial connected component, and calculate the distance values ​​from each connected component in the volume post-processed image to the initial connected component in sequence based on the representative coordinates of the initial connected component;

[0039] The connected components in the volume post-processed image whose distance value is greater than a preset distance threshold are cleared to obtain the target segmentation image.

[0040] Furthermore, to achieve the above objectives, the present invention also provides an image processing apparatus, the image processing apparatus comprising:

[0041] The acquisition module is used to acquire the image to be processed;

[0042] The image patch segmentation module is used to process the image to be processed based on a pre-trained multi-head attention mechanism model to obtain the image patch segmentation result;

[0043] The sequence learning module is used to learn the image patch segmentation results based on a pre-trained deep bidirectional learning model to obtain an initial segmented image;

[0044] The post-processing module is used to process the initial segmented image using a post-processing algorithm to obtain the target segmented image.

[0045] In addition, to achieve the above objectives, the present invention also provides a terminal device, the terminal device including a memory, a processor, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the steps of the image processing method as described above.

[0046] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an image processing program, which, when executed by a processor, implements the steps of the image processing method described above.

[0047] This invention proposes an image processing method, apparatus, terminal device, and storage medium. The method involves: acquiring an image to be processed; processing the image to be processed based on a pre-trained multi-head attention mechanism model to obtain image patch segmentation results; learning the image patch segmentation results based on a pre-trained deep bidirectional learning model to obtain an initial segmented image; and processing the initial segmented image using a post-processing algorithm to obtain a target segmented image. By using a multi-head attention mechanism model to process the image to be processed and obtain image patch segmentation results, and then using a deep bidirectional learning model to learn the image patch segmentation results to obtain the initial segmented image, connections can be established between adjacent image patches, improving the rationality and completeness of the overall segmentation result. Finally, a post-processing algorithm is used to process the initial segmented image to obtain the target segmented image, thereby improving the accuracy of image segmentation. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the image processing apparatus of the present invention belongs;

[0049] Figure 2 This is a flowchart illustrating an exemplary embodiment of the image processing method of the present invention;

[0050] Figure 3 This is a schematic diagram of the preprocessed image in an embodiment of the present invention;

[0051] Figure 4 This is a schematic flowchart illustrating another exemplary embodiment of the image processing method of the present invention;

[0052] Figure 5This is a schematic diagram of the structure of the multi-head attention mechanism model in an embodiment of the present invention;

[0053] Figure 6 This is a flowchart illustrating another exemplary embodiment of the image processing method of the present invention;

[0054] Figure 7 This is a schematic diagram of the first structure of the deep bidirectional learning model in an embodiment of the present invention;

[0055] Figure 8 This is a schematic diagram of the second structure of the deep bidirectional learning model in an embodiment of the present invention;

[0056] Figure 9 for Figure 2 A detailed flowchart of step S40 in the embodiment;

[0057] Figure 10 This is a schematic diagram illustrating the overall principle of an embodiment of the present invention;

[0058] Figure 11 This is a schematic diagram of a real label in an embodiment of the present invention;

[0059] Figure 12 This is a schematic diagram of the initial segmentation result mask0 in an embodiment of the present invention;

[0060] Figure 13 This is a schematic diagram of the connected component volume post-processing result mask1 in an embodiment of the present invention;

[0061] Figure 14 This is a schematic diagram of the mask2 result, which is the post-processing result of the connected component distance in an embodiment of the present invention.

[0062] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0063] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0064] The main solution of this invention is as follows: First, acquire the image to be processed; then, process the image to be processed based on a pre-trained multi-head attention mechanism model to obtain image patch segmentation results; finally, learn from the image patch segmentation results based on a pre-trained deep bidirectional learning model to obtain an initial segmentation image; and finally, process the initial segmentation image using a post-processing algorithm to obtain the target segmentation image. By processing the image to be processed using a multi-head attention mechanism model to obtain image patch segmentation results, and then using a deep bidirectional learning model to learn from the image patch segmentation results to obtain the initial segmentation image, connections can be established between adjacent image patches, improving the rationality and completeness of the entire segmentation result. Finally, a post-processing algorithm is used to process the initial segmentation image to obtain the target segmentation image, thereby improving the accuracy of image segmentation.

[0065] Technical terms involved in the embodiments of this invention:

[0066] CTA: Computed Tomography angiography;

[0067] MRA: Magnetic resonance angiography;

[0068] Encoder / Decoder: Encoder / Decoder;

[0069] U-Net: A deep learning network architecture, mainly consisting of an Encoder and a Decoder.

[0070] CNN: Convolutional Neural Network, a type of deep learning network architecture;

[0071] C-RNN / C-LSTM / C-GRU: Convolutional Recurrent Neural Network / Convolutional Long Short-Term Memory Network / Convolutional Gated Recurrent Neural Network;

[0072] Transformer: A deep learning network architecture;

[0073] DICOM: Medical Digital Imaging and Communication, is an international standard for medical images and related information.

[0074] Currently, vascular diseases have become a major cause of death, seriously threatening human life and health. With the rapid development of medical imaging equipment, doctors can use CTA and MRA to image blood vessels in various parts of a patient's body. However, manually analyzing vascular lesions from massive amounts of image data is obviously a very time-consuming and labor-intensive task for radiologists. In recent years, with the development of deep learning, 1. Automatic blood vessel segmentation methods based on convolutional neural networks (CNNs) have achieved significant results in analyzing vascular images. However, due to the locality of convolution operations, CNN-based methods have difficulty learning global contextual information and long-distance spatial dependencies. In addition, since direct processing of 3D medical data is very computationally intensive, each image patch is often processed separately and then integrated into the final segmentation result. However, this approach does not consider the interdependencies between adjacent image patches, thus failing to accurately segment complete blood vessels. Specifically, current blood vessel segmentation techniques have the following shortcomings:

[0075] 1. Traditional blood vessel segmentation techniques are inefficient, have high labor costs, and produce poor segmentation results;

[0076] 2. Most existing automatic blood vessel segmentation technologies are based on CNN models to extract image features, which cannot capture the global information of the input data and long-distance spatial dependencies well;

[0077] 3. Some existing automated vessel segmentation techniques use the same weights to learn the background and all vessels, resulting in poor segmentation of vessels with large differences in morphology and size. For example, the morphology and size of vessels from the aorta to the intracranial vessels in head and neck CTA are very different.

[0078] 4. Some existing automatic vessel segmentation techniques process each image block individually, ignoring the connections between adjacent image blocks. For example, the vascular structure in head and neck CTA is a spatial structure from the aorta to the carotid artery and then to the intracranial vessels;

[0079] 5. Some existing automated blood vessel segmentation technologies lack comprehensive and universal post-processing operations.

[0080] This invention provides a solution that acquires raw medical image data and performs data preprocessing (including grayscale cropping and data normalization) to crop each data point into several image patches along the depth direction. Next, a Transformer and CNN-based model is used to train each image patch, including an attention-based module. The output features of the penultimate convolutional layer of the decoder are obtained from the trained model, and these features are sequentially learned using a deep bidirectional C-LSTM (C-RNN or C-GRU) method to obtain the initial segmentation result of the entire blood vessel. Finally, connected component post-processing (including connected component volume post-processing and connected component distance post-processing) is applied to the initial segmentation result of the entire blood vessel to obtain the final segmentation result. Compared to traditional blood vessel segmentation methods, this solution achieves automatic blood vessel segmentation without manual intervention, and the combination of deep learning technology achieves higher segmentation accuracy. Compared to existing automatic blood vessel segmentation methods, this solution provides more effective blood vessel segmentation and comprehensive image post-processing techniques.

[0081] Specifically, refer to Figure 1 , Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the image processing apparatus of the present invention belongs. The image processing apparatus can be an independent device capable of image processing, and it can be carried on the terminal device in hardware or software form. The terminal device can be a smart mobile terminal with data processing capabilities, such as a mobile phone or tablet computer, or it can be a fixed terminal device or server with data processing capabilities.

[0082] In this embodiment, the terminal device to which the image processing device belongs includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.

[0083] The memory 130 stores the operating system and image processing program. The image processing device can store information such as the acquired image to be processed, the image block segmentation result obtained by processing the image to be processed based on a pre-trained multi-head attention mechanism model, the initial segmented image obtained by learning the image block segmentation result based on a pre-trained deep bidirectional learning model, and the target segmented image obtained by processing the initial segmented image through a post-processing algorithm in the memory 130. The output module 110 can be a display screen, etc. The communication module 140 can include a WIFI module, a mobile communication module, and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0084] When the image processing program in memory 130 is executed by the processor, it performs the following steps:

[0085] Obtain the image to be processed;

[0086] The image to be processed is processed based on a pre-trained multi-head attention mechanism model to obtain image block segmentation results;

[0087] The image patch segmentation results are learned based on a pre-trained deep bidirectional learning model to obtain an initial segmented image;

[0088] The initial segmented image is processed by a post-processing algorithm to obtain the target segmented image.

[0089] Furthermore, when the image processing program in memory 130 is executed by the processor, it also performs the following steps:

[0090] Obtain the original image;

[0091] The original image is normalized to obtain a standardized image;

[0092] The standardized image is cropped to grayscale to obtain the image to be processed.

[0093] Furthermore, when the image processing program in memory 130 is executed by the processor, it also performs the following steps:

[0094] Obtain sample images and their corresponding ground truth labels;

[0095] The sample image is input into the encoder for abstract feature extraction to obtain fused abstract features;

[0096] The fused abstract features are passed through the decoder layer by layer to obtain the corresponding probability map;

[0097] Calculate the loss of the probability map output by each layer in the decoder with respect to the corresponding real label to obtain the total loss;

[0098] This process is repeated to iterate the parameters until the total loss converges, thus obtaining the multi-head attention mechanism model.

[0099] Furthermore, when the image processing program in memory 130 is executed by the processor, it also performs the following steps:

[0100] Obtain the sample image and its corresponding real label;

[0101] The feature sequences in the sample images are sequentially input into a sequence learning layer for concatenation and fusion to obtain the first learning information;

[0102] The segmentation probability map is obtained based on the first learning information through the convolutional layer and the logistic regression layer;

[0103] Calculate the loss of the segmentation probability map with respect to the corresponding true label to obtain the prediction loss;

[0104] This process is repeated to iterate the parameters until the prediction loss converges, thus obtaining the deep bidirectional learning model.

[0105] Furthermore, when the image processing program in memory 130 is executed by the processor, it also performs the following steps:

[0106] The initial segmented image is post-processed using the connected component volume post-processing algorithm to obtain a volume-post-processed image.

[0107] The target segmentation image is obtained by performing distance post-processing on the volume post-processed image using the connected component distance post-processing algorithm.

[0108] Furthermore, when the image processing program in memory 130 is executed by the processor, it also performs the following steps:

[0109] Obtain each connected component in the initial segmented image and calculate the volume of each connected component in the initial segmented image;

[0110] Calculate the first volume of each connected component based on the volume of each connected component in the initial segmented image;

[0111] The rejection rate of each connected component in the initial segmentation image is calculated based on the volume of each connected component and the first volume of the connected components.

[0112] The connected components in the initial segmented image with a rejection rate less than a preset rejection rate threshold are cleared to obtain the volume post-processed image.

[0113] Furthermore, when the image processing program in memory 130 is executed by the processor, it also performs the following steps:

[0114] Obtain each connected component of the volume post-processed image, and calculate the volume of each connected component in the volume post-processed image;

[0115] Starting connected components are selected from each connected component in the volume post-processed image based on preset rules.

[0116] Calculate the representative coordinates of the initial connected component, and calculate the distance values ​​from each connected component in the volume post-processed image to the initial connected component in sequence based on the representative coordinates of the initial connected component;

[0117] The connected components in the volume post-processed image whose distance value is greater than a preset distance threshold are cleared to obtain the target segmentation image.

[0118] This embodiment, through the above-described scheme, specifically involves: acquiring the image to be processed; processing the image to be processed based on a pre-trained multi-head attention mechanism model to obtain image patch segmentation results; learning the image patch segmentation results based on a pre-trained deep bidirectional learning model to obtain an initial segmented image; and processing the initial segmented image using a post-processing algorithm to obtain the target segmented image. By using a multi-head attention mechanism model to process the image to be processed and obtain image patch segmentation results, and then using a deep bidirectional learning model to learn the image patch segmentation results to obtain the initial segmented image, connections can be established between adjacent image patches, improving the rationality and completeness of the overall segmentation result. Finally, a post-processing algorithm is used to process the initial segmented image to obtain the target segmented image, thereby improving the accuracy of image segmentation.

[0119] Based on, but not limited to, the terminal device architecture described above, embodiments of the method of the present invention are proposed.

[0120] The subject executing the method in this embodiment can be an image processing device or a terminal device, etc. This embodiment takes an image processing device as an example.

[0121] Reference Figure 2 , Figure 2 This is a schematic flowchart illustrating an exemplary embodiment of the image processing method of the present invention. The image processing method includes:

[0122] Step S10: Obtain the image to be processed;

[0123] Specifically, in this embodiment of the application, the original medical image data is used and image preprocessing is performed to obtain the image to be processed. The specific process includes:

[0124] Obtain the original image;

[0125] The original image is normalized to obtain a standardized image;

[0126] The standardized image is cropped to grayscale to obtain the image to be processed.

[0127] The original medical imaging data used in the embodiments of this application include vascular image data such as CTA (Computed Tomography Angiography) and MRA (Magnetic Resonance Angiography). This type of vascular image data is usually in DICOM format and is injected with contrast agents to make the blood vessels clearer.

[0128] Optionally, after acquiring the original image, the window width and level are set to crop the grayscale range of the medical image, and the maximum and minimum values ​​of the image are normalized to between [0,1] to obtain a standardized image. The preprocessed 3D data is then randomly cropped into image blocks of the same size along the depth direction. (Refer to...) Figure 3 , Figure 3This is a schematic diagram of the preprocessed image in an embodiment of the present invention, such as... Figure 3 As shown, taking head and neck CTA data as an example, the images can be made clearer after preprocessing.

[0129] Step S20: The image to be processed is processed based on a pre-trained multi-head attention mechanism model to obtain image patch segmentation results;

[0130] Furthermore, in this embodiment, a pre-trained multi-head attention mechanism model is used. The framework of the multi-head attention mechanism model consists of a transformer as an encoder and a CNN-based decoder. The encoder and decoder use skip connections and attention mechanisms to fuse features. The image to be processed is input into the transformer module to extract abstract features. The features are then fused with the features of the corresponding layer of the decoder through skip connections based on the attention mechanism. The fused abstract features are passed layer by layer through the CNN-based decoder to predict the corresponding probability map, thereby obtaining the image patch segmentation result.

[0131] Step S30: The image patch segmentation results are learned based on a pre-trained deep bidirectional learning model to obtain an initial segmented image;

[0132] Furthermore, in this embodiment, a pre-trained deep bidirectional learning model is used, which is not limited to an LSTM sequence model. The deep bidirectional C-LSTM model internally includes several sub-modules BDC-LSTM, which are composed of two stacked C-LSTM layers, each learning z. + and z - Contextual information of direction (z is along the depth direction of the 3D data, z + and z - (This refers to two opposite directions). Next, the information of the current sequence z and the learned context information are concatenated and fused, and then input into the next BDC-LSTM to learn other sequences. Finally, a 3D segmentation probability map is obtained through convolutional layers and softmax layers, thus obtaining the initial segmentation image.

[0133] Step S40: The initial segmented image is processed by a post-processing algorithm to obtain the target segmented image.

[0134] Furthermore, in this embodiment of the application, a post-processing algorithm is used to post-process the initially segmented image. The post-processing algorithm includes a connected component volume post-processing algorithm and / or a connected component distance post-processing algorithm. The two algorithms can be used as a cascaded processing operation, that is, the output of the connected component volume post-processing can be used as the input of the connected component distance post-processing.

[0135] It should be noted that the method of performing connected component volume post-processing followed by connected component distance post-processing in this embodiment does not constitute a limitation on the post-processing method. In other embodiments, connected component distance post-processing can be performed first, followed by connected component volume post-processing, or connected component volume post-processing or connected component distance post-processing can be performed on the image separately.

[0136] In this embodiment, the process involves acquiring an image to be processed; processing the image to be processed based on a pre-trained multi-head attention mechanism model to obtain image patch segmentation results; learning the image patch segmentation results based on a pre-trained deep bidirectional learning model to obtain an initial segmentation image; and processing the initial segmentation image using a post-processing algorithm to obtain the target segmentation image. By processing the image to be processed using a multi-head attention mechanism model to obtain image patch segmentation results, and then using a deep bidirectional learning model to learn the image patch segmentation results to obtain the initial segmentation image, connections can be established between adjacent image patches, improving the rationality and completeness of the overall segmentation result. Finally, a post-processing algorithm is used to process the initial segmentation image to obtain the target segmentation image, thereby improving the accuracy of image segmentation.

[0137] Reference Figure 4 , Figure 4 This is a schematic flowchart illustrating another exemplary embodiment of the image processing method of the present invention. Based on the above... Figure 2 In the embodiment shown, prior to step S20, the image processing method further includes:

[0138] Step S01: Train the multi-head attention mechanism model. In this embodiment, step S01 can be implemented before step S10. In other embodiments, step S01 can also be implemented between step S10 and step S20.

[0139] Compared to the above Figure 2 The embodiment shown also includes a scheme for training the multi-head attention mechanism model.

[0140] Specifically, the steps for training the multi-head attention mechanism model may include:

[0141] Step S011: Obtain the sample image and its corresponding real label;

[0142] Reference Figure 5 , Figure 5 This is a schematic diagram of the multi-head attention mechanism model in an embodiment of the present invention, as shown below. Figure 5As shown, in one implementation, the model framework consists of a transformer as an encoder and a CNN-based decoder. The large amount of raw image data collected is preprocessed to obtain sample images. Once the sample images and their corresponding real labels are obtained, they can be used for model training.

[0143] Step S012: Input the sample image into the encoder for abstract feature extraction to obtain fused abstract features;

[0144] The sample image is input into the transformer module to extract abstract features, which are then fused with the features of the corresponding layer of the decoder through skip connections using an attention-based mechanism. The mathematical formula for the attention mechanism is as follows, where f e ,f d These represent the features output by the encoder and decoder at the corresponding positions, with + and * indicating pixel-wise addition and multiplication.

[0145] Atten(f e ,f d )=f d *(Sigmoid(ReLU(f e +f d )))

[0146] Step S013: Pass the fused abstract features through the decoder layer by layer to obtain the corresponding probability map;

[0147] The fused abstract features are passed layer by layer through a CNN-based decoder to predict the corresponding probability map.

[0148] Step S014: Calculate the loss of the probability map output by each layer in the decoder with respect to the corresponding real label, and obtain the total loss;

[0149] Deep supervision is used to calculate the loss, where the loss of the predicted output probability map with respect to the true label is calculated for each layer of the decoder. The losses of each layer are weighted to obtain the total loss, and the learning parameters of the deep model are updated through backpropagation.

[0150] Step S015, repeat this process to iterate the parameters until the total loss converges, thus obtaining the multi-head attention mechanism model.

[0151] Through learning and iteration on a large amount of labeled data, the prediction loss gradually approaches zero, thus obtaining the final multi-head attention mechanism model.

[0152] This embodiment employs the above-described scheme, specifically by acquiring sample images and their corresponding ground truth labels; inputting the sample images into the encoder for abstract feature extraction to obtain fused abstract features; passing the fused abstract features layer by layer through the decoder to obtain corresponding probability maps; calculating the loss of each layer's output probability map with respect to its corresponding ground truth label to obtain the total loss; and iterating the parameters in this manner until the total loss converges to obtain the multi-head attention mechanism model. By learning and iterating on a large amount of labeled data using the model framework, which includes the encoder and decoder, the model's prediction results become more accurate, thus enabling it to process images and obtain more accurate image patch segmentation results.

[0153] Reference Figure 6 , Figure 6 This is a schematic flowchart illustrating another exemplary embodiment of the image processing method of the present invention. Based on the above... Figure 2 In the embodiment shown, prior to step S30, the image processing method further includes:

[0154] Step S02: Train the deep bidirectional learning model. In this embodiment, step S01 can be implemented before step S10. In other embodiments, step S00 can also be implemented between step S10 and step S20, or between step S20 and step S30.

[0155] Compared to the above Figure 2 The embodiment shown also includes a scheme for training the deep bidirectional learning model.

[0156] Specifically, the steps for training the deep bidirectional learning model may include:

[0157] Step S021: Obtain the sample image and its corresponding real label;

[0158] Reference Figure 7 and Figure 8 , Figure 7 This is a schematic diagram of the first structure of the deep bidirectional learning model in an embodiment of the present invention. Figure 8 This is a schematic diagram of the second structure of the deep bidirectional learning model in an embodiment of the present invention, as shown below. Figure 7 As shown, the embodiments of this application employ a deep bidirectional C-LSTM model (not limited to LSTM sequence models), such as... Figure 8 As shown, the BDC-LSTM submodule within the deep bidirectional C-LSTM model is composed of two stacked C-LSTM layers. Preprocessing the large amount of raw image data collected yields sample images. Once the sample images and their corresponding ground truth labels are obtained, they can be used for model training.

[0159] Step S022: The feature sequences in the sample images are sequentially input into the sequence learning layer for splicing and fusion to obtain the first learning information;

[0160] Specifically, BDC-LSTM is composed of two stacked C-LSTM layers, which learn z respectively. + and z - Contextual information about the direction (z is along the depth direction of the 3D data, z) + and z - (This refers to two opposite directions). Then, the information of the current sequence z and the learned context information are concatenated and fused to obtain the first learned information.

[0161] Step S023: Obtain a segmentation probability map based on the first learning information through the convolutional layer and logistic regression layer;

[0162] Furthermore, the first learned information obtained by splicing and fusing is input into the next BDC-LSTM to learn other sequences. Finally, a 3D segmentation probability map is obtained through convolutional layers and softmax layers.

[0163] Step S024: Calculate the loss of the segmentation probability map with respect to the corresponding true label to obtain the prediction loss;

[0164] The calculated loss between the probability map and the true label, i.e. the prediction loss, is used for parameter iteration.

[0165] Step S025, repeat this process to iterate the parameters until the prediction loss converges, thus obtaining the deep bidirectional learning model.

[0166] By backpropagating, the network parameters are updated until the prediction loss gradually approaches zero, thus obtaining the final deep bidirectional learning model.

[0167] This embodiment, through the above scheme, specifically acquires the sample image and its corresponding ground truth label; sequentially inputs the feature sequences from the sample image into a sequence learning layer for concatenation and fusion to obtain first learning information; obtains a segmentation probability map based on the first learning information through the convolutional layer and logistic regression layer; calculates the loss of the segmentation probability map with respect to the corresponding ground truth label to obtain the prediction loss; and iterates the parameters in this way until the prediction loss converges to obtain the deep bidirectional learning model. By training the deep bidirectional C-LSTM model, a deep bidirectional learning model that can accurately learn from the image patch segmentation results to obtain the initial segmented image is finally obtained. This enables the establishment of connections between adjacent image patches, improves the rationality and completeness of the entire segmentation result, and thus improves the accuracy of image segmentation.

[0168] Reference Figure 9 , Figure 9for Figure 2 A schematic flowchart of step S40 in this embodiment. This embodiment is based on the above. Figure 2 In the embodiment shown, step S40 includes:

[0169] Step S401: Perform volume post-processing on the initial segmented image using the connected component volume post-processing algorithm to obtain a volume post-processed image;

[0170] Specifically, the step of performing volume post-processing on the initial segmented image using a connected component volume post-processing algorithm to obtain a volume post-processed image includes:

[0171] Obtain each connected component in the initial segmented image and calculate the volume of each connected component in the initial segmented image;

[0172] Calculate the first volume of each connected component based on the volume of each connected component in the initial segmented image;

[0173] The rejection rate of each connected component in the initial segmentation image is calculated based on the volume of each connected component and the first volume of the connected components.

[0174] The connected components in the initial segmented image with a rejection rate less than a preset rejection rate threshold are cleared to obtain the volume post-processed image.

[0175] In this embodiment of the application, all connected components of the initial segmentation result mask0 are obtained, and the volume and V of all connected components are calculated. t ;

[0176] Get the volume v of the i-th connected component i And calculate the rejection rate r. reject The formula is as follows:

[0177] r reject =v i / v t

[0178] Set a threshold, such as 0.1, or other reasonable thresholds. If r reject If <0.1, then the connected component in the initial segmentation result mask0 is cleared;

[0179] Repeat the above operation until each connected component is traversed, and the segmentation result mask1 after processing the volume of the connected component is obtained.

[0180] Step S402: Perform distance post-processing on the volume post-processing image using the connected component distance post-processing algorithm to obtain the target segmentation image.

[0181] Specifically, the step of performing distance post-processing on the volume post-processed image using the connected component distance post-processing algorithm to obtain the target segmentation image includes:

[0182] Obtain each connected component of the volume post-processed image, and calculate the volume of each connected component in the volume post-processed image;

[0183] Starting connected components are selected from each connected component in the volume post-processed image based on preset rules.

[0184] Calculate the representative coordinates of the initial connected component, and calculate the distance values ​​from each connected component in the volume post-processed image to the initial connected component in sequence based on the representative coordinates of the initial connected component;

[0185] The connected components in the volume post-processed image whose distance value is greater than a preset distance threshold are cleared to obtain the target segmentation image.

[0186] In this embodiment of the application, all connected components c of the segmentation result mask1 are obtained. i And calculate the volume v of all connected components. i Select the connected component c with the largest volume. j As the initial connected component; calculate c j The endpoints in the three directions [[x js x je ],[y js y je ],[z js , z je (The subscript 's' indicates the starting point, and 'e' indicates the ending point), connected component c j The representative coordinates of the coordinates j Given by the following formula:

[0187]

[0188] Get the i-th connected component c i The representative coordinates of the coordinates i The connected component c with the largest volume is calculated using the following formula. j Distance value d i :

[0189]

[0190] Set a distance threshold, such as 1e4, or other reasonable thresholds. If d i If the value is greater than 1e4, then the connected component in the initial segmentation result mask1 is cleared.

[0191] Repeat the above steps until all connected components have been traversed and the segmentation result mask2 after processing the connected component distance is obtained, which is the target segmentation image in this embodiment.

[0192] This embodiment employs the above-described scheme, specifically using a connected component volume post-processing algorithm to perform volume post-processing on the initial segmented image, resulting in a volume-post-processed image; and then using a connected component distance post-processing algorithm to perform distance post-processing on the volume-post-processed image, resulting in the target segmented image. By using post-processing algorithms to process the initial segmented image and obtain the target segmented image, the accuracy of image segmentation is improved.

[0193] Furthermore, embodiments of the present invention also propose an image processing apparatus, the image processing apparatus comprising:

[0194] The acquisition module is used to acquire the image to be processed;

[0195] The image patch segmentation module is used to process the image to be processed based on a pre-trained multi-head attention mechanism model to obtain the image patch segmentation result;

[0196] The sequence learning module is used to learn the image patch segmentation results based on a pre-trained deep bidirectional learning model to obtain an initial segmented image;

[0197] The post-processing module is used to process the initial segmented image using a post-processing algorithm to obtain the target segmented image.

[0198] Reference Figure 10 , Figure 10 This is a schematic diagram illustrating the overall principle of an embodiment of the present invention, such as... Figure 10 As shown, the main implementation steps include:

[0199] A. Acquiring raw medical imaging data

[0200] CTA and MRA vascular image data are typically in DICOM format and are infused with contrast agents to make the vessels clearer.

[0201] B. Data Preprocessing

[0202] Set the window width and window level to crop the grayscale range of the medical image and normalize the maximum and minimum values ​​of the image to between [0,1] to obtain a standardized image. Then, randomly crop the preprocessed 3D data into image blocks of the same size along the depth direction.

[0203] C. The image patch segmentation result is obtained using a Transformer-CNN multi-head attention mechanism model. The model framework consists of a transformer as the encoder and a CNN-based decoder. The encoder and decoder use skip connections and an attention mechanism for feature fusion. The specific steps are as follows:

[0204] (1) Input the image patch into the transformer module to extract abstract features, and perform feature fusion with the features of the corresponding layer of the decoder through skip connections. The mathematical formula of the attention mechanism is as follows, where f e ,f d These represent the features output by the encoder and decoder at the corresponding positions, with + and * indicating pixel-wise addition and multiplication.

[0205] Atten(f e ,f d )=f d *(Sigmoid(ReLU(f e +f d )))

[0206] (2) The fused abstract features are passed through a CNN-based decoder layer by layer to predict the corresponding probability map;

[0207] (3) Deep supervision is used to calculate the loss. For each layer of the decoder, the loss of the probability map of the predicted output with respect to the true label is calculated. The loss of each layer is weighted to obtain the total loss, and the learning parameters of the deep model are updated through backpropagation.

[0208] (4) Through learning and iteration with a large amount of labeled data, the prediction loss gradually approaches zero, thus obtaining the final deep model.

[0209] D. Use a deep bidirectional C-LSTM to learn the feature sequences of image patches and obtain the initial segmentation results of the entire image.

[0210] The deep bidirectional C-LSTM model used (not limited to LSTM sequence models) has a sub-module called BDC-LSTM. BDC-LSTM is composed of two stacked C-LSTM layers, each learning z... + and z - Contextual information of direction (z is along the depth direction of the 3D data, z + and z -(This refers to two opposite directions). Next, the information from the current sequence z and the learned context information are concatenated and fused, and then input into the next BDC-LSTM for learning other sequences. Finally, a 3D segmentation probability map is obtained through convolutional layers and softmax layers, and the loss relative to the true label is calculated. Through backpropagation, the network parameters are updated until the prediction loss gradually approaches zero, thus obtaining the final model.

[0211] E. Apply image post-processing algorithms to the initial segmentation results to obtain the final segmentation results.

[0212] The post-processing algorithm proposed in this invention includes connected component volume post-processing and connected component distance post-processing algorithms. The two algorithms are a cascaded processing operation, that is, the output of the connected component volume post-processing is the input of the connected component distance post-processing.

[0213] The specific implementation steps for post-processing of connected component volume are as follows:

[0214] 1) Obtain all connected components of the initial segmentation result mask0, and calculate the volume V of all connected components. t ;

[0215] 2) Obtain the volume v of the i-th connected component. i And calculate the rejection rate r. reject The formula is as follows:

[0216] r reject =v i / v t

[0217] 3) Set a threshold, such as 0.1, or other reasonable thresholds. If r reject If <0.1, then the connected component in the initial segmentation result mask0 is cleared;

[0218] 4) Repeat steps 2)-3) until each connected component has been processed, and obtain the segmentation result mask1 after processing the volume of the connected components.

[0219] The specific implementation steps of connected component distance post-processing are as follows:

[0220] 1) Obtain all connected components c of the segmentation result mask1 i And calculate the volume v of all connected components. i Select the connected component c with the largest volume. j As the initial connected component; calculate c j The endpoints in the three directions [[x js x je ],[y js y je ],[z js , zje (The subscript 's' indicates the starting point, and 'e' indicates the ending point), connected component c j The representative coordinates of the coordinates j Given by the following formula:

[0221]

[0222] 2) Obtain the i-th connected component c i The representative coordinates of the coordinates i The connected component c with the largest volume is calculated using the following formula. j Distance value d i :

[0223]

[0224] 3) Set a distance threshold, such as 1e4, or other reasonable thresholds. If d i If the value is greater than 1e4, then the connected component in the initial segmentation result mask1 is cleared.

[0225] 4) Repeat steps 2)-3) until each connected component has been processed, and obtain the segmentation result mask2 after processing the connected component distance.

[0226] Reference Figure 11 , 12 13 and 14 Figure 11 This is a schematic diagram of a real label in an embodiment of the present invention. Figure 12 This is a schematic diagram of the initial segmentation result mask0 in an embodiment of the present invention. Figure 13 This is a schematic diagram of the mask1 result, which is the post-processing result of the connected component volume in an embodiment of the present invention. Figure 14 This is a schematic diagram of the connected component distance post-processing result mask2 in an embodiment of the present invention. The numbers below represent the segmentation accuracy. The comparison shows that the connected component distance post-processing result mask2 has the highest segmentation accuracy.

[0227] Compared to traditional vessel segmentation techniques that require manual feature extraction, resulting in high costs, low efficiency, and inaccurate segmentation results, this solution employs an automatic segmentation method based on deep learning networks to segment vessel images without manual intervention and with high accuracy. Most existing automatic vessel segmentation techniques use CNN-based deep models, which perform poorly in preserving global and spatial information. This solution, however, uses a transformer-based encoder, which better captures the global contextual information and long-range spatial dependencies of the input features. Some existing automatic vessel segmentation techniques lack the ability to learn local image details, leading to poor segmentation results for vessels with significant morphological and size differences. This solution uses attention mechanisms and deep supervised learning to preserve feature details, thus enabling more accurate segmentation of vessels with large morphological and size variations. Some existing automatic vessel segmentation techniques process each image patch individually, ignoring the connections between adjacent patches. This solution proposes using a deep bidirectional C-LSTM-based approach to learn image patch sequences, resulting in a more reasonable and complete segmentation outcome. Some existing automated blood vessel segmentation techniques lack comprehensive and universal post-processing operations; however, this solution proposes two cascaded post-processing operations, which makes the final segmentation results more accurate.

[0228] Furthermore, this embodiment of the invention also proposes a terminal device, which includes a memory, a processor, and an image processing program stored in the memory and executable on the processor. When the image processing program is executed by the processor, it implements the steps of the image processing method described above.

[0229] Since this image processing program employs all the technical solutions of all the foregoing embodiments when executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the foregoing embodiments, which will not be elaborated here.

[0230] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing an image processing program, which, when executed by a processor, implements the steps of the image processing method described above.

[0231] Since this image processing program employs all the technical solutions of all the foregoing embodiments when executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the foregoing embodiments, which will not be elaborated here.

[0232] Compared to existing technologies, the image processing method, apparatus, terminal device, and storage medium proposed in this invention acquire an image to be processed; process the image to be processed based on a pre-trained multi-head attention mechanism model to obtain image patch segmentation results; learn from the image patch segmentation results based on a pre-trained deep bidirectional learning model to obtain an initial segmented image; and process the initial segmented image using a post-processing algorithm to obtain a target segmented image. By processing the image to be processed using a multi-head attention mechanism model to obtain image patch segmentation results, and then using a deep bidirectional learning model to learn from the image patch segmentation results to obtain an initial segmented image, connections can be established between adjacent image patches, improving the rationality and completeness of the overall segmentation result. Finally, the post-processing algorithm is used to process the initial segmented image to obtain the target segmented image, thereby improving the accuracy of image segmentation.

[0233] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0234] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0235] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of this application.

[0236] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An image processing method, characterized in that, The image processing method includes the following steps: Obtain the image to be processed; The image to be processed is processed based on a pre-trained multi-head attention mechanism model to obtain image patch segmentation results. The multi-head attention mechanism model includes an encoder and a decoder. The encoder and the decoder use skip connections and attention mechanisms for feature fusion. The encoder is a Transformer and the decoder is a CNN. Before the step of processing the image to be processed by the pre-trained multi-head attention mechanism model to obtain the image patch segmentation result, the following steps are included: Acquire sample images and their corresponding ground truth labels; input the sample images into the encoder for abstract feature extraction to obtain fused abstract features; pass the fused abstract features layer by layer through the decoder to obtain the corresponding probability maps; calculate the loss of the probability maps output by each layer in the decoder with respect to the corresponding ground truth labels to obtain the total loss; repeat this process to iterate the parameters until the total loss converges to obtain the multi-head attention mechanism model. The image patch segmentation results are learned based on a pre-trained deep bidirectional learning model to obtain an initial segmented image; The initial segmented image is processed by a post-processing algorithm to obtain a target segmented image, including: performing volume post-processing on the initial segmented image using a connected component volume post-processing algorithm to obtain a volume post-processed image; and performing distance post-processing on the volume post-processed image using a connected component distance post-processing algorithm to obtain the target segmented image. The step of performing distance post-processing on the volume post-processed image using the connected component distance post-processing algorithm to obtain the target segmentation image includes: Obtain each connected component of the volume post-processed image, and calculate the volume of each connected component in the volume post-processed image; Based on preset rules, a starting connected component is selected from each connected component in the volume post-processed image, wherein the connected component with the largest volume is selected as the starting connected component. Calculate the representative coordinates of the initial connected component, and calculate the distance values ​​from each connected component in the volume post-processed image to the initial connected component in sequence based on the representative coordinates of the initial connected component; The connected components in the volume post-processed image whose distance value is greater than a preset distance threshold are cleared to obtain the target segmentation image.

2. The image processing method as described in claim 1, characterized in that, The steps for acquiring the image to be processed include: Obtain the original image; The original image is normalized to obtain a standardized image; The standardized image is cropped to grayscale to obtain the image to be processed.

3. The image processing method as described in claim 1, characterized in that, The deep bidirectional learning model includes sequence learning layers, convolutional layers, and logistic regression layers. Before the step of learning the image patch segmentation results based on the pre-trained deep bidirectional learning model to obtain the initial segmented image, the following steps are also included: Obtain the sample image and its corresponding real label; The feature sequences in the sample images are sequentially input into a sequence learning layer for concatenation and fusion to obtain the first learning information; The segmentation probability map is obtained based on the first learning information through the convolutional layer and the logistic regression layer; Calculate the loss of the segmentation probability map with respect to the corresponding true label to obtain the prediction loss; This process is repeated to iterate the parameters until the prediction loss converges, thus obtaining the deep bidirectional learning model.

4. The image processing method as described in claim 1, characterized in that, The step of performing volume post-processing on the initial segmented image using the connected component volume post-processing algorithm to obtain a volume post-processed image includes: Obtain each connected component in the initial segmented image and calculate the volume of each connected component in the initial segmented image; Calculate the first volume of each connected component based on the volume of each connected component in the initial segmented image; The rejection rate of each connected component in the initial segmentation image is calculated based on the volume of each connected component and the sum of the first volumes of the connected components, where the volume of each connected component is v. i The sum of the first volumes of the connected components is The rejection rate is r reject =v i / v t ; The connected components in the initial segmented image with a rejection rate less than a preset rejection rate threshold are cleared to obtain the volume post-processed image.

5. An image processing apparatus, characterized in that, The image processing device includes: The acquisition module is used to acquire the image to be processed; The image patch segmentation module is used to process the image to be processed based on a pre-trained multi-head attention mechanism model to obtain the image patch segmentation result. The multi-head attention mechanism model includes an encoder and a decoder. The encoder and the decoder use skip connections and attention mechanisms for feature fusion. The encoder is a Transformer and the decoder is a CNN. Before the step of processing the image to be processed by the pre-trained multi-head attention mechanism model to obtain the image patch segmentation result, the following steps are included: The process involves: acquiring sample images and their corresponding ground truth labels; inputting the sample images into the encoder for abstract feature extraction to obtain fused abstract features; passing the fused abstract features layer by layer through the decoder to obtain corresponding probability maps; calculating the loss of the probability maps output by each layer of the decoder with respect to their corresponding ground truth labels to obtain the total loss; and iterating the parameters in this manner until the total loss converges to obtain the multi-head attention mechanism model. The sequence learning module is used to learn the image patch segmentation results based on a pre-trained deep bidirectional learning model to obtain an initial segmented image; The post-processing module is used to process the initial segmented image using a post-processing algorithm to obtain a target segmented image, including: performing volume post-processing on the initial segmented image using a connected component volume post-processing algorithm to obtain a volume post-processed image; and performing distance post-processing on the volume post-processed image using a connected component distance post-processing algorithm to obtain the target segmented image. The post-processing model is further configured to acquire each connected component of the volume post-processed image and calculate the volume of each connected component in the volume post-processed image; select a starting connected component from each connected component in the volume post-processed image based on a preset rule, wherein the connected component with the largest volume is selected as the starting connected component; calculate the representative coordinates of the starting connected component, and calculate the distance values ​​from each connected component in the volume post-processed image to the starting connected component according to the representative coordinates of the starting connected component; and clear the connected components in the volume post-processed image whose distance values ​​are greater than a preset distance threshold to obtain the target segmentation image.

6. A terminal device, characterized in that, The terminal device includes a memory, a processor, and an image processing program stored in the memory and executable on the processor. When the image processing program is executed by the processor, it implements the steps of the image processing method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image processing program, which, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1-4.

Images