Focus image segmentation method and system based on pre-activation multi-scale

Abstract

The invention belongs to the technical field of image processing, and discloses a focus image segmentation method and system based on pre-activation multi-scale, and the method comprises the steps: constructing a Crohn disease stenosis lesion segmentation model, and carrying out the segmentation processing of the pre-collected Crohn disease stenosis image data through the Crohn disease stenosis lesion segmentation model, and obtaining a lesion image; lesion segmentation processing is carried out on the obtained lesion image, combined feature extraction is carried out on the lesion image subjected to segmentation processing, and a lesion segmentation model is constructed and trained in combination with a deep learning technology; and utilizing the trained lesion segmentation model to perform lesion segmentation processing on the Crohn disease stenosis image data acquired in real time to obtain a lesion segmentation result. According to the method, the Crohn disease stenosis lesion can be automatically segmented efficiently and accurately, so that a clinician can be helped to quickly and accurately judge the nature of the Crohn stenosis lesion, and the Crohn stenosis lesion is assisted to make a high-quality clinical decision.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to a method and system for segmenting lesion images based on pre-activated multi-scale. Background Art

[0002] The cumulative probability of stenosis in Crohn's disease (CD) within 1 year, 5 years, 10 years, and 20 years after diagnosis is approximately 7.2%, 12.4%, 15.2%, and 21.6% respectively. The lesion site of stricturing CD (Montreal B2) is mostly in the small intestine. Approximately 20% of stricturing CD is asymptomatic or has mild symptoms, and can only be detected by colonoscopy or computed tomography enterography (CTE). However, more than 1 / 3 of stricturing CD will progress to surgical complications and ultimately require surgical treatment. The stricturing lesions associated with CD can be divided into inflammatory stenosis and fibrotic stenosis. Inflammatory stenosis is mainly caused by the edema and thickening of the full layer of the intestinal tract due to acute inflammation, and this process is usually reversible and can recover after the inflammation subsides with drug treatment. Fibrotic stenosis, on the other hand, is the continuous repair of the intestinal tract under the repeated stimulation of chronic inflammation for several months or even years, resulting in the deposition of extracellular matrix and the replacement of the intestinal wall by scar tissue, ultimately losing the normal function of the intestine.

[0003] Although it is currently believed that inflammation is the initiating factor and necessary condition for the occurrence of fibrosis, fibrosis has other signaling pathways independent of inflammation. Sometimes, anastomotic sites will gradually develop fibrosis and stenosis even in the absence of significant inflammation. Currently, it is difficult to determine the nature of fibrotic stricture lesions in Crohn's disease, and it often requires surgical resection to clarify. However, at this time, the opportunity for conservative treatment and intestinal preservation has been lost, and the patient is at risk of perioperative complications and impaired quality of life. The ability to accurately diagnose stricturing lesions in Crohn's disease is limited. One of the important reasons is the lack of standard image definitions, examination stability, and quantitative scoring criteria. At the same time, such patients often have intestinal obstruction and it is difficult to perform colonoscopy tissue biopsy.

[0004] Currently, the commonly used CTE cannot effectively distinguish the nature of stricture lesions. Because CD often evolves into chronic inflammation after repeated acute inflammation attacks, and inflammatory stenosis will also progress to fibrotic stenosis over time, and this process is difficult to accurately reflect in the conventional CTE signs.

[0005] Therefore, how to provide a method and system for segmenting lesion images based on pre-activated multi-scale is an urgent problem to be solved currently. Summary of the Invention

[0006] Embodiments of the present invention provide a method and system for segmenting lesion images based on pre-activated multi-scale to solve the problems in the prior art.

[0007] To provide a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary section is not a general review, nor is it intended to identify key / important constituent elements or delineate the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0008] According to a first aspect of an embodiment of the present invention, a method for segmenting lesion images based on pre-activated multi-scale is provided.

[0009] In one embodiment, the method for segmenting lesion images based on pre-activated multi-scale includes:

[0010] Construct a segmentation model for Crohn's disease stricture lesions based on a pre-activated multi-scale residual network, and perform segmentation processing on pre-collected Crohn's disease stricture image data through the segmentation model for Crohn's disease stricture lesions to obtain lesion images;

[0011] Use the manual segmentation method to perform lesion segmentation processing on the obtained lesion images, extract combined features from the segmented lesion images, and construct and train a lesion segmentation model in combination with deep learning techniques;

[0012] Use the trained lesion segmentation model to perform lesion segmentation processing on real-time acquired Crohn's disease stricture image data to obtain a lesion segmentation result.

[0013] In one embodiment, the constructing a segmentation model for Crohn's disease stricture lesions based on a pre-activated multi-scale residual network, and performing segmentation processing on pre-collected Crohn's disease stricture image data through the segmentation model for Crohn's disease stricture lesions to obtain lesion images includes:

[0014] Construct a segmentation model for Crohn's disease stricture lesions composed of a multi-scale feature extraction module, a full-size skip connection layer, a pre-activation module, and a bottleneck layer;

[0015] In the multi-scale feature extraction module, in combination with the pre-activation module, use dilated convolution to extract features from the pre-acquired Crohn's disease stricture image data to obtain a set of feature maps containing different scale information;

[0016] In the bottleneck layer, perform dense compression processing on the extracted features through a preset dropout rate to obtain compressed feature maps;

[0017] In the full-size skip connection layer, fuse the compressed feature maps output by the bottleneck layer with the feature maps of the corresponding levels in the encoder part to obtain fused feature maps;

[0018] In the residual module, perform full-size residual connection between the fused feature maps and the Crohn's disease stricture images, and generate lesion images through upsampling and downsampling processing.

[0019] In one embodiment, in the multi-scale feature extraction module, in combination with a pre-activation module, dilated convolution is used to extract features from the previously obtained Crohn's disease stricture type image data, and the set of feature maps containing different scale information includes:

[0020] The Crohn's disease stricture type image data is pre-activated through the activation function in the pre-activation module;

[0021] The pre-activated Crohn's disease stricture type image data is initially feature-extracted through the convolutional layer in the multi-scale feature extraction module to obtain an initial feature map;

[0022] For the initial feature map, two-dimensional dilated convolution operation is performed through the dilated convolution channel and different scale feature maps are output according to the operation result, so as to obtain a set of feature maps containing different scale information.

[0023] In one embodiment, the expression for the size of the feature map for performing two-dimensional dilated convolution operation through the dilated convolution channel and outputting different scale feature maps according to the operation result is:

[0024]

[0025] In the formula, H out / W out represents the height and length of the output feature map; H in / W in represents the height and length of the input feature map; p represents padding; s represents the stride of the convolution operation; d represents the dilation rate; k represents the size of the convolution kernel.

[0026] In one embodiment, in the bottleneck layer, the extracted features are densely compressed through a preset dropout rate to obtain a compressed feature map, including:

[0027] Several layers of dense connection blocks are used as the bottleneck layer, and the set of feature maps containing different scale information is used as the input of the bottleneck layer;

[0028] Based on the preset dropout rate, a dropout operation of the number of channels of the feature map is performed on each layer in the dense connection block;

[0029] The channels of the feature map that are not subjected to the dropout operation form a compressed feature map.

[0030] In one embodiment, after constructing the Crohn's disease stricture type lesion segmentation model based on the pre-activated multi-scale residual network, it includes:

[0031] A binary classification loss function is used to quantify the difference between the output data of the Crohn's disease stricture type lesion segmentation model and the true label data;

[0032] The difference between the foreground pixel region segmented by the Crohn's disease stricture lesion segmentation model and the true foreground pixel region is measured using a similarity index coefficient.

[0033] In one embodiment, the expression of the binary classification loss function is:

[0034]

[0035] In the formula, loss(x,class) represents the binary classification loss function; x represents the output data; class represents the label data; j represents the count.

[0036] In one embodiment, the expression of the similarity index coefficient is:

[0037]

[0038] In the formula, Dice(R,G) represents the similarity index; R represents the segmented foreground pixel region; G represents the true foreground pixel region; TP represents true positive; FP represents false positive; FN represents false negative.

[0039] In one embodiment, using the manual segmentation method to perform lesion segmentation processing on the obtained lesion images, and through the combined feature extraction of the segmented lesion images, the construction and training of the lesion segmentation model in combination with deep learning technology includes:

[0040] Based on medical image visualization software, perform lesion segmentation processing on the obtained lesion images using the manual segmentation method, generate a three-dimensional region of interest and extract radiomics features;

[0041] Based on a 3D deep learning model, combined with the Self-Attention mechanism, analyze the region of interest of the image, and extract 3D image deep learning features with self-attention mechanism;

[0042] Construct combined features based on radiomics features and 3D image deep learning features, and use several machine learning methods to construct and train a machine learning model;

[0043] Evaluate the accuracy of each machine learning model, and select the model with the highest accuracy as the final deep learning model;

[0044] Use the final deep learning model as the lesion segmentation model.

[0045] According to the second aspect of the embodiments of the present invention, a lesion image segmentation system based on pre-activated multi-scale is provided.

[0046] In one embodiment, the lesion image segmentation system based on pre-activated multi-scale includes:

[0047] A lesion image segmentation unit for constructing a Crohn's disease stricture lesion segmentation model based on a pre-activated multi-scale residual network, and performing segmentation processing on pre-collected Crohn's disease stricture imaging data through the Crohn's disease stricture lesion segmentation model to obtain lesion images;

[0048] A module construction unit for performing lesion segmentation processing on the obtained lesion images by using the manual segmentation method, extracting combined features from the segmented lesion images, and constructing and training a lesion segmentation model in combination with deep learning techniques;

[0049] A lesion segmentation unit for performing lesion segmentation processing on real-time acquired Crohn's disease stricture imaging data by using the trained lesion segmentation model to obtain a lesion segmentation result.

[0050] According to the third aspect of the embodiments of the present invention, a computer device is provided.

[0051] In some embodiments, the computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0052] According to the fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided.

[0053] In one embodiment, a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0054] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0055] 1. The present invention can efficiently and accurately perform automatic segmentation on Crohn's disease stricture lesions to help clinicians quickly and accurately judge the nature of Crohn's stricture lesions, thereby assisting them in making high-quality clinical decisions.

[0056] 2. In the present invention, the amount of edge information that can be extracted from the feature map by dilated convolution will decrease as the resolution decreases. Multi-scale feature fusion can retain this part of edge information, which is of great significance for solving the problems of blurred boundaries and overlapping of edge information and background pixels.

[0057] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention.

[0059] Figure 1 It is a flowchart of a lesion image segmentation method based on pre-activated multi-scale shown according to an exemplary embodiment;

[0060] Figure 2 It is a schematic block diagram of a lesion image segmentation system based on pre-activated multi-scale shown according to an exemplary embodiment;

[0061] Figure 3 It is a schematic structural diagram of a computer device shown according to an exemplary embodiment;

[0062] Figure 4 It is an overall structure diagram of a pre-activated multi-scale residual U-Net in a lesion image segmentation method based on pre-activated multi-scale shown according to an exemplary embodiment. Detailed implementation manners

[0063] The following description and drawings fully illustrate the specific embodiments herein, enabling those skilled in the art to practice them. Parts and features of some embodiments may be included in or substituted for parts and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims and all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, and do not require or imply any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a structure, device or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such structure, device or equipment. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the structure, device or equipment including the said element. The embodiments herein are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same and similar parts among the embodiments can be referred to each other.

[0064] The terms "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. in this text indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only for the convenience of describing this text and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation on the present invention. In the description of this text, unless otherwise specified and defined, the terms "mounted", "connected", and "coupled" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the communication inside two elements. It can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0065] In this text, unless otherwise stated, the term "a plurality of" means two or more.

[0066] In this text, the character " / " indicates that the objects before and after are in an "or" relationship. For example, A / B means: A or B.

[0067] In this text, the term "and / or" is an associative relationship describing an object, indicating that there can be three relationships. For example, A and / or B means: A or B, or, A and B these three relationships.

[0068] It should be understood that although the various steps in the flowchart are shown sequentially in the direction of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this text, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, at least a part of the steps in the figure may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these sub-steps or stages is not necessarily sequential either, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0069] Each module in the device or system of this application can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to the above-mentioned modules.

[0070] Without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0071] Figure 1An embodiment of the lesion image segmentation method based on pre-activated multi-scale of the present invention is shown.

[0072] In this alternative embodiment, the lesion image segmentation method based on pre-activated multi-scale includes:

[0073] Step S101, constructing a Crohn's disease stricture lesion segmentation model based on a pre-activated multi-scale residual network, and performing segmentation processing on pre-collected Crohn's disease stricture imaging data through the Crohn's disease stricture lesion segmentation model to obtain a lesion image;

[0074] Step S102, performing lesion segmentation processing on the obtained lesion image by using the manual segmentation method, extracting combined features from the segmented lesion image, and constructing and training a lesion segmentation model by combining deep learning techniques;

[0075] Step S103, using the trained lesion segmentation model to perform lesion segmentation processing on real-time obtained Crohn's disease stricture imaging data to obtain a lesion segmentation result.

[0076] In this alternative embodiment, when constructing a Crohn's disease stricture lesion segmentation model based on a pre-activated multi-scale residual network and performing segmentation processing on pre-collected Crohn's disease stricture imaging data through the Crohn's disease stricture lesion segmentation model to obtain a lesion image, a Crohn's disease stricture lesion segmentation model composed of a multi-scale feature extraction module, a full-size skip connection layer, a pre-activation module, and a bottleneck layer can be constructed; in the multi-scale feature extraction module, in combination with the pre-activation module, dilated convolution is used to extract features from the pre-obtained Crohn's disease stricture imaging data to obtain a set of feature maps containing different scale information; in the bottleneck layer, the extracted features are densely compressed through a preset dropout rate to obtain a compressed feature map; in the full-size skip connection layer, the compressed feature map output by the bottleneck layer is fused with the feature map of the corresponding level in the encoder part to obtain a fused feature map; in the residual module, the fused feature map is subjected to full-size residual connection with the Crohn's disease stricture imaging, and through upsampling and downsampling processing, a lesion image is generated.

[0077] In this alternative embodiment, in the multi-scale feature extraction module, when combining the pre-activation module and using dilated convolution to extract features from the pre-obtained Crohn's disease stricture imaging data to obtain a set of feature maps containing different scale information, the Crohn's disease stricture imaging data can be pre-activated through the activation function in the pre-activation module; the convolutional layer in the multi-scale feature extraction module is used to perform preliminary feature extraction on the pre-activated Crohn's disease stricture imaging data to obtain a preliminary feature map; for the preliminary feature map, two-dimensional dilated convolution operation is performed through a dilated convolution channel and different scale feature maps are output according to the operation result to obtain a set of feature maps containing different scale information.

[0078] In this alternative embodiment, when performing intensive compression processing on the extracted features through a preset dropout rate in the bottleneck layer to obtain a compressed feature map, several dense connection blocks can be used as the bottleneck layer, and a set of feature maps containing different scale information can be used as the input of the bottleneck layer; based on the preset dropout rate, a dropout operation on the number of channels of the feature map is performed on each layer in the dense connection block; the channels of the feature map that are not subject to the dropout operation form the compressed feature map.

[0079] After constructing the Crohn's disease stricture lesion segmentation model based on the pre-activated multi-scale residual network in this alternative embodiment, it includes: using a binary classification loss function to quantify the difference between the output data of the Crohn's disease stricture lesion segmentation model and the true label data; using a similarity index coefficient to measure the difference between the foreground pixel region segmented by the Crohn's disease stricture lesion segmentation model and the true foreground pixel region.

[0080] In this alternative embodiment, the expression of the binary classification loss function is:

[0081]

[0082] In the formula, loss(x,class) represents the binary classification loss function; x represents the output data; class represents the label data; j represents the count.

[0083] In this alternative embodiment, the expression of the similarity index coefficient is:

[0084]

[0085] In the formula, Dice(R,G) represents the similarity index; R represents the segmented foreground pixel region; G represents the true foreground pixel region; TP represents true positive; FP represents false positive; FN represents false negative.

[0086] In addition, the Jaccard coefficient or Hausdorff distance can also be used to measure the difference between the foreground pixel region segmented by the Crohn's disease stricture lesion segmentation model and the true foreground pixel region.

[0087] The Jaccard coefficient indicates the degree of difference between the foreground pixels of the segmented image and the true foreground pixel region. The specific calculation is as follows:

[0088]

[0089] In the formula, Jaccard(X,Y) represents the degree of difference between the foreground pixel region of the segmented image and the true foreground pixel region, R represents the foreground pixel region of the segmented image, and G represents the true foreground pixel region.

[0090] The Hausdorff distance represents the maximum value of the shortest distance from a point in one point set to another point set. The two point sets in the segmentation task are the foreground pixels and the true foreground pixel region respectively. This metric is a measure of shape similarity and is calculated as follows:

[0091] Hausdorff(X,Y) = max(sup x∈X inf y∈Y d(x,y),sup y∈Y inf x∈X d(x,y));

[0092] Among them, Hausdorff(X,Y) represents the similarity between the actual prediction result and the true result, X represents the actual prediction result, Y represents the true result, sup and inf represent the upper and lower bounds respectively, and d(x,y) represents the Euclidean distance between the actual prediction result x and the true result y, where x belongs to X and y belongs to Y.

[0093] To facilitate the understanding of the above technical solution of the present invention, the above technical solution of the present invention will be further described from the perspectives of architecture and principle as follows:

[0094] In practical applications, the present invention is based on a pre-activated multi-scale residual U-Net for automatic segmentation of Crohn's disease stricture lesions. Its overall structure diagram is as Figure 4 shown (MIB represents the multi-scale Inception block, TDB represents the downsampling module, Concat represents feature concatenation, and Fusion represents the pre-activation module). This network consists of a multi-scale Inception module (multi-scale feature extraction module), full-scale skip connections, and a pre-activation module. First, four multi-scale Inception modules are used to extract multi-scale features from the input image layer by layer; then, a bottleneck layer designed with dense connections is used to densely compress the features and reduce redundancy. Finally, full-scale skip connections are used for multi-layer feature fusion to reconstruct the contour. A pre-activation sub-module is inserted into each module to propagate feature information back and forth.

[0095] Next, the extended convolutional Inception module for multi-scale feature extraction, the full-scale skip connection structure for context information fusion, the pre-activation module, and the bottleneck layer structure will be introduced separately in the order of encoding and decoding in the model.

[0096] (1) Multi-scale Inception module;

[0097] To detect the features of the target organ at different scales, multi-scale inference is usually used in the network model. In the encoding stage, multi-scale feature fusion is performed according to the receptive field ranges of different sizes to obtain a high-resolution and rich-semantic feature map. In traditional segmentation models, the convolutional layer extracts features and then downsamples the image through pooling to reduce the image size and increase the receptive field. The pooling size and convolutional kernel size determine the size of the receptive field. If the model is shallow, it is difficult to model fuzzy boundaries. Deepening the model by stacking small convolutional kernels instead of large ones can be very cumbersome and computationally expensive, which usually leads to overfitting problems. Inspired by the parallel multi-branch network structure of GoogLeNet, a multi-scale Inception module is proposed, which applies dilated convolution for multi-scale feature fusion in the encoding stage.

[0098] The implementation method is as follows: Dilated convolution (also known as atrous convolution) allows controlling the receptive field by only changing the dilation rate of the convolution while keeping the size of the convolutional kernel unchanged, without increasing the number of parameters or computational redundancy. The input of the dilated convolution is set as F: Z 2 →R, making Ω r =[-r,r] 2 ∩Z 2 , f: Ω r →R as the discrete filter, and the expression for the size of the feature map outputting different-scale feature maps through the two-dimensional dilated convolution operation in the atrous convolution channel and based on the operation result is:

[0099]

[0100] In the formula, H out / W out represents the height and length of the output feature map; H in / W in represents the height and length of the input feature map; p represents padding; s represents the stride of the convolution operation; d represents the dilation rate; k represents the size of the convolutional kernel.

[0101] The multi-scale Inception module extracts features of different scales from the input feature map through four sets of parallel dilated convolution channels, and then connects the four sets of output feature maps. Then, a set of 1×1 convolution kernels is used for feature fusion. A pre-activation module is used before each convolution operation. In the entire network coding process, a combination of four such Inception modules and a maximum pooling module is used as a four-layer downsampling layer. The multi-scale module only increases the number of feature map channels, and the maximum pooling module reduces the size of the feature map by half. The multi-scale Inception module can learn multi-scale features with fewer parameters, which is of great significance for solving the limitation of small data sets of medical images. During the coding process, the multi-scale Inception module implicitly learns the position information of the left ventricle through convolution with a large dilation rate, and extracts the edge information of the left ventricle through convolution with a small dilation rate. The amount of edge information extracted from the feature map will decrease as the resolution decreases. Multi-scale feature fusion can retain this part of the edge information, which is of great significance for solving the problem of blurred boundaries and the overlap of edge information and background pixels.

[0102] (2) Bottleneck layer;

[0103] The bottleneck layer structure is common in ResNet, which uses 1×1 convolution operations to allow the network structure to flexibly change dimensions. In the basic U-Net structure, the bottleneck layer is used during upsampling and downsampling conversions to reduce the number of channels to achieve dimensionality reduction. Compared with the last layer in the downsampling stage, the bottleneck layer contains fewer nodes, which can simplify the dimensional representation of features while ensuring that the size of the feature map remains unchanged. Inspired by DenseNet, this chapter uses a four-layer densely connected block as the bottleneck layer. Each layer in the densely connected block uses a dropout operation (Dropout) with a loss rate of 0.5 to prevent overfitting of the network. After this bottleneck layer, the number of feature map channels is reduced from 136 to 32, which reduces the number of network parameters while helping to reasonably combine features to eliminate redundancy, stimulating the network's compressed feature representation.

[0104] (3) pre-activation module;

[0105] In a deep network, the distribution of data in each layer is different, which makes it difficult for network training to converge. Batch normalization is widely used before the activation function ReLU to normalize the input of the activation function and solve the impact of data shift and increase. Existing studies have widely confirmed that the pre-activation model can effectively alleviate the overfitting phenomenon.

[0106] In the Inception structure, using the batch normalization + ReLU pre-activation module before each convolution operation can ensure that all heavy layers are normalized. Compared with after activation, the multi-scale features extracted by parallel dilated convolution using the pre-activation module are retained. Then the output feature maps of all channels are reused together with the pre-activation module. Batch normalization reduces the feature complexity, and ReLU increases the non-linearity between the four convolutional layers, reducing the dependency between parameters. For average pooling, the first activation may filter out negative values in the original convolutional output, resulting in some information loss. However, the max pooling module is used in all networks, and adding pre-activation before max pooling does not cause loss of feature information. In this mode, the overall network training convergence speed is improved, and feature information can not only be propagated forward but also backward. The entire network effectively controls gradient explosion and gradient vanishing, preventing the overfitting problem.

[0107] (4) Residual module;

[0108] In many studies, feature maps at different levels carry different semantic information. Low-scale feature maps focus on spatial information, highlighting the boundaries of target organs; high-level semantic feature maps locate the positions of target organs. During downsampling to increase the robustness of input image perturbations and upsampling to restore the image size, the fine-grained details of the image boundary may be diluted. Therefore, a full-size residual connection is designed to improve the segmentation accuracy and accelerate the model training speed.

[0109] Among them, different methods are used for ablation experiments and comparative tests, and the test results are shown in Tables 1 and 2.

[0110] Table 1 Results of ablation experiments

[0111] Method Dice Jaccard Hausdorff Base 37.5 27.5 20.8 Base+Multi-Skip 50.8 38.8 19.3 Base+Multi-Scale 53.7 42.5 18.6 Base+Multi-Scale+Multi-Skip 58.3 46.0 18.2 Base+Multi-Scale+Multi-Skip+PA 63.4 50.1 17.6

[0112] Table 2 Results of comparative tests

[0113] Method Dice Jaccard Hausdorff U-Net 37.5 27.5 20.8 UNet++ 49.6 38.4 19.7 Swin-Unet 55.3 42.8 19.0 TransUNet 57.6 46.1 18.4 Ours 63.4 50.1 17.6

[0114] Among them, Method represents the method, Base represents the basic method, Base+Multi-Skip represents the basic method + multi-hop connection; Base+Multi-Scale represents the basic method + multi-scale; Base+Multi-Scale+Multi-Skip represents the basic method + multi-scale + multi-hop connection; Base+Multi-Scale+Multi-Skip+PA represents the basic method + multi-scale + multi-hop connection + position attention / other specific attention mechanisms;

[0115] U-Net, UNet++, Swin-Unet, and TransUNet are all deep learning algorithms for image segmentation.

[0116] In this alternative embodiment, the method of manually segmenting the obtained lesion images for lesion segmentation processing, and extracting combined features from the segmented lesion images, and constructing and training a lesion segmentation model by combining deep learning technologies includes: based on medical image visualization software, performing lesion segmentation processing on the obtained lesion images by the manual segmentation method to generate a three-dimensional region of interest and extract radiomics features; based on a 3D deep learning model, combining the Self-Attention mechanism, analyzing the region of interest of the image, and extracting 3D image deep learning features with self-attention mechanism; constructing combined features based on radiomics features and 3D image deep learning features, and constructing and training a machine learning model by using several machine learning methods; evaluating the accuracy of each machine learning model, and selecting the model with the highest accuracy as the final deep learning model; using the final deep learning model as the lesion segmentation model.

[0117] To facilitate the understanding of the above technical solution of the present invention, the above technical solution of the present invention will be further described from the perspectives of architecture and principle as follows:

[0118] In actual application, the present invention adopts the manual segmentation method to ensure the accuracy of lesion segmentation. The segmentation work is completed by two radiologists with more than 5 years of abdominal radiology work experience using ITK-SNAP (version

[0119] 3.6.0). In order to comprehensively reflect the internal heterogeneity of the lesion, thin-slice images in the venous phase are selected to outline all levels of the lesion, carefully avoiding artifacts and gas, and automatically generating a three-dimensional region of interest (VOI). Images of 30 patients are randomly selected, and the above segmentation operations are performed by two operators respectively, and radiomics features are extracted respectively to evaluate the inter-observer consistency of this segmentation scheme.

[0120] Manually designed features (HCF) are automatically obtained based on the Python platform installation package PyRadiomics, including 14 morphological features (volume, surface area, surface area to volume ratio, compactness, etc.); 18 first-order features (mean, variance, uniformity, skewness, entropy, etc.); 75 texture features (including gray-level co-occurrence matrix, gray-level run-length matrix, gray-level size zone matrix, and gray-level dependence matrix); 1209 filter transformation features (square, square root, wavelet, etc.).

[0121] Based on a 3D deep learning model (3D-ResNet), adding the Self-Attention mechanism, analyzing the region of interest of the image, and finally extracting 3D image deep learning features with "self-attention mechanism".

[0122] Compare the use of deep learning models named 3D-Resnet18, 3D-densenet121, 3D-MobilenetV3, and 3D-ShuffleNetv2x05 to extract 3D deep learning features of the region of interest in images. This model contains a convolutional module and a classification module, and the construction process is divided into three main steps: 1) Connect a linear classifier using a convolutional network to obtain an effective feature extractor; 2) Freeze the convolutional network and connect a fully connected network with 512 hidden layer nodes to further integrate high-dimensional features and obtain a high-precision classifier; 3) Unfreeze all the parameters of the network and perform end-to-end training with a smaller learning rate.

[0123] The overall process is as follows. Pre-train a deep learning model with a 3D-Resnet architecture on the ImageNet dataset using transfer learning technology. Since this study is a binary classification problem, while ImageNet is a 1000-classification task, the 1000 neurons in the last fully connected layer of 3D-Resnet are changed to 2 neurons to adapt to the prediction task. Use CrossEntropyLoss (cross-entropy loss function) as the loss function for model training, the optimizer is Adam, the initial learning rate is 0.001, the batch size is set to 16, and the epoch is 1000. Select the model with the highest accuracy as the final deep learning model. Finally, fix the network parameters as the feature extractor of the deep learning model, remove the softmax layer, and use the output value of the last layer node as the deep learning feature. In addition, in order to fully capture the internal correlation of data or features during model training and reduce the dependence on external information, a self-attention mechanism is introduced during model training. After the CT images are processed in the same way, they are input into 3D-Resnet, and finally 512 3D image deep learning features are extracted through the 3D deep convolutional network.

[0124] Based on the manually designed features (HCF) and deep learning (DL) features of CT images, machine learning models with different feature combinations are constructed. Eleven machine learning methods are used to construct machine learning models respectively, including Logistic Regression, ExtraTrees, NaiveBayes, SVM, XGBoost, RandomForest, LightGBM, GradientBoosting, AdaBoost, KNN, and MLP. The diagnostic metrics of all machine learning models for the training set and validation set are output, including AUC value, accuracy, sensitivity, specificity, precision, positive predictive value, negative predictive value, recall rate, F1 value, etc. The prediction efficacy of each model is evaluated, and the best parameters of the model are searched and matched through hyperparameter tuning, so as to improve the performance of the model. The Delong test is applied to evaluate the statistical differences in the AUC values of different models.

[0125] Figure 2 An embodiment of the lesion image segmentation system based on pre-activated multi-scale of the present invention is shown.

[0126] In this alternative embodiment, the lesion image segmentation system based on pre-activated multi-scale includes:

[0127] A lesion image segmentation unit 201, configured to construct a Crohn's disease stricture lesion segmentation model based on a pre-activated multi-scale residual network, and perform segmentation processing on the pre-collected Crohn's disease stricture image data through the Crohn's disease stricture lesion segmentation model to obtain a lesion image;

[0128] A module construction unit 202, configured to perform lesion segmentation processing on the obtained lesion image by using the manual segmentation method, extract combined features from the segmented lesion image, and construct and train a lesion segmentation model in combination with deep learning technology;

[0129] A lesion segmentation unit 203, configured to perform lesion segmentation processing on the real-time acquired Crohn's disease stricture image data by using the trained lesion segmentation model to obtain a lesion segmentation result.

[0130] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as Figure 3As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0131] Those skilled in the art can understand that Figure 3 the structure shown in is only a block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0132] In addition, the present invention also provides a computer device, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, it implements the steps in the above method embodiments.

[0133] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0134] Those of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided by the present invention can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0135] The present invention is not limited to the structures already described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A method for segmenting lesion images based on pre-activated multi-scale, characterized in that, Including: Construct a Crohn's disease stricture lesion segmentation model based on a pre-activated multi-scale residual network, and use the Crohn's disease stricture lesion segmentation model to segment the pre-collected Crohn's disease stricture imaging data to obtain lesion images; Use the manual segmentation method to perform lesion segmentation on the obtained lesion images, extract combined features from the segmented lesion images, and construct and train a lesion segmentation model by combining deep learning techniques; Use the trained lesion segmentation model to perform lesion segmentation on the real-time obtained Crohn's disease stricture imaging data to obtain a lesion segmentation result.

2. The method for segmenting lesion images based on pre-activated multi-scale according to claim 1, wherein The constructing a Crohn's disease stricture lesion segmentation model based on a pre-activated multi-scale residual network, and using the Crohn's disease stricture lesion segmentation model to segment the pre-collected Crohn's disease stricture imaging data to obtain lesion images includes: Construct a Crohn's disease stricture lesion segmentation model composed of a multi-scale feature extraction module, a full-size skip connection layer, a pre-activation module, and a bottleneck layer; In the multi-scale feature extraction module, in combination with the pre-activation module, use dilated convolution to extract features from the pre-obtained Crohn's disease stricture imaging data to obtain a set of feature maps containing different scale information; In the bottleneck layer, perform dense compression processing on the extracted features through a preset dropout rate to obtain compressed feature maps; In the full-size skip connection layer, fuse the compressed feature maps output by the bottleneck layer with the feature maps of the corresponding levels in the encoder part to obtain fused feature maps; In the residual module, perform full-size residual connection between the fused feature maps and the Crohn's disease stricture imaging, and generate lesion images through upsampling and downsampling processing.

3. The method for segmenting lesion images based on pre-activated multi-scale according to claim 2, wherein The in the multi-scale feature extraction module, in combination with the pre-activation module, using dilated convolution to extract features from the pre-obtained Crohn's disease stricture imaging data to obtain a set of feature maps containing different scale information includes: Perform pre-activation processing on the Crohn's disease stricture imaging data through the activation function in the pre-activation module; Use the convolutional layer in the multi-scale feature extraction module to perform preliminary feature extraction on the pre-activated Crohn's disease stricture imaging data to obtain preliminary feature maps; For the preliminary feature maps, perform two-dimensional dilated convolution operations through the dilated convolution channels and output different scale feature maps according to the operation results to obtain a set of feature maps containing different scale information.

4. The method for segmenting lesion images based on pre-activated multi-scale according to claim 3, wherein The expression for the size of the feature maps obtained by performing two-dimensional dilated convolution operations through the dilated convolution channels and outputting different scale feature maps according to the operation results is: where, H out / W out represents the height and length of the output feature map; H in / W in represents the height and length of the input feature map; p represents padding; s represents the stride of the convolution operation; d represents the dilation rate; k represents the convolution kernel size.

5. The method for segmenting lesion images based on pre-activated multi-scale according to claim 3, characterized in that The in the bottleneck layer, performing dense compression processing on the extracted features through a preset dropout rate to obtain compressed feature maps includes: Use several layers of dense connection blocks as the bottleneck layer, and use the set of feature maps containing different scale information as the input of the bottleneck layer; Based on the preset dropout rate, perform a dropout operation on the number of channels of the feature maps in each layer of the dense connection block; The channels of the feature maps not subjected to the dropout operation form the compressed feature maps.

6. The method for segmenting lesion images based on pre-activated multi-scale according to claim 1, wherein, After constructing the Crohn's disease stricture lesion segmentation model based on the pre-activated multi-scale residual network, it includes: The binary classification loss function is used to quantify the difference between the output data of the Crohn's disease stricture lesion segmentation model and the true label data; The similarity index coefficient is used to measure the difference between the foreground pixel region segmented by the Crohn's disease stricture lesion segmentation model and the true foreground pixel region.

7. The method for segmenting lesion images based on pre-activated multi-scale according to claim 6, wherein The expression of the binary classification loss function is: In the formula, loss(x,class) represents the binary classification loss function; x represents the output data; class represents the label data; j represents the count.

8. The method for segmenting lesion images based on pre-activated multi-scale according to claim 6, characterized in that, The expression of the similarity index coefficient is: In the formula, Dice(R,G) represents the similarity index; R represents the segmented foreground pixel region; G represents the true foreground pixel region; TP represents true positive; FP represents false positive; FN represents false negative.

9. The method for segmenting lesion images based on pre-activated multi-scale according to claim 1, wherein The method of using manual segmentation to perform lesion segmentation on the obtained lesion images, extracting combined features from the segmented lesion images, and constructing and training a lesion segmentation model by combining deep learning techniques includes: Based on medical image visualization software, perform lesion segmentation on the obtained lesion images by manual segmentation, generate a three-dimensional region of interest, and extract radiomics features; Based on a 3D deep learning model, combined with the Self-Attention mechanism, analyze the image region of interest, and extract 3D image deep learning features with self-attention mechanism; Construct combined features based on radiomics features and 3D image deep learning features, and use several machine learning methods to construct and train a machine learning model; Evaluate the accuracy of each machine learning model, and select the model with the highest accuracy as the final deep learning model; Use the final deep learning model as the lesion segmentation model.

10. A lesion image segmentation system based on pre-activated multi-scale, characterized in that, It includes: A lesion image segmentation unit, which is used to construct a Crohn's disease stricture lesion segmentation model based on a pre-activated multi-scale residual network, and perform segmentation processing on pre-collected Crohn's disease stricture imaging data through the Crohn's disease stricture lesion segmentation model to obtain lesion images; A module construction unit, which is used to perform lesion segmentation on the obtained lesion images by manual segmentation, extract combined features from the segmented lesion images, and construct and train a lesion segmentation model by combining deep learning techniques; A lesion segmentation unit, which is used to perform lesion segmentation on real-time acquired Crohn's disease stricture imaging data by using the trained lesion segmentation model to obtain a lesion segmentation result.

Images