A medical image segmentation method based on two-stream network

Abstract

Embodiments of the present application relate to the field of medical image segmentation technology, and more particularly to a medical image segmentation method based on a two-stream network. The method comprises: inputting a CCM image to be detected into a two-stream network, the two-stream network consisting of a segmentation branch and a detection branch, the segmentation branch and the detection branch sharing a feature extractor. The segmentation branch consists of an encoder, a decoder, a targeted convolution module, and a 1×1 convolution layer. The feature extractor serves as an encoder in the segmentation branch, and the detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer, and a boundary refinement module. Finally, the segmentation branch outputs the Langerhans cell segmentation result of the CCM image to be detected, and the detection branch outputs the Langerhans cell detection result of the CCM image to be detected. This method effectively improves the accuracy of Langerhans cell segmentation and detection, thereby effectively assisting doctors in diagnosing dry eye.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of medical image segmentation, and in particular to a medical image segmentation method based on a dual-stream network. Background Art

[0002] CCM images generated based on CCM (Confocal Corneal Microscopy) play an important role in medical diagnosis. CCM is a high-resolution microscopy technology that allows doctors to diagnose and monitor various corneal diseases, such as keratitis, corneal ulcers, and dry eye, by analyzing CCM images. Currently, clinical diagnosis of dry eye can be done through a variety of methods, including the Schirmer test, corneal fluorescence staining, and tear film analysis. However, these evaluation methods have some common problems, one of which is poor repeatability, and their reliability may vary depending on the operator's level. Especially in the case of severe dry eye, their accuracy is even more questionable.

[0003] The number and morphology of Langerhans cells in CCM images can serve as a basis for the clinical diagnosis of dry eye. Segmenting Langerhans cells in CCM images using image segmentation technology can effectively improve the accuracy and efficiency of clinical diagnosis of dry eye. However, current image segmentation technology has the following defects and deficiencies when dealing with Langerhans cells in CCM images:

[0004] 1) The Langerhans cells in CCM images have a small pixel ratio and little visual information, making it difficult to extract discriminative features, which leads to low segmentation accuracy.

[0005] 2) Current image segmentation technology is susceptible to noise when processing small targets with low resolution, making it difficult to achieve accurate pixel-level segmentation.

[0006] 3) When segmenting small target objects, current image segmentation technology often ignores the integration of detection task information to optimize the segmentation task. Summary of the Invention

[0007] In view of this, an embodiment of the present application proposes a medical image segmentation method based on a dual-stream network, which can accurately segment and detect Langerhans cells in CCM images. The image segmentation results obtained can well assist doctors in diagnosing dry eye.

[0008] In the first aspect, an embodiment of the present application proposes a medical image segmentation method based on a dual-stream network, comprising: inputting a CCM image to be detected into a feature extractor of the dual-stream network; wherein the dual-stream network is composed of a segmentation branch and a detection branch, the segmentation branch and the detection branch share the same feature extractor, the segmentation branch is composed of an encoder, a decoder, a targeted convolution module and a 1×1 convolution layer, the feature extractor serves as an encoder in the segmentation branch, and the detection branch is composed of a feature extractor, an RPN network (Region Proposal Network), a ROI pooling layer, a fully connected layer and a boundary refinement module; the feature extractor extracts features of different levels of the CCM image to be detected, obtains feature maps of different levels and a first feature map that is finally output, inputs the feature maps of different levels into the decoder of the segmentation branch, and inputs the first feature map into the RPN network of the detection branch; the decoder upsamples the feature maps of different levels, outputs preliminary semantic segmentation results to the targeted convolution module, the RPN network performs proposal box detection on the first feature map, and outputs the proposal box to the targeted convolution module and the ROI pooling layer; the targeted convolution module The block uses the proposed box output by the RPN network to perform targeted convolution on the preliminary semantic segmentation result output by the decoder. After the output is processed by linear fusion of different channels in a 1×1 convolution layer, the Langerhans cell segmentation result of the CCM image to be detected is obtained, and it is used as the input of the boundary refinement module to accurately detect the detection result of the branch; the ROI pooling layer is used to extract fixed-size feature maps from candidate regions of different sizes; the fully connected layer is used for regression to obtain preliminary detection results; the boundary refinement module is used to combine the preliminary detection results with the segmentation results of the segmentation branch to obtain the final Langerhans cell detection result of the CCM image to be detected.

[0009] Optionally, the feature extractor adopts a ResNet structure, and the feature extractor consists of a 7×7 convolutional layer and four downsampling layers. The four downsampling layers are connected sequentially, namely the first downsampling layer, the second downsampling layer, the third downsampling layer and the fourth downsampling layer. The input of the 7×7 convolutional layer is the output of the first upsampling layer; the four downsampling layers are used to extract features of different levels of the CCM image to be detected, and obtain feature maps of four different levels, among which the feature map output by the fourth downsampling layer is the first feature map; the four downsampling layers input the feature maps of different levels output by each of them into the decoder through jump connections, and the fourth downsampling layer also inputs the first feature map into the RPN network of the detection branch.

[0010] Optionally, the decoder consists of four sequentially connected upsampling layers, namely the first upsampling layer, the second upsampling layer, the third upsampling layer and the fourth upsampling layer; the first downsampling layer is jump-connected to the fourth upsampling layer, the second downsampling layer is jump-connected to the third upsampling layer, the third downsampling layer is jump-connected to the second upsampling layer, and the fourth downsampling layer is jump-connected to the first upsampling layer. The feature map output by the first downsampling layer is input into the fourth upsampling layer, the feature map output by the second downsampling layer is input into the third upsampling layer, the feature map output by the third downsampling layer is input into the second upsampling layer, the feature map output by the first downsampling layer is input into the fourth upsampling layer, and the fourth upsampling layer finally outputs the second feature map to the targeted convolution module.

[0011] Optionally, the RPN network consists of a parallel classification layer and a regression layer. The first feature map is input into the classification layer and the regression layer at the same time. The classification layer is used to perform target classification on the first feature map, and the regression layer is used to perform bounding box regression correction on the first feature map. The RPN network fuses the outputs of the classification layer and the regression layer to obtain a proposal box, and outputs it to the targeted convolution module and the ROI pooling layer.

[0012] Optionally, the targeted convolution module receives the proposed box information output by the RPN network in the detection branch and sets the feature pixels outside the proposed box list in the preliminary semantic segmentation results to zero, ultimately optimizing the semantic segmentation effect and accelerating the network's reasoning speed.

[0013] The forward propagation of the targeted convolution module is implemented by the following formula:

[0014] ;

[0015] in, represents the first Layer at point The output, represents the first The activation of the layer, represents the first The filter of the layer has a filter size of , represents the size of the second feature map, Represents the set of proposal boxes output by the RPN network.

[0016] Optionally, the first feature map and the set of suggested boxes output by the RPN network are input into the ROI pooling layer for ROI pooling operation. After being processed by the fully connected layer, the preliminary detection results are obtained and enter the boundary refinement module. For the semantic segmentation mask, the boundary refinement module uses contour detection to determine all Langerhans cells, and then uses the size of The one-dimensional convolution learns the feature distribution of the semantic segmentation mask on the four boundaries, and obtains the feature of the semantic segmentation mask on the left boundary ; For the suggestion box, the boundary refinement module uses the coordinates of the left boundary of the suggestion box to calculate the boundary weight distribution ,Will and After multiplication, the coordinates are normalized as a whole to obtain the final corrected coordinates. The preliminary detection results are corrected based on the final corrected coordinates to finally obtain the Langerhans cell detection results of the CCM image to be detected.

[0017] Optionally, when training the two-stream network, sample CCM images are obtained, and relevant experts manually annotate some of the sample CCM images, including target detection annotation and pixel-level instance segmentation annotation of Langerhans cells in the sample CCM images, to obtain labeled data; combined with the labeled data, a semi-supervised image segmentation method based on contrastive learning is used to annotate the remaining unlabeled sample CCM images to generate pseudo labels, and finally the annotation of all sample CCM images is completed; based on the annotated sample CCM images, a training set, a validation set, and a test set are divided into a ratio of 3:1:1; the two-stream network is trained based on the training set and the validation set, the initial learning rate is set to 0.00001, the optimizer uses an adaptive data estimation optimizer, and the batch size is set to 8.

[0018] In the above manner, a dual-stream network is used to segment and detect Langerhans cells in the CCM image to be detected. The segmentation branch and detection branch in the dual-stream network promote each other, and can perform high-precision segmentation of small targets (i.e., Langerhans cells in the CCM image). The image intersection-over-union performance has been effectively improved, proving the connection between Langerhans cells and dry eye. The Langerhans cell segmentation results and detection results of the CCM image to be detected output by the dual-stream network can provide auxiliary information for the clinical diagnosis of dry eye, effectively improving the efficiency and accuracy of clinical diagnosis.

[0019] In the second aspect, an embodiment of the present application proposes a medical image segmentation system based on a dual-stream network, the system including: a model construction module for constructing a dual-stream network consisting of a segmentation branch and a detection branch, the segmentation branch and the detection branch share the same feature extractor, the segmentation branch consists of an encoder, a decoder, a targeted convolution module and a 1×1 convolution layer, the feature extractor serves as an encoder in the segmentation branch, and the detection branch consists of a feature extractor, an RPN network and an ROI pooling layer, a fully connected layer and a boundary refinement module; a sample acquisition module for acquiring sample CCM images and performing label annotation, and based on the annotated sample CCM images, dividing the training set, validation set and test set into a ratio of 3:1:1; a model training module for training the dual-stream network based on the training set and validation set; a model usage module for segmenting and detecting the CCM images to be detected based on the trained dual-stream network.

[0020] In a third aspect, an embodiment of the present application proposes an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a medical image segmentation method based on a dual-stream network as described in the first aspect above.

[0021] In a fourth aspect, an embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a medical image segmentation method based on a dual-stream network as described in the first aspect above.

[0022] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the following is a brief introduction to the drawings required for use in the embodiments of the present application or the description of the related technologies. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 This is a flowchart of a medical image segmentation method based on a dual-stream network provided by an embodiment of the present application;

[0025] Figure 2 This is a schematic diagram of the structure of a dual-flow network provided by an embodiment of the present application;

[0026] Figure 3 is a schematic diagram of a sample CCM image and its annotation provided by an embodiment of the present application;

[0027] Figure 4 Schematic diagram of the forward propagation of a targeted convolution module provided by one embodiment of the present application;

[0028] Figure 5 is a schematic diagram of the working principle of a boundary refinement module provided by an embodiment of the present application;

[0029] Figure 6 is a Langerhans cell segmentation result diagram of a CCM image to be detected provided by an embodiment of the present application;

[0030] Figure 7 This is a Langerhans cell detection result diagram of a CCM image to be detected provided by an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of the specific structure of a medical image segmentation system based on a dual-stream network provided by another embodiment of the present application;

[0032] Figure 9 It is a schematic diagram of the specific structure of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the various embodiments of the present application, many technical details are proposed to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is only for the convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0034] In order to solve the problem of low segmentation accuracy of the currently proposed image segmentation technology when facing the scene of segmenting small targets and large background interference, an embodiment of the present application proposes a medical image segmentation method based on a dual-stream network, which is applied to electronic devices, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is described using the server as an example. The implementation details of the medical image segmentation method based on a dual-stream network proposed in this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details, and is not necessary for the implementation of this solution. The specific process of the medical image segmentation method based on a dual-stream network proposed in this embodiment can be as follows: Figure 1 Shown, including:

[0035] Step 101: Input the CCM image to be detected into the feature extractor of the dual-stream network, wherein the dual-stream network consists of a segmentation branch and a detection branch. The segmentation branch and the detection branch share the same feature extractor. The segmentation branch consists of an encoder, a decoder, a targeted convolution module and a 1×1 convolution layer. The feature extractor serves as an encoder in the segmentation branch. The detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer and a boundary refinement module.

[0036] In the specific implementation, when the server performs segmentation and detection on the CCM image to be detected, it needs to input the CCM image to be detected into the feature extractor of the two-stream network. The two-stream network is pre-built and trained by the server. The two-stream network consists of a segmentation branch and a detection branch. The segmentation branch and the detection branch share the same feature extractor. The segmentation branch consists of an encoder, a decoder, a targeted convolution module, and a 1×1 convolutional layer. The feature extractor exists as an encoder in the segmentation branch. The detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer, and a boundary refinement module.

[0037] In an example, the specific structure of the two-stream network can be as follows Figure 2 shown.

[0038] In one example, when training a two-stream network, the server first obtains sample CCM images, and relevant experts manually annotate some of the sample CCM images, including target detection annotation and pixel-level instance segmentation annotation of Langerhans cells in the sample CCM images, to obtain annotated data. Considering that the manual annotation process is very time-consuming, for most sample CCM images, the server needs to combine the annotated data and use a semi-supervised image segmentation method based on contrastive learning to annotate the sample CCM images of the unlabeled data to generate pseudo labels, and finally complete the annotation of all sample CCM images. Based on the annotated sample CCM images, the server divides the training set, validation set, and test set in a ratio of 3:1:1, and trains the two-stream network based on the training set and validation set. The initial learning rate is set to 0.00001, the optimizer uses the adaptive estimation optimizer, and the batch size is set to 8.

[0039] In one example, a sample CCM image and its annotations can be shown as follows: Figure 3 As shown in the figure, from left to right are the sample CCM image, the target detection annotation schematic image and the pixel-level instance segmentation annotation result image.

[0040] In step 102, the feature extractor extracts features at different levels of the CCM image to be detected, obtains feature maps at different levels and a first feature map as the final output, inputs the feature maps at different levels into the decoder of the segmentation branch, and inputs the first feature map into the RPN network of the detection branch.

[0041] In the specific implementation, the feature extractor is responsible for extracting features at different levels of the CCM image to be detected, obtaining feature maps at different levels and the first feature map of the final output, inputting the feature maps at different levels into the decoder of the segmentation branch, and inputting the first feature map into the RPN network of the detection branch.

[0042] In one example, Figure 2 As shown in the figure, the feature extractor adopts the ResNet structure. The feature extractor consists of a 7×7 convolutional layer and four downsampling layers. The four downsampling layers are connected sequentially, namely the first downsampling layer, the second downsampling layer, the third downsampling layer and the fourth downsampling layer from top to bottom. The input of the 7×7 convolutional layer is the output of the first upsampling layer.

[0043] The four downsampling layers are respectively used to extract features at different levels of the CCM image to be detected, and obtain feature maps at four different levels, wherein the feature map output by the fourth downsampling layer is the first feature map.

[0044] The four downsampling layers input the feature maps of different levels output by each layer into the decoder respectively, and the fourth downsampling layer also inputs the first feature map into the RPN network of the detection branch.

[0045] The four downsampling layers are composed of two 3×3 convolutional layers, one 1×1 convolutional layer and one 2×2 pooling layer. The step size of the pooling layer is set to 2. After each convolution operation, the data is activated and pooled. The number of channels in each layer is twice that of the previous input.

[0046] The shallow texture features extracted by the feature extractor can provide category information for the classification of image pixels, while the deep abstract features extracted can provide location information for semantic segmentation.

[0047] In step 103, the decoder upsamples the feature maps at different levels and outputs the preliminary semantic segmentation results to the targeted convolution module. The RPN network performs proposal box detection on the first feature map and outputs the proposal box to the targeted convolution module and ROI pooling layer.

[0048] In the specific implementation, the decoder is responsible for upsampling feature maps at different levels to restore feature space information and size, and outputting the second feature map (i.e., the preliminary semantic segmentation result) to the targeted convolution module. The RPN network is responsible for detecting the proposal box of the first feature map, and then performing target classification and bounding box regression correction, and outputting the proposal box to the targeted convolution module and ROI pooling layer.

[0049] In one example, the feature extractor adopts the ResNet structure. Since the feature extractor exists as a decoder in the segmentation branch, the segmentation branch follows the ResUNet structure in structure. Figure 2 As shown, the encoder of the segmentation branch consists of four sequentially connected upsampling layers: the first upsampling layer, the second upsampling layer, the third upsampling layer, and the fourth upsampling layer. The first downsampling layer is skip-connected to the fourth upsampling layer, the second downsampling layer is skip-connected to the third upsampling layer, the third downsampling layer is skip-connected to the second upsampling layer, and the fourth downsampling layer is skip-connected to the first upsampling layer. The feature map output by the first downsampling layer is input to the fourth upsampling layer, the feature map output by the second downsampling layer is input to the third upsampling layer, the feature map output by the third downsampling layer is input to the second upsampling layer, the feature map output by the first downsampling layer is input to the fourth upsampling layer, and the fourth upsampling layer finally outputs the second feature map to the targeted convolution module.

[0050] Each of the four upsampling layers consists of a 1×1 convolutional layer and a 2×2 deconvolutional layer. The function of the deconvolutional layer is to enlarge the image that has been reduced in the network until it is restored to its original size.

[0051] It's understandable that the ResUNet architecture combines the fast segmentation speed of the UNet network with the advantages of the ResNet network, which increases network depth while reducing training error. Introducing residual connections based on the UNet architecture not only increases the layer depth of the model structure, but also effectively overcomes the gradient diffusion problem, while also strengthening the segmentation branch's ability to capture detailed features.

[0052] In one example, Figure 2 As shown in the figure, the RPN network consists of a parallel classification layer and a regression layer. The first feature map is input into the classification layer and the regression layer at the same time. The classification layer is used to classify the target of the first feature map, and the regression layer is used to perform bounding box regression correction on the first feature map. The RPN network fuses the outputs of the classification layer and the regression layer to obtain the proposed box, and outputs it to the targeted convolution module and the ROI pooling layer.

[0053] In step 104, the targeted convolution module uses the proposed box output by the RPN network to perform targeted convolution on the preliminary semantic segmentation result output by the decoder. After the output is processed by the 1×1 convolution layer for linear fusion of different channels, the Langerhans cell segmentation result of the CCM image to be detected is obtained, and it is used as the input of the boundary refinement module to accurately detect the detection result of the branch.

[0054] In the specific implementation, after the RPN network and decoder complete their respective tasks, the targeted convolution module uses the proposal box output by the RPN network and the preliminary semantic segmentation result output by the decoder to perform targeted convolution. After its output is processed by a 1×1 convolution layer, the Langerhans cell segmentation result of the CCM image to be detected is obtained, and it is used as the input of the boundary refinement module for accurate detection of the detection results of the branch.

[0055] In one example, Figure 4 As shown in the figure, the targeted convolution module uses the proposal boxes output by the RPN network and the preliminary semantic segmentation results for targeted convolution. This design enables the segmentation task to focus more on the region of interest, thereby improving segmentation accuracy. By performing convolution operations on the proposal boxes output by the RPN network, the targeted convolution module can perform more detailed feature extraction for each candidate region, which helps the network better understand the pixel-level features within each region, thereby optimizing the segmentation effect. Compared with the traditional ROI pooling layer, the targeted convolution module performs convolution operations directly on the output list of the RPN, which can retain more feature details, especially in the boundary area of ​​the target. In addition, the targeted convolution module can process all proposal boxes at once during training and inference, which simplifies the calculation process and speeds up training.

[0056] In one example, the forward propagation of the targeted convolution module is implemented as follows:

[0057] ;

[0058] in, represents the first layer of the targeted convolutional module Layer at point The output, represents the first layer of the targeted convolutional module The activation of the layer, represents the first layer of the targeted convolutional module The filter of the layer has a filter size of , represents the size of the second feature map, Represents the set of proposal boxes output by the RPN network.

[0059] In step 105, the ROI pooling layer is used to extract fixed-size feature maps from candidate regions of different sizes, the fully connected layer is used to regress to obtain preliminary detection results, and the boundary refinement module is used to combine the preliminary detection results with the segmentation results of the segmentation branch to obtain the final Langerhans cell detection results of the CCM image to be detected.

[0060] In the specific implementation, the first feature map and the set of suggested boxes output by the RPN network are input into the ROI pooling layer for ROI pooling operation. After the output is processed by the fully connected layer, the preliminary detection result is obtained. Then, the semantic segmentation mask of the segmentation branch is used to refine the boundary of the preliminary detection result to obtain the Langerhans cell detection result of the CCM image to be detected.

[0061] In one example, Figure 2 、 Figure 5 As shown in Figure 2, the detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer, and a boundary refinement module. The first feature map and the proposed box output by the RPN network are input to the ROI pooling layer for ROI pooling operation. After processing by the fully connected layer, the preliminary detection result is obtained and enters the boundary refinement module. For the semantic segmentation mask, the boundary refinement module uses contour detection to determine all Langerhans cells, and then uses a size of The one-dimensional convolution learns the feature distribution of the semantic segmentation mask on the four boundaries, and obtains the feature of the semantic segmentation mask on the left boundary For the proposed box, the edge refinement module uses the coordinates of the left boundary of the proposed box to calculate the boundary weight distribution ,Will and After multiplication, the coordinates are normalized as a whole to obtain the final corrected coordinates. The preliminary detection results are corrected based on the final corrected coordinates to finally obtain the Langerhans cell detection results of the CCM image to be detected.

[0062] It is worth noting that the detection branch of the two-stream network follows a two-stage detection structure, which is more accurate than other one-stage detection networks. The RPN network targets natural image objects, and the parameters of the anchor box are set to values ​​with a wide range of scales and aspect ratios. The size of Langerhans cells is very small. Even the smallest RPN anchor box is much larger than the size of Langerhans cells in CCM images. Therefore, it is necessary to adjust the size of the RPN initial anchor box. The server first counts the size of the Langerhans cells in the dataset. The statistical results show that the size of most Langerhans cells is in the range of [100, 800], while the initial anchor box sizes in RPN are 128 and 128, respectively. 2 , 256 2 , 512 2, which is much larger than the size of Langerhans cells themselves. In order to better adapt to the size of Langerhans cells, the server changes the initial anchor box size of the RPN network to 10 in the dual-stream model detection branch. 2 , 20 2 , 40 2 , while the original aspect ratio remains unchanged, which are 1:1, 1:2, and 2:1 respectively. By adjusting the size of the anchor box, the two-stream network can be more adapted to Langerhans cells and enhance the network's detection sensitivity for small targets.

[0063] In one example, Figure 6 and Figure 7 They are the Langerhans cell segmentation result graph and the detection result graph of the CCM image to be detected, respectively. Figure 6 and Figure 7 It can be seen that the dual-stream network can well segment and detect Langerhans cells in CCM images, and the segmentation results are accurate, which effectively solves the problem of low segmentation accuracy of the currently proposed image segmentation technology when facing scenes with small targets and large background interference.

[0064] In this embodiment, a dual-stream network is used to segment and detect Langerhans cells in the CCM image to be detected. The segmentation branch and detection branch in the dual-stream network promote each other, and can perform high-precision segmentation of small targets (i.e., Langerhans cells in the CCM image). The image intersection-over-union performance is effectively improved, proving the connection between Langerhans cells and dry eye. The Langerhans cell segmentation results and detection results of the CCM image to be detected output by the dual-stream network can provide auxiliary information for the clinical diagnosis of dry eye, effectively improving the efficiency and accuracy of clinical diagnosis.

[0065] The step division of the above various methods is only for the purpose of clear description. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0066] Another embodiment of the present application proposes a medical image segmentation system based on a dual-stream network. The following is a detailed description of the implementation details of the medical image segmentation system based on a dual-stream network proposed in this embodiment. The following content is only for the convenience of understanding the implementation details and is not necessary for the implementation of this embodiment. Figure 8 20 is a schematic diagram of the structure of a medical image segmentation system based on a dual-stream network proposed in this embodiment. The system includes: a model construction module 201, a sample acquisition module 202, a model training module 203 and a model use module 204.

[0067] The model construction module 201 is used to construct a dual-stream network consisting of a segmentation branch and a detection branch. The segmentation branch and the detection branch share the same feature extractor. The segmentation branch consists of an encoder, a decoder, a targeted convolution module and a 1×1 convolution layer. The feature extractor serves as an encoder in the segmentation branch. The detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer and a boundary refinement module.

[0068] The sample acquisition module 202 is used to acquire sample CCM images and perform labeling. Based on the labeled sample CCM images, the training set, validation set, and test set are divided according to the ratio of 3:1:1.

[0069] The model training module 203 is used to train the dual-stream network based on the training set and the validation set.

[0070] The model using module 204 is used to segment and detect the CCM image to be detected based on the trained two-stream network.

[0071] It is worth mentioning that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovation of this application, this embodiment does not include units that are not closely related to solving the technical problem proposed by this application. However, this does not mean that other units do not exist in this embodiment.

[0072] It is not difficult to find that this embodiment is a system embodiment corresponding to the above-mentioned method embodiment, and this embodiment can be implemented in conjunction with the above-mentioned method embodiment. The relevant technical details and technical effects mentioned in the above-mentioned embodiments are still valid in this embodiment, and to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above-mentioned embodiments.

[0073] Another embodiment of the present application provides an electronic device, the specific structure of which is as follows: Figure 9 As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 to enable the at least one processor 301 to execute a medical image segmentation method based on a dual-stream network described in the above-mentioned method embodiments.

[0074] The memory and processor are connected via a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and will not be described further in this article. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by the processor is transmitted via an antenna on a wireless medium. Furthermore, the antenna also receives data and transmits it to the processor.

[0075] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0076] Another embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a medical image segmentation method based on a dual-stream network as described in the above method embodiments.

[0077] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program. The program is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: a USB flash drive, a mobile hard drive, ROM (Read-Only Memory), RAM (Random Access Memory), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0078] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A medical image segmentation method based on a two-stream network, characterized in that: include: The CCM image to be detected is input into the feature extractor of the two-stream network; the two-stream network consists of a segmentation branch and a detection branch. The segmentation branch and the detection branch share the same feature extractor. The segmentation branch consists of an encoder, a decoder, a targeted convolution module, and a 1×1 convolution layer. The feature extractor acts as an encoder in the segmentation branch. The detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer, and a boundary refinement module. The feature extractor extracts features at different levels of the CCM image to be detected, obtains feature maps at different levels and the first feature map as the final output, inputs the feature maps at different levels into the decoder of the segmentation branch, and inputs the first feature map into the RPN network and ROI pooling layer of the detection branch; The decoder upsamples the feature maps at different levels and outputs the preliminary semantic segmentation results to the targeted convolution module. The RPN network performs proposal box detection on the first feature map and outputs the proposal box to the targeted convolution module and ROI pooling layer. The targeted convolution module uses the proposed boxes output by the RPN network to perform targeted convolution on the preliminary semantic segmentation results output by the decoder. After the output is processed by a 1×1 convolution layer for linear fusion of different channels, the Langerhans cell segmentation result of the CCM image to be detected is obtained, which is used as the input of the boundary refinement module to accurately detect the detection results of the branch. The ROI pooling layer is used to extract fixed-size feature maps from candidate regions of different sizes; The fully connected layer is used for regression to obtain preliminary detection results; The boundary refinement module is used to combine the preliminary detection results and the segmentation results of the segmentation branch to obtain the final Langerhans cell detection results of the CCM image to be detected.

2. The medical image segmentation method based on a two-stream network according to claim 1, characterized in that: The feature extractor adopts the ResNet structure. The feature extractor consists of a 7×7 convolutional layer and four downsampling layers. The four downsampling layers are connected in sequence, namely the first downsampling layer, the second downsampling layer, the third downsampling layer, and the fourth downsampling layer. The output of the 7×7 convolutional layer is the input of the first downsampling layer. The four downsampling layers are used to extract features at different levels of the CCM image to be detected, and obtain feature maps at four different levels. The feature map output by the fourth downsampling layer is the first feature map. The four downsampling layers input the feature maps of different levels output by each of them into the decoder through jump connections. The fourth downsampling layer also inputs the first feature map into the RPN network of the detection branch.

3. The medical image segmentation method based on a dual-stream network according to claim 2, characterized in that: The decoder consists of four sequentially connected upsampling layers, namely the first upsampling layer, the second upsampling layer, the third upsampling layer and the fourth upsampling layer; The first downsampling layer is jump-connected to the fourth upsampling layer, the second downsampling layer is jump-connected to the third upsampling layer, the third downsampling layer is jump-connected to the second upsampling layer, and the fourth downsampling layer is jump-connected to the first upsampling layer; The feature map output by the first downsampling layer is input into the fourth upsampling layer, the feature map output by the second downsampling layer is input into the third upsampling layer, the feature map output by the third downsampling layer is input into the second upsampling layer, the feature map output by the first downsampling layer is input into the fourth upsampling layer, and the fourth upsampling layer finally outputs the second feature map to the targeted convolution module.

4. The medical image segmentation method based on a dual-stream network according to claim 3, characterized in that: The RPN network consists of a parallel classification layer and a regression layer. The first feature map is input into the classification layer and the regression layer at the same time. The classification layer is used to classify the target of the first feature map, and the regression layer is used to perform bounding box regression correction on the first feature map. The RPN network fuses the outputs of the classification layer and the regression layer to obtain the proposed box, and outputs it to the targeted convolution module and the ROI pooling layer.

5. The medical image segmentation method based on a dual-stream network according to claim 4, characterized in that: The targeted convolution module receives the proposed box information output by the RPN network in the detection branch and sets the feature pixels outside the proposed box list in the preliminary semantic segmentation results to zero, ultimately optimizing the semantic segmentation effect and accelerating the network's reasoning speed. The forward propagation of the targeted convolution module is implemented by the following formula: ； in, represents the first layer of the targeted convolutional module Layer at point The output, represents the first layer of the targeted convolutional module The activation of the layer, represents the first layer of the targeted convolutional module The filter of the layer has a filter size of , represents the size of the second feature map, Represents the set of proposal boxes output by the RPN network.

6. The medical image segmentation method based on a dual-stream network according to claim 1, characterized in that: The first feature map and the set of suggested boxes output by the RPN network are input into the ROI pooling layer for ROI pooling operation. After processing by the fully connected layer, the preliminary detection results are obtained and enter the boundary refinement module; For the semantic segmentation mask, the boundary refinement module uses contour detection to determine all Langerhans cells, and then uses the The one-dimensional convolution learns the feature distribution of the semantic segmentation mask on the four boundaries, and obtains the feature of the semantic segmentation mask on the left boundary ; For the suggestion box, the boundary refinement module uses the coordinates of the left boundary of the suggestion box to calculate the boundary weight distribution ,Will and After multiplication, the coordinates are normalized as a whole to obtain the final corrected coordinates. The preliminary detection results are corrected based on the final corrected coordinates to finally obtain the Langerhans cell detection results of the CCM image to be detected.

7. A medical image segmentation method based on a dual-stream network according to any one of claims 1 to 6, characterized in that: When training the two-stream network, sample CCM images are obtained, and relevant experts manually annotate some of the sample CCM images, including target detection and pixel-level instance segmentation annotation of Langerhans cells in the sample CCM images, to obtain annotated data; Combined with the labeled data, a semi-supervised image segmentation method based on contrastive learning is used to annotate the remaining unlabeled sample CCM images to generate pseudo labels, and finally complete the annotation of all sample CCM images; Based on the labeled sample CCM images, the training set, validation set, and test set are divided into two parts according to the ratio of 3:1:

1. The two-stream network is trained based on the training set and the validation set. The initial learning rate is set to 0.00001, the optimizer uses the adaptive data estimation optimizer, and the batch size is set to 8.

8. A medical image segmentation system based on a two-stream network, characterized in that: include: The model building module is used to build a two-stream network consisting of a segmentation branch and a detection branch. The segmentation branch and the detection branch share the same feature extractor. The segmentation branch consists of an encoder, a decoder, a targeted convolution module, and a 1×1 convolution layer. The feature extractor acts as an encoder in the segmentation branch. The detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer, and a boundary refinement module. The sample acquisition module is used to obtain sample CCM images and label them. Based on the labeled sample CCM images, the training set, validation set, and test set are divided according to the ratio of 3:1:

1. Model training module, used to train the two-stream network based on the training set and validation set; The model usage module is used to segment and detect the CCM image to be detected based on the trained two-stream network; During the model usage, the feature extractor extracts features at different levels of the CCM image to be detected, obtains feature maps at different levels and the first feature map as the final output, inputs the feature maps at different levels into the decoder of the segmentation branch, and inputs the first feature map into the RPN network and ROI pooling layer of the detection branch; The decoder upsamples the feature maps at different levels and outputs the preliminary semantic segmentation results to the targeted convolution module. The RPN network performs proposal box detection on the first feature map and outputs the proposal box to the targeted convolution module and ROI pooling layer. The targeted convolution module uses the proposed boxes output by the RPN network to perform targeted convolution on the preliminary semantic segmentation results output by the decoder. After the output is processed by a 1×1 convolution layer for linear fusion of different channels, the Langerhans cell segmentation result of the CCM image to be detected is obtained, which is used as the input of the boundary refinement module to accurately detect the detection results of the branch. The ROI pooling layer is used to extract fixed-size feature maps from candidate regions of different sizes; The fully connected layer is used for regression to obtain preliminary detection results; The boundary refinement module is used to combine the preliminary detection results and the segmentation results of the segmentation branch to obtain the final Langerhans cell detection results of the CCM image to be detected.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a medical image segmentation method based on a dual-stream network as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement a medical image segmentation method based on a dual-stream network as described in any one of claims 1 to 7.

Images