Medical image segmentation method based on double-flow network

Abstract

The embodiment of the invention relates to the technical field of medical image segmentation, in particular to a medical image segmentation method based on a double-flow network, and the method comprises the steps: inputting a to-be-detected CCM image into the double-flow network which is composed of a segmentation branch and a detection branch, and enabling the segmentation branch and the detection branch to share a feature extractor, the segmentation branch is composed of an encoder, a decoder, a target convolution module and a 1 * 1 convolution layer, the feature extractor serves as the encoder in the segmentation branch, and the detection branch is composed of the feature extractor, an RPN network, an ROI pooling layer, a full connection layer and a boundary refining module; and finally, outputting a Langerhans cell segmentation result of the CCM image to be detected by the segmentation branch, and outputting a Langerhans cell detection result of the CCM image to be detected by the detection branch. According to the method, the segmentation and detection precision of the Langerhans cells is effectively improved, so that a doctor is well assisted in diagnosing the xerophthalmia.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of medical image segmentation, and in particular to a medical image segmentation method based on a dual-stream network. Background Art

[0002] CCM images generated based on CCM (Confocal Corneal Microscopy) play an important role in medical diagnosis. CCM is a high-resolution microscopy technology. Doctors can diagnose and monitor various corneal diseases, such as keratitis, corneal ulcers, dry eyes, etc. by analyzing CCM images. At present, clinical diagnosis of dry eyes can be done through a variety of means, including Schirmer test, corneal fluorescence staining, and tear film analysis, but these evaluation methods have some common problems, one of which is poor repeatability, and its reliability may vary depending on the operator's level, especially in the case of severe dry eyes, its accuracy is even more questionable.

[0003] The number and morphological information of Langerhans cells in CCM images can be used as the basis for the clinical diagnosis of dry eye. Segmenting Langerhans cells in CCM images through image segmentation technology can effectively improve the diagnostic accuracy and efficiency of dry eye in clinical diagnosis. However, the current image segmentation technology has the following defects and deficiencies when facing Langerhans cells in CCM images:

[0004] 1) The pixels of Langerhans cells in CCM images account for a small proportion and have little visual information, making it difficult to extract discriminative features, which results in low segmentation accuracy.

[0005] 2) Current image segmentation technology is susceptible to noise when processing small targets with low resolution, making it difficult to achieve accurate pixel-level segmentation.

[0006] 3) When segmenting small target objects, current image segmentation technology often ignores the combination of detection task information to optimize the segmentation task. Summary of the invention

[0007] In view of this, an embodiment of the present application proposes a medical image segmentation method based on a dual-stream network, which can accurately segment and detect Langerhans cells in CCM images. The image segmentation results obtained can well assist doctors in diagnosing dry eye.

[0008] In the first aspect, an embodiment of the present application proposes a medical image segmentation method based on a two-stream network, comprising: inputting a CCM image to be detected into a feature extractor of the two-stream network; wherein the two-stream network is composed of a segmentation branch and a detection branch, the segmentation branch and the detection branch share the same feature extractor, the segmentation branch is composed of an encoder, a decoder, a targeted convolution module and a 1×1 convolution layer, the feature extractor serves as an encoder in the segmentation branch, and the detection branch is composed of a feature extractor, an RPN network (Region Proposal Network), a ROI pooling layer, a fully connected layer and a boundary refinement module; the feature extractor extracts features of different levels of the CCM image to be detected, obtains feature maps of different levels and a first feature map that is finally output, inputs the feature maps of different levels into the decoder of the segmentation branch, and inputs the first feature map into the RPN network of the detection branch; the decoder upsamples the feature maps of different levels, outputs a preliminary semantic segmentation result to the targeted convolution module, the RPN network performs a proposal box detection on the first feature map, and outputs the proposal box to the targeted convolution module and the ROI pooling layer; the targeted convolution module The block uses the proposal box output by the RPN network to perform targeted convolution on the preliminary semantic segmentation result output by the decoder. After the output is processed by linear fusion of different channels through a 1×1 convolution layer, the Langerhans cell segmentation result of the CCM image to be detected is obtained, and it is used as the input of the boundary refinement module to accurately detect the detection result of the branch; the ROI pooling layer is used to extract feature maps of fixed size from candidate regions of different sizes; the fully connected layer is used for regression to obtain preliminary detection results; the boundary refinement module is used to combine the preliminary detection results with the segmentation results of the segmentation branch to obtain the final Langerhans cell detection result of the CCM image to be detected.

[0009] Optionally, the feature extractor adopts a ResNet structure, and the feature extractor consists of a 7×7 convolutional layer and four downsampling layers. The four downsampling layers are connected in sequence, namely the first downsampling layer, the second downsampling layer, the third downsampling layer and the fourth downsampling layer. The input of the 7×7 convolutional layer is the output of the first upsampling layer; the four downsampling layers are respectively used to extract features of different levels of the CCM image to be detected, and obtain feature maps of four different levels, among which the feature map output by the fourth downsampling layer is the first feature map; the four downsampling layers input the feature maps of different levels output by each of them into the decoder through jump connections, and the fourth downsampling layer also inputs the first feature map into the RPN network of the detection branch at the same time.

[0010] Optionally, the decoder consists of four sequentially connected upsampling layers, namely a first upsampling layer, a second upsampling layer, a third upsampling layer and a fourth upsampling layer; the first downsampling layer is jump-connected to the fourth upsampling layer, the second downsampling layer is jump-connected to the third upsampling layer, the third downsampling layer is jump-connected to the second upsampling layer, the fourth downsampling layer is jump-connected to the first upsampling layer, the feature map output by the first downsampling layer is input into the fourth upsampling layer, the feature map output by the second downsampling layer is input into the third upsampling layer, the feature map output by the third downsampling layer is input into the second upsampling layer, the feature map output by the first downsampling layer is input into the fourth upsampling layer, and the fourth upsampling layer finally outputs the second feature map to the targeted convolution module.

[0011] Optionally, the RPN network consists of a parallel classification layer and a regression layer, the first feature map is simultaneously input into the classification layer and the regression layer, the classification layer is used to perform target classification on the first feature map, the regression layer is used to perform bounding box regression correction on the first feature map, the RPN network fuses the outputs of the classification layer and the regression layer to obtain a proposal box, and outputs it to the targeted convolution module and the ROI pooling layer.

[0012] Optionally, the targeted convolution module receives the suggestion box information output by the RPN network in the detection branch, sets the feature pixels outside the suggestion box list in the preliminary semantic segmentation result to zero, and finally optimizes the semantic segmentation effect and speeds up the network reasoning speed;

[0013] The forward propagation of the targeted convolution module is implemented by the following formula:

[0014]

[0015] in, represents the output of the lth layer of the targeted convolutional module at point (i, j), x l-1 represents the activation of the l-1th layer of the targeted convolutional module, w l represents the filter of the lth layer of the targeted convolution module, the size of the filter is k×k, m×n represents the size of the second feature map, and ε represents the set of proposal boxes output by the RPN network.

[0016] Optionally, the first feature map and the set of suggested boxes output by the RPN network are input into the ROI pooling layer for ROI pooling operation, and after being processed by the fully connected layer, a preliminary detection result is obtained and enters the boundary refinement module; for the semantic segmentation mask, the boundary refinement module uses contour detection to determine all Langerhans cells, and then uses a one-dimensional convolution with a size of k to learn the feature distribution of the semantic segmentation mask on the four boundaries, and obtains the feature F(S) of the semantic segmentation mask on the left boundary. x); For the suggestion box, the boundary refinement module uses the coordinates of the left boundary of the suggestion box to calculate the boundary weight distribution W(l), and converts F(S x ) is multiplied by W(l) and then normalized as a whole to obtain the final corrected coordinates. The preliminary detection result is corrected based on the final corrected coordinates, and finally the Langerhans cell detection result of the CCM image to be detected is obtained.

[0017] Optionally, when training the two-stream network, sample CCM images are obtained, and relevant experts manually annotate some of the sample CCM images, including target detection annotation and pixel-level instance segmentation annotation of Langerhans cells in the sample CCM images, to obtain labeled data; combined with the labeled data, a semi-supervised image segmentation method based on contrastive learning is used to annotate the remaining unlabeled sample CCM images to generate pseudo labels, and finally the annotation of all sample CCM images is completed; based on the annotated sample CCM images, a training set, a validation set, and a test set are divided according to a ratio of 3:1:1; the two-stream network is trained based on the training set and the validation set, the initial learning rate is set to 0.00001, the optimizer uses an adaptive data estimation optimizer, and the batch size is set to 8.

[0018] In the above manner, the dual-stream network is used to segment and detect Langerhans cells in the CCM image to be detected. The segmentation branch and detection branch in the dual-stream network promote each other, and can perform high-precision segmentation of small targets (i.e., Langerhans cells in the CCM image). The image intersection-over-union performance has been effectively improved, proving the connection between Langerhans cells and dry eye. The Langerhans cell segmentation results and detection results of the CCM image to be detected output by the dual-stream network can provide auxiliary information for the clinical diagnosis of dry eye, effectively improving the efficiency and accuracy of clinical diagnosis.

[0019] In the second aspect, an embodiment of the present application proposes a medical image segmentation system based on a dual-stream network, the system comprising: a model building module, used to construct a dual-stream network consisting of a segmentation branch and a detection branch, the segmentation branch and the detection branch share the same feature extractor, the segmentation branch consists of an encoder, a decoder, a targeted convolution module and a 1×1 convolution layer, the feature extractor acts as an encoder in the segmentation branch, and the detection branch consists of a feature extractor, an RPN network and an ROI pooling layer, a fully connected layer and a boundary refinement module; a sample acquisition module, used to acquire sample CCM images and perform label annotation, and based on the annotated sample CCM images, divide the training set, validation set and test set into a ratio of 3:1:1; a model training module, used to train the dual-stream network based on the training set and the validation set; a model usage module, used to segment and detect the CCM images to be detected based on the trained dual-stream network.

[0020] In a third aspect, an embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a medical image segmentation method based on a two-stream network as described in the first aspect above.

[0021] In a fourth aspect, an embodiment of the present application proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement a medical image segmentation method based on a dual-stream network as described in the first aspect above.

[0022] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 is a flowchart of a medical image segmentation method based on a dual-stream network provided by an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of a dual-flow network structure provided by an embodiment of the present application;

[0026] Figure 3 is a schematic diagram of a sample CCM image and its annotation provided by an embodiment of the present application;

[0027] Figure 4 is a schematic diagram of forward propagation of a targeted convolution module provided by an embodiment of the present application;

[0028] Figure 5 is a schematic diagram of the working principle of a boundary refinement module provided by an embodiment of the present application;

[0029] Figure 6 is a Langerhans cell segmentation result diagram of a CCM image to be detected provided by an embodiment of the present application;

[0030] Figure 7is a Langerhans cell detection result diagram of a CCM image to be detected provided by an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of a specific structure of a medical image segmentation system based on a dual-stream network provided by another embodiment of the present application;

[0032] Fig. 9 It is a specific structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. In the various embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is only for the convenience of description, and the specific implementation mode of the present application should not constitute any limitation. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0034] In order to solve the problem of low segmentation accuracy of currently proposed image segmentation technologies when facing the scene of segmenting small targets and large background interference, an embodiment of the present application proposes a medical image segmentation method based on a dual-stream network, which is applied to an electronic device, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is explained by taking the server as an example. The implementation details of the medical image segmentation method based on a dual-stream network proposed in this embodiment are described in detail below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution. The specific process of the medical image segmentation method based on a dual-stream network proposed in this embodiment can be as follows: Figure 1 As shown, including:

[0035] Step 101, input the CCM image to be detected into the feature extractor of the two-stream network, wherein the two-stream network consists of a segmentation branch and a detection branch, the segmentation branch and the detection branch share the same feature extractor, the segmentation branch consists of an encoder, a decoder, a targeted convolution module and a 1×1 convolution layer, the feature extractor acts as an encoder in the segmentation branch, and the detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer and a boundary refinement module.

[0036] In the specific implementation, when the server segments and detects the CCM image to be detected, it needs to input the CCM image to be detected into the feature extractor of the two-stream network. The two-stream network is pre-built and trained by the server. The two-stream network consists of a segmentation branch and a detection branch. The segmentation branch and the detection branch share the same feature extractor. The segmentation branch consists of an encoder, a decoder, a targeted convolution module, and a 1×1 convolutional layer. The feature extractor exists as an encoder in the segmentation branch. The detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer, and a boundary refinement module.

[0037] In an example, the specific structure of the two-stream network can be as follows Figure 2 shown.

[0038] In one example, when training a two-stream network, the server first obtains sample CCM images, and relevant experts manually annotate some of the sample CCM images, including target detection annotation and pixel-level instance segmentation annotation of Langerhans cells in the sample CCM images, to obtain annotated data. Considering that the manual annotation process is very time-consuming, for most sample CCM images, the server needs to combine the annotated data and use a semi-supervised image segmentation method based on contrastive learning to annotate the sample CCM images of unlabeled data to generate pseudo labels, and finally complete the annotation of all sample CCM images. Based on the annotated sample CCM images, the server divides the training set, validation set, and test set in a ratio of 3:1:1, and trains the two-stream network based on the training set and validation set, where the initial learning rate is set to 0.00001, the optimizer uses an adaptive data estimation optimizer, and the batch size is set to 8.

[0039] In one example, a sample CCM image and its annotations can be shown as follows: Figure 3 As shown, from left to right are the sample CCM image, the target detection annotation schematic image and the pixel-level instance segmentation annotation result image.

[0040] Step 102: The feature extractor extracts features of different levels of the CCM image to be detected, obtains feature maps of different levels and a first feature map that is finally output, inputs the feature maps of different levels into the decoder of the segmentation branch, and inputs the first feature map into the RPN network of the detection branch.

[0041] In the specific implementation, the feature extractor is responsible for extracting features of different levels of the CCM image to be detected, obtaining feature maps of different levels and the first feature map of the final output, inputting the feature maps of different levels into the decoder of the segmentation branch, and inputting the first feature map into the RPN network of the detection branch.

[0042] In an example, Figure 2As shown in the figure, the feature extractor adopts the ResNet structure. The feature extractor consists of a 7×7 convolutional layer and four downsampling layers. The four downsampling layers are connected sequentially, which are the first downsampling layer, the second downsampling layer, the third downsampling layer and the fourth downsampling layer from top to bottom. The input of the 7×7 convolutional layer is the output of the first upsampling layer.

[0043] The four downsampling layers are respectively used to extract features at different levels of the CCM image to be detected, and obtain feature maps at four different levels, wherein the feature map output by the fourth downsampling layer is the first feature map.

[0044] The four downsampling layers input the feature maps of different levels output by each of them into the decoder respectively, and the fourth downsampling layer also inputs the first feature map into the RPN network of the detection branch.

[0045] The four downsampling layers are composed of two 3×3 convolutional layers, one 1×1 convolutional layer and one 2×2 pooling layer. The step size of the pooling layer is set to 2. After each convolution operation, the data is activated and pooled. The number of channels in each layer is twice that of the previous input.

[0046] The shallow texture features extracted by the feature extractor can provide category information for the classification of image pixels, while the deep abstract features extracted can provide location information for semantic segmentation.

[0047] In step 103, the decoder upsamples the feature maps at different levels and outputs the preliminary semantic segmentation results to the targeted convolution module. The RPN network performs proposal box detection on the first feature map and outputs the proposal box to the targeted convolution module and the ROI pooling layer.

[0048] In the specific implementation, the decoder is responsible for upsampling feature maps at different levels to restore feature space information and size, and output the second feature map (i.e., the preliminary semantic segmentation result) to the targeted convolution module. The RPN network is responsible for detecting the proposal box on the first feature map, and then outputs the proposal box to the targeted convolution module and ROI pooling layer after target classification and bounding box regression correction.

[0049] In one example, the feature extractor adopts the ResNet structure. Since the feature extractor exists as a decoder in the segmentation branch, the segmentation branch follows the ResUNet structure in structure. Figure 2As shown, the encoder of the segmentation branch consists of four sequentially connected upsampling layers, namely the first upsampling layer, the second upsampling layer, the third upsampling layer and the fourth upsampling layer. The first downsampling layer is jump-connected to the fourth upsampling layer, the second downsampling layer is jump-connected to the third upsampling layer, the third downsampling layer is jump-connected to the second upsampling layer, and the fourth downsampling layer is jump-connected to the first upsampling layer. The feature map output by the first downsampling layer is input into the fourth upsampling layer, the feature map output by the second downsampling layer is input into the third upsampling layer, the feature map output by the third downsampling layer is input into the second upsampling layer, the feature map output by the first downsampling layer is input into the fourth upsampling layer, and the fourth upsampling layer finally outputs the second feature map to the targeted convolution module.

[0050] Each of the four upsampling layers consists of a 1×1 convolutional layer and a 2×2 deconvolutional layer. The function of the deconvolutional layer is to enlarge the reduced image in the network until it is restored to its original size.

[0051] It is understandable that the ResUNet structure combines the advantages of the UNet network's fast segmentation speed and the ResNet network's advantages of increasing network depth while reducing training errors. Introducing residual connections based on the UNet architecture can not only increase the layer depth of the model structure, but also effectively overcome the gradient diffusion problem, while also enhancing the segmentation branch's ability to capture detailed features.

[0052] In an example, Figure 2 As shown in the figure, the RPN network consists of a parallel classification layer and a regression layer. The first feature map is input into the classification layer and the regression layer at the same time. The classification layer is used to classify the target of the first feature map, and the regression layer is used to perform bounding box regression correction on the first feature map. The RPN network fuses the outputs of the classification layer and the regression layer to obtain the proposal box, and outputs it to the targeted convolution module and the ROI pooling layer.

[0053] In step 104, the targeted convolution module uses the suggestion box output by the RPN network to perform targeted convolution on the preliminary semantic segmentation result output by the decoder. After the output is processed by linear fusion of different channels through a 1×1 convolution layer, the Langerhans cell segmentation result of the CCM image to be detected is obtained, and it is used as the input of the boundary refinement module to accurately detect the detection result of the branch.

[0054] In the specific implementation, after the RPN network and the decoder complete their respective tasks, the targeted convolution module uses the proposal box output by the RPN network and the preliminary semantic segmentation result output by the decoder for targeted convolution. After its output is processed by a 1×1 convolution layer, the Langerhans cell segmentation result of the CCM image to be detected is obtained, and it is used as the input of the boundary refinement module for the detection result of the accurate detection branch.

[0055] In an example, Figure 4 As shown in the figure, the targeted convolution module uses the proposal boxes output by the RPN network and the preliminary semantic segmentation results for targeted convolution. This design enables the segmentation task to focus more on the region of interest, thereby improving the segmentation accuracy. By performing convolution operations on the proposal boxes output by the RPN network, the targeted convolution module is able to perform more detailed feature extraction for each candidate region, which helps the network better understand the pixel-level features in each region, thereby optimizing the segmentation effect. Compared with the traditional ROI pooling layer, the targeted convolution module performs convolution operations directly on the output list of the RPN, which can retain more feature details, especially in the boundary area of ​​the target. In addition, the targeted convolution module can process all proposal boxes at once during training and inference, which simplifies the calculation process and speeds up training.

[0056] In one example, the forward propagation of the targeted convolution module is implemented by the following formula:

[0057]

[0058] in, represents the output of the lth layer of the targeted convolutional module at point (i, j), x l-1 represents the activation of the l-1th layer of the targeted convolutional module, w l represents the filter of the lth layer of the targeted convolution module, the size of the filter is k×k, m×n represents the size of the second feature map, and ε represents the set of proposal boxes output by the RPN network.

[0059] In step 105, the ROI pooling layer is used to extract feature maps of fixed size from candidate regions of different sizes, the fully connected layer is used to regress to obtain preliminary detection results, and the boundary refinement module is used to combine the preliminary detection results with the segmentation results of the segmentation branch to obtain the final Langerhans cell detection results of the CCM image to be detected.

[0060] In the specific implementation, the first feature map and the set of suggestion boxes output by the RPN network are input into the ROI pooling layer for ROI pooling operation. The output is processed by the fully connected layer to obtain a preliminary detection result. Then, the semantic segmentation mask of the segmentation branch is used to refine the boundary of the preliminary detection result to obtain the Langerhans cell detection result of the CCM image to be detected.

[0061] In an example, Figure 2 , Figure 5As shown in the figure, the detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer, and a boundary refinement module. The first feature map and the suggestion box output by the RPN network are input into the ROI pooling layer for ROI pooling operation. After being processed by the fully connected layer, the preliminary detection result is obtained and enters the boundary refinement module. For the semantic segmentation mask, the boundary refinement module uses contour detection to determine all Langerhans cells, and then uses a one-dimensional convolution of size k to learn the feature distribution of the semantic segmentation mask on the four boundaries, and obtains the feature F(S) of the semantic segmentation mask on the left boundary. x ). For the proposed box, the edge refinement module uses the coordinates of the left boundary of the proposed box to calculate the boundary weight distribution W(l), and converts F(S x ) is multiplied by W(l) and then normalized as a whole to obtain the final corrected coordinates. The preliminary detection result is corrected based on the final corrected coordinates, and finally the Langerhans cell detection result of the CCM image to be detected is obtained.

[0062] It is worth noting that the detection branch of the two-stream network follows a two-stage detection structure, which is more accurate than other one-stage detection networks. The RPN network targets natural image objects, and the parameters of the anchor boxes are set to values ​​with a wide range of scales and aspect ratios. The size of Langerhans cells is very small. Even the smallest RPN anchor box is much larger than the size of Langerhans cells in CCM images. Therefore, it is necessary to adjust the initial RPN anchor box size. The server first counts the size of Langerhans cells in the dataset. The statistical results show that the size of most Langerhans cells is in the range of [100, 800], while the initial anchor box sizes in RPN are 128 and 128, respectively. 2 , 256 2 ,512 2 , which is much larger than the size of Langerhans cells themselves. In order to better adapt to the size of Langerhans cells, the server changes the initial anchor box size of the RPN network to 10 in the two-stream model detection branch. 2 , 20 2 , 40 2 , while the original length-to-width ratio remains unchanged, which are 1:1, 1:2, and 2:1 respectively. By adjusting the size of the anchor box, the two-stream network can be more adapted to Langerhans cells and enhance the network's detection sensitivity to small targets.

[0063] In one example, Figure 6 and Figure 7 They are the Langerhans cell segmentation result diagram and the detection result diagram of the CCM image to be detected. Figure 6 and Figure 7It can be seen that the two-stream network can well segment and detect Langerhans cells in CCM images, and the accuracy of the segmentation results is good, which effectively solves the problem of low segmentation accuracy of the currently proposed image segmentation technology when facing the scene of segmenting small targets and large background interference.

[0064] In this embodiment, a dual-stream network is used to segment and detect Langerhans cells in the CCM image to be detected. The segmentation branch and the detection branch in the dual-stream network promote each other, and can perform high-precision segmentation of small targets (i.e., Langerhans cells in the CCM image). The image intersection-over-union performance is effectively improved, which proves the connection between Langerhans cells and dry eye. The Langerhans cell segmentation results and detection results of the CCM image to be detected output by the dual-stream network can provide auxiliary information for the clinical diagnosis of dry eye, effectively improving the efficiency and accuracy of clinical diagnosis.

[0065] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0066] Another embodiment of the present application proposes a medical image segmentation system based on a dual-stream network. The implementation details of the medical image segmentation system based on a dual-stream network proposed in this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for implementing this example. Figure 8 20 is a schematic diagram of the structure of a medical image segmentation system based on a dual-stream network proposed in this embodiment. The system includes: a model building module 201, a sample acquisition module 202, a model training module 203 and a model use module 204.

[0067] The model building module 201 is used to build a two-stream network consisting of a segmentation branch and a detection branch. The segmentation branch and the detection branch share the same feature extractor. The segmentation branch consists of an encoder, a decoder, a targeted convolution module and a 1×1 convolution layer. The feature extractor acts as an encoder in the segmentation branch. The detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer and a boundary refinement module.

[0068] The sample acquisition module 202 is used to acquire sample CCM images and perform label annotation, and based on the labeled sample CCM images, divide them into a training set, a validation set and a test set in a ratio of 3:1:1.

[0069] The model training module 203 is used to train the two-stream network based on the training set and the validation set.

[0070] The model using module 204 is used to segment and detect the CCM image to be detected based on the trained two-stream network.

[0071] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0072] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiments.

[0073] Another embodiment of the present application provides an electronic device, whose specific structure is as follows: Fig. 9 As shown, it includes: at least one processor 301; and a memory 302 that is communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute a medical image segmentation method based on a dual-stream network described in the above-mentioned method embodiments.

[0074] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be further described in this article. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be one component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0075] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0076] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a medical image segmentation method based on a dual-stream network described in the above method embodiments.

[0077] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), disk or optical disk and other media that can store program codes.

[0078] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A medical image segmentation method based on a two-stream network, characterized in that: include: Input the CCM image to be detected into the feature extractor of the two-stream network; wherein the two-stream network consists of a segmentation branch and a detection branch, the segmentation branch and the detection branch share the same feature extractor, the segmentation branch consists of an encoder, a decoder, a targeted convolution module and a 1×1 convolution layer, the feature extractor acts as an encoder in the segmentation branch, and the detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer and a boundary refinement module; The feature extractor extracts features of different levels of the CCM image to be detected, obtains feature maps of different levels and the first feature map of the final output, inputs the feature maps of different levels into the decoder of the segmentation branch, and inputs the first feature map into the RPN network of the detection branch; The decoder upsamples the feature maps at different levels and outputs the preliminary semantic segmentation results to the targeted convolution module. The RPN network performs proposal box detection on the first feature map and outputs the proposal box to the targeted convolution module and ROI pooling layer. The targeted convolution module uses the proposed box output by the RPN network to perform targeted convolution on the preliminary semantic segmentation result output by the decoder. After the output is processed by the 1×1 convolution layer for linear fusion of different channels, the Langerhans cell segmentation result of the CCM image to be detected is obtained, and it is used as the input of the boundary refinement module to accurately detect the detection results of the branch; The ROI pooling layer is used to extract feature maps of fixed size from candidate regions of different sizes; The fully connected layer is used for regression to obtain preliminary detection results; The boundary refinement module is used to combine the preliminary detection results with the segmentation results of the segmentation branch to obtain the final Langerhans cell detection results of the CCM image to be detected.

2. According to claim 1, a medical image segmentation method based on a two-stream network is characterized in that: The feature extractor adopts the ResNet structure. The feature extractor consists of a 7×7 convolution layer and four downsampling layers. The four downsampling layers are connected in sequence, namely the first downsampling layer, the second downsampling layer, the third downsampling layer and the fourth downsampling layer. The input of the 7×7 convolution layer is the output of the first upsampling layer. The four downsampling layers are respectively used to extract features of different levels of the CCM image to be detected, and obtain feature maps of four different levels, wherein the feature map output by the fourth downsampling layer is the first feature map; The four downsampling layers input the feature maps of different levels output by each of them into the decoder through jump connections. The fourth downsampling layer also inputs the first feature map into the RPN network of the detection branch.

3. The medical image segmentation method based on a two-stream network according to claim 2, characterized in that: The decoder consists of four sequentially connected upsampling layers, namely the first upsampling layer, the second upsampling layer, the third upsampling layer and the fourth upsampling layer; The first downsampling layer is jump-connected to the fourth upsampling layer, the second downsampling layer is jump-connected to the third upsampling layer, the third downsampling layer is jump-connected to the second upsampling layer, and the fourth downsampling layer is jump-connected to the first upsampling layer; The feature map output by the first downsampling layer is input into the fourth upsampling layer, the feature map output by the second downsampling layer is input into the third upsampling layer, the feature map output by the third downsampling layer is input into the second upsampling layer, the feature map output by the first downsampling layer is input into the fourth upsampling layer, and the fourth upsampling layer finally outputs the second feature map to the targeted convolution module.

4. The medical image segmentation method based on a two-stream network according to claim 1, characterized in that: The RPN network consists of a parallel classification layer and a regression layer. The first feature map is input into the classification layer and the regression layer at the same time. The classification layer is used to classify the target of the first feature map, and the regression layer is used to perform bounding box regression correction on the first feature map. The RPN network fuses the outputs of the classification layer and the regression layer to obtain the proposal box, and outputs it to the targeted convolution module and the ROI pooling layer.

5. The medical image segmentation method based on a two-stream network according to claim 4, characterized in that: The targeted convolution module receives the suggestion box information output by the RPN network in the detection branch, sets the feature pixels outside the suggestion box list in the preliminary semantic segmentation results to zero, and finally optimizes the semantic segmentation effect and speeds up the network reasoning speed; The forward propagation of the targeted convolution module is implemented by the following formula: in, represents the output of the lth layer of the targeted convolutional module at point (i, j), x l-1 represents the activation of the l-1th layer of the targeted convolutional module, w l represents the filter of the lth layer of the targeted convolution module, the size of the filter is k×k, m×n represents the size of the second feature map, and ε represents the set of proposal boxes output by the RPN network.

6. The medical image segmentation method based on a two-stream network according to claim 1, characterized in that: The first feature map and the set of suggested boxes output by the RPN network are input into the ROI pooling layer for ROI pooling operation. After being processed by the fully connected layer, the preliminary detection results are obtained and enter the boundary refinement module. For the semantic segmentation mask, the boundary refinement module uses contour detection to determine all Langerhans cells, and then uses a one-dimensional convolution with a size of k to learn the feature distribution of the semantic segmentation mask on the four boundaries, and obtains the feature F(S) of the semantic segmentation mask on the left boundary. x ); For the proposed box, the boundary refinement module uses the coordinates of the left boundary of the proposed box to calculate the boundary weight distribution W(l) and converts F(S x ) is multiplied by W(l) and then normalized as a whole to obtain the final corrected coordinates. The preliminary detection result is corrected based on the final corrected coordinates, and finally the Langerhans cell detection result of the CCM image to be detected is obtained.

7. A medical image segmentation method based on a two-stream network according to any one of claims 1 to 6, characterized in that: When training the two-stream network, sample CCM images are obtained, and relevant experts manually annotate some of the sample CCM images, including target detection annotation and pixel-level instance segmentation annotation of Langerhans cells in the sample CCM images, to obtain annotated data; Combined with the labeled data, the semi-supervised image segmentation method based on contrastive learning is used to annotate the remaining unlabeled sample CCM images to generate pseudo labels, and finally the labeling of all sample CCM images is completed; Based on the labeled sample CCM images, the training set, validation set, and test set are divided into training set, validation set, and test set in a ratio of 3:1:1; The two-stream network is trained based on the training set and the validation set. The initial learning rate is set to 0.00001, the optimizer uses the adaptive data estimation optimizer, and the batch size is set to 8.

8. A medical image segmentation system based on a two-stream network, characterized in that: include: The model building module is used to build a two-stream network consisting of a segmentation branch and a detection branch. The segmentation branch and the detection branch share the same feature extractor. The segmentation branch consists of an encoder, a decoder, a targeted convolution module and a 1×1 convolution layer. The feature extractor acts as an encoder in the segmentation branch. The detection branch consists of a feature extractor, an RPN network, a ROI pooling layer, a fully connected layer and a boundary refinement module. The sample acquisition module is used to acquire sample CCM images and label them. Based on the labeled sample CCM images, the training set, validation set, and test set are divided according to the ratio of 3:1:

1. Model training module, used to train the two-stream network based on the training set and the validation set; The model uses a module to segment and detect CCM images based on the trained two-stream network.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a medical image segmentation method based on a dual-stream network as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement a medical image segmentation method based on a dual-stream network as described in any one of claims 1 to 7.

Images